NVIDIA and nouveau

The release of source code for NVIDIA graphics hardware was perhaps something of a surprise; at least at a quick glance, it seems like that could lead to an in-tree, officially supported driver. For many years, though, the nouveau project has been working on an upstream driver for NVIDIA hardware, so an obvious question is what happens with nouveau in light of the NVIDIA announcement. Kernel graphics maintainer Dave Airlie gave a talk at the 2022 Linux Plumbers Conference (LPC) to help shed some light on that subject.

NVIDIA

He began by giving a brief history of NVIDIA hardware, with a timeline that can be seen in his slides. The timeline was in part "cobbled together from Wikipedia" and is not completely accurate, he said, but shows just "how far back NVIDIA hardware stretches". While the timeline starts in 1999, things started getting interesting in 2006 with the NV50, he said. It introduced the per-context virtual memory addresses; that feature represented a major turning point for graphics hardware.

There is roughly a two-year cadence to the NVIDIA releases starting in 2010 with "Fermi" (GF1xx). Vulkan support was added in the "Kepler" (GK1xx) hardware in 2012. In 2014, "Maxwell" came in two versions (GM1xx and GM2xx); the latter, also known as "Maxwell 2", introduced signed firmware. That pace more or less continued with "Pascal" (GP1xx) in 2016, "Volta" (GV1xx) in 2017, "Turing" (TU1xx) in 2018, and "Ampere" (GA1xx) in 2020. Turing brought support for the GPU system processor (GSP); he explained the importance of that feature a bit later in the talk.

Starting with Maxwell 2, NVIDIA decided that firmware for its devices could not simply be loaded unsigned, for security and other reasons. So firmware needed to be signed by NVIDIA and loaded into the multiple processors on the device. This made life hard for the nouveau project because it required complicated boot sequences for poking multiple firmware images into the device in a specific order that was "very hard to get right".

NVIDIA and nouveau had worked out an arrangement where NVIDIA would provide signed firmware, but it was still difficult to get any of the hardware working. Even when all of the right things were done at boot time, the devices came up in their base configuration. The devices were powered-on and functioning, but "you can't make it reclock, you can't make it go faster". Manually choosing a performance level for NVIDIA devices is known as "reclocking". There was also no power-management functionality available to the driver. This was a watershed moment for nouveau, Airlie said, because it did not make sense to put a lot of effort into a driver for graphics hardware running in its slowest possible mode, while not being battery-friendly either.

The GSP is a RISC-V-based processor that was added to the GPU for the Turing and later hardware. The GPU already had "six or seven little processors on it", but the GSP is meant to be "the one to rule them all". The firmware file for the GSP is around 30 or 40MB; most of the earlier firmware blobs were on the order 256KB, so the GSP is a substantial increase in size. But it is a single firmware image for the device that initializes the rest of the processors. Effectively, NVIDIA moved much of its proprietary kernel driver into the GSP.

That all happened around the same time as the announcement of the open-source NVIDIA kernel drivers, he said. Those are based on a fork of the NVIDIA proprietary driver that only interfaces directly to the GSP; it turns out that there is nothing all that interesting in the API between the kernel and the GSP, so it could be released as open source. Since NVIDIA has customers who are interested in open-source drivers, it makes sense for the company to do so. However, the drivers do not look or act like the existing kernel graphics drivers so they are not able to go into the upstream kernel, Airlie said.

nouveau

That is the current state in the NVIDIA world, which made for a good lead-in to talk about nouveau. That project started in around 2007 to reverse-engineer NVIDIA GPUs to create Linux drivers. It supports hardware from NV04 (1999) through Ampere "in various states of disrepair".

But the project has stagnated some recently due to various factors. One big problem that a community open-source graphics project faces is that once someone gets good at working on it, that becomes known, and they get hired away to work on some other graphics hardware. There is really only one full-time nouveau developer, Ben Skeggs at Red Hat, working on the project.

Also, once the signed firmware came about, with its lack of reclocking and power-management features, it was disheartening for the project; there was no way that the open-source driver was ever going to be able to compete with the proprietary one. It was hard to justify putting in a lot of effort into nouveau. Beyond that, Skeggs spent a lot of time just trying to get the firmware provided by NVIDIA to load and run the hardware.

For the most part, the nouveau kernel driver is just for hardware enablement at this point. The firmware that NVIDIA provides is not the same as what is used by the proprietary driver, so it is not well-tested. Only NVIDIA can really debug problems with that firmware, so there have to be multiple round-trips with NVIDIA engineers. More recently, though, the project has been adding GSP support because that provides a high-level interface to things like reclocking, so the hope is that the nouveau kernel driver can use the standard NVIDIA GSP firmware and drive the hardware that way; "we will see".

OpenGL and Vulkan

There is a nouveau OpenGL driver in Mesa. He believes it has passed the OpenGL 4.5 conformance tests, but has never been submitted for certification. Up until recently, it had "horribly broken multithreading context support" so it worked for older single-threaded games and the like but not for programs like Firefox or modern games; that has been fixed recently, though. The driver has not seen a lot of optimization work, however, due to the lack of reclocking support for the hardware.

A Vulkan driver for nouveau was recently started by Jason Ekstrand, with help from Karol Herbst and Airlie. At the time of the talk, that was a bit of news, but things have progressed since that time. The driver is targeting Vulkan 1.0 for hardware from Kepler up through Ampere and is passing lots of the conformance tests at this point. But in order to finish the driver, and make it work the way they want it to, there is a need to add new user-space APIs to the kernel.

There are three features needed to get Vulkan really working, he said. The first is to split the physical memory allocations (for buffer objects) from the GPU virtual memory allocations. In nouveau, that's all done in one step, which is fine for OpenGL but does not work for Vulkan where more control over the mappings is required.

The second is that synchronization objects and ways to handle and work with them need to be added so that the scheduler can wait for existing GPU work to complete before sending new tasks. It is the way to do proper interleaving of GPU work, Airlie said. The final piece is a virtual-memory-handling interface that is called VM_BIND; it is something that is being looked at for the Intel driver and the amdgpu driver already has many of pieces of it. It is an API both for virtual memory and for command submission that is also geared toward the needs of Vulkan.

Those are all non-trivial projects, he said. Once the GSP support is working and reclocking can be done, these are the next steps for nouveau, but they are going to take some time. The Vulkan driver developers have already started looking at that effort, but there are somewhat circular dependencies that make it difficult to see how to do the work incrementally. It will be a lot of code to review, so getting it into the upstream kernel in a piece-wise fashion will be challenging.

Future

There are some upcoming problems that have not yet been faced, he said. A 30 or 40MB firmware image is rather large; normally, those are put into the initramfs. But putting multiple initramfs images into the boot partition may overrun the space available. The problem gets worse because there may be a need to ship multiple NVIDIA firmware images due to a lack of a stable firmware ABI. The nouveau project will have to pick and choose which firmware versions to support, but each will need to be available; he has wondered if there might be a way to delay firmware loading until after the real root filesystem is mounted, but has not really worked that out yet.

In the long run, it may not really make sense to pound the NVIDIA firmware API into the nouveau driver, which has its own ideas about how everything works. A new driver that leaves behind the existing nouveau legacy and only talks to the GSP using NVIDIA's API may be the right path instead. In addition, the ability to reclock the hardware and accelerate the GPU may allow creating a cross-platform compute stack, to replace the vendor-specific solutions (e.g. CUDA) that exist today. All of those solutions are on their own island, lacking any real developer community, but maybe that could be changed; "we've done it for Vulkan, we've done it for OpenGL, I don't see why we can't do it", though it will take a lot of time—and likely a lot of money.

An audience member asked about Vulkan Compute as a possibility, but Airlie said it was not geared toward the same kinds of problems as CUDA and others. It is better than OpenGL Compute, but is still a long way from what the real compute stacks provide. Ekstrand echoed that, noting that while there is desire to see Vulkan Compute handle more of the "scientific" computing use cases, it will never be a full-stack solution; at most Vulkan can provide the run-time piece, he said.

There was some discussion of the problem with the size of the GSP firmware and initramfs that Airlie had described, including several suggestions of ways to approach the problem. The YouTube video of the talk is available for those who are interested in that discussion or more of the details elsewhere in the talk.

[I would like to thank LWN subscribers for supporting my travel to Dublin for Linux Plumbers Conference.]