All writing

~/writing/dlssg-open-ampere-proton

Systems debugging
14 min read

The Ampere kernels were in the DLL the whole time

NVIDIA's DLSS Frame Generation refuses RTX 30-series cards. Since the 310 runtime that refusal is policy, not hardware. On Linux every layer between the runtime and the GPU is open, so I fixed the layers instead of the DLL. One evening, eleven game launches, and the 3080 was generating frames.

Does DLSS Frame Generation need a 40-series card? NVIDIA says yes. When it shipped in 2022 the answer was honest: the frame interpolation leaned on the optical flow accelerator, a block Ada has and Ampere does not. Then DLSS 4 replaced the optical flow stage with a neural network, and the 310 runtime stopped needing that hardware. The 40-series requirement stayed. A Windows mod called dlssg_for_sm86 exists precisely because the requirement is now a check, not a dependency: it runs NVIDIA’s real 310 runtime on a 3080 and generates frames.

I wanted that in Microsoft Flight Simulator 2024, on Linux, on an RTX 3080. The mod does not work under Proton. It is also closed source, which surprised me, since its notices file cites a GPL licence and then ships nothing but binaries. So I looked at what it actually does, decided none of it was deep, and rebuilt the Linux version in the open. It took one evening. The interesting part is not that it worked. It is where the Ampere kernels turned out to be.

Disclaimer

This project was largely driven by Claude Code assistance. It wrote the extractor, the replacement CUDA DLL and the dxvk-nvapi patch.

Why the mod dies on Proton

The mod’s DLL embeds three PE images: a version.dll proxy with hooks on the NGX loader, an “SM86 backend”, and an unmodified copy of NVIDIA’s nvngx_dlssg.dll. Its own notes describe the trick as normalising the PTX ISA declaration in NVIDIA’s kernels, and its bundled runtime carries 406 PTX-only fatbins for sm_75 and sm_86. A text transform followed by a compiler, not reverse engineering.

The reason it fails on Proton is more basic. The backend loads nvcuda.dll and resolves cuInit by name. Proton’s nvcuda.dll is a 36 KB placeholder built from an empty spec file: one address slot in the export table, no name table, no cuInit string anywhere in the file. The lookup returns null, the mod logs “Missing cuInit”, and falls back to the game’s own frame generation path, which on a 3080 is nothing. Two people in the mod’s issue tracker had diagnosed exactly this. Nobody had answered it.

Is it a recompile or a reverse engineer?

Before building anything I wanted one fact: whether NVIDIA’s runtime carries PTX, which the driver can JIT for any SM at load time, or only SASS, the final machine code for one architecture. SASS only would mean disassembling and re-encoding instructions, a different project.

CUDA fatbins are easy to find in a file: a fixed magic, a header, then entries that each say PTX or ELF and which SM. A hundred lines of Python walks the MSFS runtime and lists them. The first pass reported 101 decompression failures out of 101 compressed PTX entries, and I spent a few minutes assuming NVIDIA used a non-standard LZ4 variant. Standard LZ4 was fine. The compressed-size field is at offset 16 in the entry header, not 20.

MSFS 2024 nvngx_dlssg.dll 310.6, fatbin survey
elf sm_89 raw:   31
ptx sm_89 lz4:   70
ptx sm_120 lz4:  31
decompress failures: 0

Every fatbin carries PTX. All of it declares .target sm_89 or sm_120. None of it uses FP8 instructions, which is the main thing Ada can do that Ampere cannot. So the question became whether the driver would accept the same PTX with the target line rewritten. That test is direct: load libcuda.so from Python, hand each fatbin to cuModuleLoadFatBinary, and count the results.

Driver JIT test, RTX 3080, driver 615.71
fatbins 70, sm_89 ptx entries with literal .target found 70
unpatched fatbin load results: {209: 70}
patched   fatbin load results (0=OK): {0: 70}

Error 209 is CUDA_ERROR_NO_BINARY_FOR_GPU. Untouched, the driver rejects all seventy. With the eleven-byte string .target sm_89 rewritten to .target sm_86 inside the LZ4 stream, and the entry’s arch field updated to match, all seventy compile and load. Seven minutes from “I wonder” to a proof that the kernel layer was mechanical. The rewrite works in place because the target line is the first thing in the PTX and LZ4 stores it as a literal run; nothing later in the block references it by offset.

That settled the design. On Linux the runtime does not talk to the GPU directly. It calls NVAPI, which under Proton is dxvk-nvapi, which forwards cubin uploads to vkd3d-proton, which uses VK_NVX_binary_import to hand them to the driver. That path already exists because it is how DLSS Frame Generation works on a 4090 under Proton today. Everything in it is open. So the rewrite lives in dxvk-nvapi, on the upload path, and NVIDIA’s signed DLL is never modified.

nvcuda, twice

The CUDA half looked solved before I started. SveSop maintains a CUDA driver relay for Wine, and it exports every function the runtime asks for. There are six: cuInit, cuDeviceGet, cuDeviceGetCount, cuDeviceGetName, cuDeviceComputeCapability and cuDeviceGetLuid. Device enumeration and an adapter identity; all the compute goes through NVAPI. The relay built in a minute and under the system Wine its test program reported cuInit success.

Under Proton’s own Wine it did not load at all, status c0000135 with the file in the right place. The relay is old-style winelib, a tiny PE shim over an nvcuda.dll.so on the Unix side, and Proton’s Wine 11 does not load those. A useful dead end, because it showed how little was needed: six calls, a name, two integers and a LUID. That does not need a CUDA context.

So the second nvcuda is a pure PE, 175 lines of C, no CUDA headers, no Unix half. It answers the six calls plus the context and attribute stubs NGX touches on the way, reports whatever compute capability the environment asks for, and gets the LUID by creating a Vulkan instance and reading VkPhysicalDeviceIDProperties. That is the same LUID DXGI hands the game, which is the point of the call: the runtime checks that the CUDA device and the D3D12 adapter are the same GPU.

nvcuda_lite.c, the calls that matter
#define API __declspec(dllexport) CUresult WINAPI
 
API cuInit(unsigned flags) { init_once(); g_inited = 1; return CUDA_SUCCESS; }
API cuDeviceGetCount(int *c) { init_once(); if (c) *c = 1; return CUDA_SUCCESS; }
API cuDeviceGet(CUdevice *d, int ordinal) {
    init_once(); if (ordinal != 0) return CUDA_ERROR_INVALID_DEVICE;
    if (d) *d = 0; return CUDA_SUCCESS;
}
API cuDeviceComputeCapability(int *major, int *minor, CUdevice dev) {
    init_once(); if (major) *major = g_cc_major; if (minor) *minor = g_cc_minor;
    return CUDA_SUCCESS;
}
API cuDeviceGetLuid(char *luid, unsigned *nodeMask, CUdevice dev) {
    init_once();
    if (!g_luid_valid) return CUDA_ERROR_NOT_SUPPORTED;
    if (luid) memcpy(luid, &g_luid, 8); if (nodeMask) *nodeMask = 1;
    return CUDA_SUCCESS;
}

It compiled to 22 KB, and Proton refused to load it too. This time the loader trace said why:

trace:module:get_load_order_value got standard key  for L"nvcuda"
warn:module:load_dll Failed to load module L"nvcuda.dll"; status=c0000135

An empty load-order key. Proton’s launch script sets dlloverrides["nvcuda"] = "b" whenever NVAPI is enabled: builtin only, never native. Wine decides what “builtin” means by looking for a sixteen-byte string in the DOS stub at offset 0x40, and my plain PE did not have it. Eleven lines of Python stamp it in.

stamp_builtin.py
e_lfanew = struct.unpack_from('<I', d, 0x3c)[0]
sig = b'Wine builtin DLL\0'
assert e_lfanew >= 0x40 + len(sig), 'no room in DOS stub'
d[0x40:0x40 + len(sig)] = sig
cuInit success!
NVIDIA GeForce RTX 3080 10GB (Low Hash Rate)
SM version: 8.9

A working nvcuda.dll for NGX on any Proton build, which is the file every Linux reporter in the mod’s issue thread was stuck behind.

The gate opens, then the runtime lies to itself

The test vehicle was Ghost of Tsushima, which has DLSS Frame Generation and no Microsoft sign-in. Its log from the previous run had the line I was after:

[NxReflex] DLSSG support result:Result::eErrorNoSupportedAdapterFound

With nvcuda-lite in place, the patched dxvk-nvapi installed, and three environment variables set, the same line read eOk. Frame Generation was selectable in the menu. And the retarget logged exactly one event, a failure: the PTX entry’s .target literal was not found.

The game ships DLSS-G runtime 3.7, and its PTX entries decompress to blocks that are 38 percent printable with no .version line. Encrypted, or at least not text. Only the 310 runtimes carry plaintext PTX, which is why the closed mod bundles its own 310 copy. MSFS ships 310.6, the files are drop-in for each other, and NGX validated the swapped DLL’s signature without complaint. Second launch.

retargeted: 27  untouched: 0
     27 Retargeted fatbin PTX entry sm_89 -> sm_86
      2 <-NvAPI_D3D12_CreateCubinComputeShaderEx: Invalid argument
      1 <-NvAPI_D3D12_CreateCubinComputeShaderExV2: Error

Twenty-seven fatbins retargeted and loaded, then driver faults inside libnvidia-glcore and a frozen game. Each error came with a second line: Cubin upload: not a fatbin (magic 1179403647). 1179403647 is 0x464C457F, an ELF header. These uploads were bare cubins, SASS only, no PTX, compiled for sm_89, and they were the neural network: k_conv_fp16_nhwc, k_pooling, k_upscale, k_element_wise. The frame-gen model itself, with no text to rewrite.

The runtime picks which build to upload, so the obvious move is to make it pick the Ampere one. Drop nvcuda-lite’s reported compute capability from 8.9 to 8.6, relaunch: twenty-five bare sm_89 uploads, twenty-five failures. The runtime chooses by NVAPI’s architecture ID, not CUDA’s, and logs m_gpuArch = 0x190, which is AD100. Fine: stop spoofing Ada through NVAPI. Relaunch: eErrorNoSupportedAdapterFound. The gate is on the same value.

I then added a per-caller override to dxvk-nvapi, resolving the return address to a module name so Streamline could see Ada while the runtime saw the real card. Streamline passed. The runtime asked for itself, saw Ampere, and refused.

The contradiction

The runtime gates on one architecture value and selects kernels by the same value. Report Ampere and it refuses to run. Report Ada and it uploads Ada machine code that Ampere cannot execute. No setting of that one value gets a 3080 through both checks. The fix has to be on the upload path, after the choice is made.

The twin

While those launches were failing I had a scan running over the raw ELF blobs in the 310.6 DLL, the ones outside any fatbin. There are 78 of them.

MSFS 310.6: ELF cubins total 109, inside fatbins 31, raw (outside fatbins) 78
 raw sm distribution: {86: 39, 89: 39}

Thirty-nine kernels for sm_89 and thirty-nine for sm_86, same names, side by side in the same file. NVIDIA had compiled the frame-gen network for Ampere, shipped it in every copy of the runtime, and never let an Ampere card select it.

So the patch’s second half does the selection the runtime refuses to do. When a bare cubin arrives whose SM is above the target, dxvk-nvapi reads the runtime DLL from disk, indexes every sm_86 cubin in it, and substitutes the twin. By kernel name alone the mapping is ambiguous: k_conv_fp16_nhwc appears sixteen times per architecture. Adding .text size and the constant banks gets 28 of 39 one-to-one and leaves eleven with identical sizes. The tiebreaker is the machine code. Ampere and Ada SASS are close enough that the upload and its true twin agree on almost every 16-byte instruction, while the wrong sibling scores a fraction lower.

aw050_sm89 best sm86 by SASS similarity: [(1.0, 'aw039_sm86'), (0.9926, 'aw044_sm86'), (0.9926, 'aw029_sm86')]
assigned 39 of 39, one-to-one: True

One wrinkle: none of the uploaded sizes matched any raw cubin in the DLL, because the runtime grows each cubin’s .nv.shared section before upload to the shared memory the launch needs. The substitution copies that size from the upload onto the twin, then hands the twin to the driver.

dxvk-nvapi patch, the substitution
for (auto& t : g_twins) {
    if (t.kernel != up.kernel) continue;
    bool strict = t.textSize == up.textSize && t.c2 == up.c2 && t.c0Size == up.c0Size;
    double s = elf::score(up, t) + (strict ? 1.0 : 0.0);
    if (s > bestScore) { bestScore = s; best = &t; }
}
elf::Cubin twin = *best;
uint64_t shared = elf::sharedSize(up);
if (shared) elf::setSharedSize(twin, shared);
return twin.data;

Ninth launch:

retargeted: 27  substituted: 39  no-twin: 0  failures: 0
Cubin twins: indexed 39 sm_86 cubins from ...\nvngx_dlssg.dll
Cubin substitute: 'k_conv_fp16_nhwc' sm_89 size 28576 -> sm_86 twin size 28576 (score 2, strict candidates 1, shared 12288)
faults: 12

Every kernel loaded. Every kernel launched. Twelve driver faults anyway.

The bug was mine

I had been blaming the faults on the failed network kernels for four launches. With zero failures the faults were still there, and the trace log handed me the real cause: Failed to find CuBIN in m_cubinSmemMap, defaulting to 0. The runtime was launching a shader handle dxvk-nvapi had never recorded.

The V2 create path takes a parameter struct and returns the new handle inside it. When my code substituted a cubin it built a local copy of the struct, pointed it at the replacement bytes, and passed the copy down to vkd3d-proton, which wrote the handle into the copy. The caller’s struct kept whatever garbage it started with, the runtime launched that, and the driver faulted. Every fault of the evening was this one line.

         auto result = m_vkd3dDevice->CreateCubinComputeShaderExV2(params);
+        if (retargeted)
+            callerParams->hShader = local.hShader;   // hand the created handle back to the caller

Eleventh launch. I clicked Play and watched the top-left of the screen, where NVIDIA’s own DLSS-G indicator overlay was enabled.

It works

Fatbins retargeted
27
PTX sm_89 to sm_86, JIT by the driver
Network kernels
39 of 39
swapped for NVIDIA's own sm_86 builds
Driver faults
12 0
after the handle fix
GPU
98%, 310 W
the 3080 doing frame-gen work

The overlay reported frame generation active, and the game ran with the 3080 flat out. The one artefact is a depth-of-field flicker in the pause menu, where the out-of-focus background trails the character, and the same artefact is reported for this game on Ada under Windows, so I am not chasing it. The runtime also calls a descriptor-object query eighteen times with null descriptors for surfaces the game does not provide, dxvk-nvapi rejects those, and nothing visible comes of it.

Time from “send it” to the overlay was sixty-five minutes and eleven launches of the game. Most of those launches were the contradiction section, which is the normal shape of this kind of work: the mechanism was right after the second launch, and the next six were working out the self-imposed kinks.

The tools

The repo is three small things: nvcuda-lite, the 22 KB nvcuda.dll installed over Proton’s stub; the dxvk-nvapi patch against a pinned upstream commit, with the retarget and the substitution behind DXVK_NVAPI_CUBIN_RETARGET_SM=86; and an installer that takes a Steam app ID, finds the library, Proton build and prefix from Steam’s own config, checks that the game’s runtime is 310.x, and prints the launch options. Those are the whole configuration:

DXVK_NVAPI_GPU_ARCH=AD100 NVCUDA_LITE_CC=8.9 DXVK_NVAPI_CUBIN_RETARGET_SM=86 %command%

Nothing from NVIDIA is redistributed. The runtime comes from the game you own, and the Ampere kernels are already inside it. Nothing from the closed mod was used. Turing should work with the two numbers set to 75 and 7.5, and I have not tried it.

Credits

jp7677’s dxvk-nvapi and HansKristian-Work’s vkd3d-proton already carried DLSS Frame Generation on Ada through VK_NVX_binary_import; this patch rides on that path. tB0nE and alperenalbay diagnosed the empty nvcuda.dll export table in the closed mod’s issue tracker, which is where the whole chain starts. SveSop’s nvcuda established that NGX could be served a CUDA relay under Wine at all. nvcuda-lite was written independently against the same API and covers only the calls NGX makes.

Code: github.com/dniminenn/dlssg-open, MIT, with a prebuilt release.

The point

Every piece of this was a check, not a capability. The CUDA DLL exists to answer six questions the runtime already knows the answers to. The architecture override tells one function to return a different integer. The PTX retarget changes eleven bytes of a string. The substitution picks a kernel that NVIDIA compiled, tested, and shipped in the same file, addressed to the exact GPU that was being told it could not have it.

None of it was hard to find. The upload path was upstream, the diagnosis was in an issue thread, the kernels were in the DLL. It was scattered across projects that had each stopped one layer short, and the only thing this evening added was walking the whole chain once with every layer open.

Ampere did not need Ada’s hardware. It needed Ada’s permission, and on Linux, permission is an environment variable.