Coder Legion - Recent posts tagged reverse-engineering

Reverse-Engineering NVIDIA: Changing a register operand

Fri, 25 Sep 2026 14:41:55 +0000

Motivation

In the previous article I have
started reverse-engineering a simple CUDA binary by modifying the value of
a pointer variable.

After identifying where the value was stored and successfully running the modified CUDA binary, it was time to find out what the corresponding assembly instructions actually looked like at the machine-code level.

Finding the immediate size

Now, to be able to determine the size and format of the instruction (or at
least the largest constant the instruction from the previous article can hold),
I started to increase the value word-by-word. Instead of simple 0xcc from the
last time, I did:

#include <stdio.h>

__global__ void test(int *a) {
    a[0] = 0xecc;
}

int main(int argc, const char* argv[]) {
    cudaError_t err;
    int *a;
    cudaMallocManaged(&a, sizeof(int));
    *a = 0x01;
    test<<<1, 1>>>(a);
    if ((err = cudaGetLastError()) != cudaSuccess)
        fprintf(stderr, "launch: %s\n", cudaGetErrorString(err));

    cudaDeviceSynchronize();
    printf("*a = %x\n", *a);

    return 0;
}

And yes, it was there:

$ xxd -e out/test-old.sm_50.cubin | grep ecc
00000690: ecc7f000 01000000 05070002 4c980780   ...............L

Now to try a bit larger value like '0xdecc':

$ xxd -e out/test-old.sm_50.cubin | grep decc

Searching for the complete 0xdecc pattern didn't find anything. Searching for
the original 0xecc did, however, and the surrounding bytes had changed:

$ xxd -e out/test-old.sm_50.cubin | grep ecc
00000690: ecc7f000 0100000d 05070002 4c980780   ...............L

And there it was. If we compare this line with the hex dump from above, we can
see that we have an extra d in 0100000d. So I tried increasing the word
size in xxd:

$ xxd -e -g 8 out/test-old.sm_50.cubin | grep decc
00000690: 0100000decc7f000 4c98078005070002   ...............L

Now the bytes looked much more like a structured 64-bit value rather than an
arbitrary sequence of bytes. The next thing to do was simply increase the test
value and see how much the apparent immediate field could hold.
The result with the largest value was:

$ xxd -e -g 8 out/test-old.sm_50.cubin | grep ffffffff
00000690: 010ffffffff7f000 4c98078005070002   ...............L

After that, it was impossible to fit more into an int value as nvcc
complained about truncation. I naturally tried to find out what will
happen if I use uint64_t instead of int:

$ xxd -e -g 8 out/test-old.sm_50.cubin | grep ffffffff -C1
00000680: 001fc400fe4007f1 010eeeeeeee7f004   ..@.............
00000690: 010ffffffff7f005 eedd200000070204   ............. ..
000006a0: 001f9c00fde007ef 50b0000000070f00   ...............P

So we discovered something. We now had two apparent MOVs and I wasn't sure
whether they differed because they targeted different registers or because they
loaded the high and low parts of a value.

Adding a static variable

After seeing how the MOV behaved while changing the value, I tried a little
experiment. I added one more variable with a lot of bytes:

#include <stdint.h>

__device__ uint64_t x = 0xdaec;

__global__ void test(uint64_t *a) {
        x = 0xccccccccdddddddd;
        a[0] = 0xaaaaaaaabbbbbbbb;
}

After looking for my usual patterns, I got:

$ xxd -e -g 8 out/test-old.sm_50.cubin | grep bbbb -B5 -A7
00000780: 001fc400fe2007f6 4c98078000870001   .. ............L
00000790: 010000000007f002 010000000007f003   ................
000007a0: 001fc000fe4007f1 010dddddddd7f008   ..@.............
000007b0: 010cccccccc7f009 4c98078005070004   ...............L
000007c0: 001fc400fc2007f1 eedd200000070208   .. .......... ..
000007d0: 4c98078005170005 010bbbbbbbb7f006   .......L........
000007e0: 001fbc00fe2007f2 010aaaaaaaa7f007   .. .............
000007f0: eedd200000070406 50b0000000070f00   ..... .........P
00000800: 001ffc00fc4007ef 50b0000000070f00   ..@............P
00000810: 50b0000000070f00 e30000000007000f   .......P........
00000820: 001f8000fc0007ff e2400fffff07000f   ..............@.
00000830: 50b0000000070f00 50b0000000070f00   .......P.......P
00000840: 000000000000daec 0000000000000000   ................

Looking at this we can filter some of these out. We can filter by patterns I
created. Also, I noticed that the line beginning with 0xeed appeared a lot
in my dumps. Here it appeared two times, so I filtered useful stuff out:

$ xxd -c 8 -e -g 8 out/test-old.sm_50.cubin | grep 'bbbb\|cccc\|aaaa\|dddd\|eedd'
000007a8: 010dddddddd7f008   ........
000007b0: 010cccccccc7f009   ........
000007c8: eedd200000070208   ..... ..
000007d8: 010bbbbbbbb7f006   ........
000007e8: 010aaaaaaaa7f007   ........
000007f0: eedd200000070406   ..... ..

The interesting part was that the final byte changed systematically as I introduced different values. This made it look increasingly likely that the final byte represented a register operand.

This is current test output:

$ ./test-old 
*a = aaaaaaaabbbbbbbb

Can we make it display the value of variable x? Let's try hacking 0xeed2
instructions. My hypothesis was that the low bits of these instructions encode
the register operand, similarly to what we observed in the 0x10 lines:

$ xxd -c 8 -e -g 8 test-old | grep eedd2
0008a848: eedd200000070208   ..... ..
0008a870: eedd200000070406   ..... ..

I used bvi to change the final byte of the second instruction from 06 to
08:

xxd -c 8 -e -g 8 test-old | grep eedd2
0008a848: eedd200000070208   ..... ..
0008a870: eedd200000070408   ..... ..

And (not) surprisingly, I got:

$ ./test-old
*a = ccccccccdddddddd

We actually switched registers from which the value for a[0] was loaded.

Conclusion

For now we can identify (and modify to a certain extent) two instructions:

The first instruction appears to be a MOV-like instruction, with a format that
currently looks like:

010VVVVVVVV7f00R

The VVVVVVVV appears to represent an immediate value, while R appears to
identify the destination register.

The second instruction appears to select a register containing the value that is
ultimately written to a[0]. Its encoding currently appears to be:

eedX200000070Y0R

where R appears to identify the register. X/Y remain unknown; I suspect one
of them encodes information about the operand size, since I observed different
values when experimenting with 32-bit int versus 64-bit uint64_t values.

Reverse-Engineering NVIDIA: Modifying a CUDA binary

Sun, 06 Sep 2026 18:41:28 +0000

Motivation

For quite a while I wanted to find out how NVIDIA really works under the hood.
One of the greatest unknowns for me was how GPU code is compiled, packaged and
loaded. So I accepted my own challenge: let's try and decode machine code
running on GPU.

Test code

I needed to start really simple and to get as much data as possible. My first
CUDA program to try and decipher was simply:

#include <stdio.h>

__global__ void test(int *a) {
        a[0] = 0x00;
}

int main(int argc, const char* argv[]) {
        cudaError_t err;
        int *a;
        cudaMallocManaged(&a, sizeof(int));
        *a = 0x01;
        test<<<1, 1>>>(a);
        if ((err = cudaGetLastError()) != cudaSuccess)
                fprintf(stderr, "launch: %s\n", cudaGetErrorString(err));

        cudaDeviceSynchronize();
        printf("*a = %x\n", *a);

        return 0;
}

I won't get into too many details here (I am not a CUDA expert), but essentially, the function marked with __global__ is the kernel that will run on the device. The cudaMallocManaged() function takes care of allocating memory that can be accessed by both the host and the device, saving us some additional memory-management code. The test<<<1, 1>>>(a) syntax launches the test kernel with one block containing one thread, while the CUDA runtime takes care of loading the kernel for execution on the GPU. Finally, cudaDeviceSynchronize() waits for the GPU to finish executing the kernel, ensuring that the result written to a is available when we access it from the host.

Build

I also created a reusable build command that will also give me something to
start with:

nvcc -arch=sm_50 -lineinfo --keep --keep-dir out -o test test.cu

Note: for this to work, the out directory needs to be created before running
nvcc compiler. Now I got a folder full of really useful files:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ ls out/
test.cpp1.ii    test.cudafe1.cpp     test_dlink.fatbin    test_dlink.reg.c        test.fatbin.c   test.ptx
test.cpp4.ii    test.cudafe1.gpu     test_dlink.fatbin.c  test_dlink.sm_50.cubin  test.module_id  test.sm_50.cubin
test.cudafe1.c  test.cudafe1.stub.c  test_dlink.o         test.fatbin             test.o

The test.sm_50.cubin file sounded like exactly what I was looking for:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ file out/test.sm_50.cubin 
out/test.sm_50.cubin: ELF 64-bit LSB executable, NVIDIA CUDA architecture,, statically linked, not stripped

First hack

Now I wanted to try changing the value of a in the test function and see how
cubin file will behave. I had no idea what I was dealing with, so I started
checking the file with random small values to get the one with least repetitions
in the CUDA binary. After some random attempts I realized that the best one is:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ hexdump out/test.sm_50.cubin | grep bc
0000680 07f1 fde0 bc00 001f 02ff 0007 2000 eedc
00008f0 0000 0000 0000 0000 04bc 0000 0000 0000

So I decided to change my test function to:

__global__ void test(int *a) {
        a[0] = 0xbc;
}

If everything went well, I would see an additional line here containing my bc.
The reason I started with really small values is because I had no idea what kind
of byte ordering I will encounter. After rebuilding the binary, I now got:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ hexdump out/test.sm_50.cubin | grep bc
0000310 f1bc 06d4 02fc 0000 0209 0000 0000 0000
0000690 f000 0bc7 0000 0100 0002 0507 0780 4c98
00006a0 07f2 fe20 bc00 001f 0003 0517 0780 4c98

Well, I definitely got a bit more different hex dump. So I tried just changing
it again, this time to 0xcc (just randomly) to see what I will get. And I got:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ hexdump out/test.sm_50.cubin | grep cc
0000690 f000 0cc7 0000 0100 0002 0507 0780 4c98

This was promising. Notice that this is the same as address 0x0000690 in the
previous run, except we got 0cc7 instead of 0bc7. Now I could try and find
this kind of pattern in the real binary:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ hexdump test | grep 0cc7
000cc70 528b 4830 e789 5c89 1024 c748 2444 0008
008a700 0001 0087 0780 4c98 f000 0cc7 0000 0100

We can see our value on address 0x008a700. Right now, if we run the binary,
we get:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ ./test 
*a = cc

Now, I used bvi to quickly find the address and changed 0cc7 to 0dd7.
So the result was:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ hexdump test | grep 0dd7
000dd70 01bb 0000 e900 ff76 ffff 0f66 441f 0000
008a700 0001 0087 0780 4c98 f000 0dd7 0000 0100

Well, hopefully, I didn't corrupt the binary. When I tried running it:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ ./test 
*a = dd

Bingo. We didn't recompile the CUDA source, regenerate the CUBIN, or even touch
the host executable's source code. We changed bytes inside the compiled binary,
and the GPU executed the modified instruction.

Final thoughts

We managed to modify code running on the GPU without recompiling it. More
importantly, we have found what appears to be an instruction field containing
our constant. Given what the kernel does, it is likely part of a mov-like
instruction, although we haven't decoded it yet.

So the next question is obvious: what is the real encoding of the instruction?