Coder Legion - Recent posts tagged gpu

Animating a 3D Gift Box with WebGPU

Sun, 04 Oct 2026 12:49:20 +0000

What is WebGPU, in one paragraph?
WebGPU is a modern browser API that gives JavaScript direct, low-overhead access to your graphics card.

It replaces WebGL, which was built mainly for drawing.
WebGPU is modeled on native APIs like Vulkan, Metal and Direct3D 12, and it adds compute shaders, so the GPU can run any parallel math, not just triangles. That’s why it matters for AI: neural networks are mostly giant matrix multiplications, which is exactly what GPUs are good at.

Learn more :-
WebGPU is a modern browser API

From Naive CUDA to Performance Engineering: My First GPU Matmul Journey

Tue, 29 Sep 2026 18:11:21 +0000

I have been diving deep into GPU architecture recently—balancing hands-on coding with studying Programming Massively Parallel Processors (PMPP)—and I am thrilled to share that I just earned my AMD Certified Associate credential along the way. To truly cement this architectural knowledge, I decided to stop just reading about kernels and actually build a complete GPU performance-engineering project from scratch. I chose matrix multiplication as the starting point because it exposes almost every critical aspect of GPU programming, from memory hierarchy to thread cooperation.

The journey began with a naive CUDA implementation. The basic idea was simple: assign one CUDA thread the responsibility of calculating exactly one element of the output matrix. Each thread determines its row and column based on its block and thread indices, and then performs the entire dot product independently. This gave me a functional, mathematically correct baseline. Having a baseline is extremely important because optimization is meaningless if you have nothing to compare your improvements against.

However, the naive implementation immediately highlights a fundamental GPU bottleneck: the memory wall. When several neighboring threads calculate their respective elements, they all need to fetch the same rows and columns from global memory repeatedly. Global memory has high latency and finite bandwidth, meaning these redundant data fetches cripple the kernel's actual compute potential. The solution, which aligns directly with the architectural concepts I have been studying, is to exploit data reuse.

To solve the memory bottleneck, I implemented a tiled matrix multiplication kernel using shared memory. Instead of every thread independently pulling data from global memory, threads within a block cooperate to load small portions—or "tiles"—of the input matrices into the SM's ultra-fast shared memory. They synchronize to ensure the tile is fully loaded, and then reuse those values for multiple calculations. I also made sure this wasn't just a toy implementation; I added proper boundary checks and zero-padding in shared memory so the kernel accurately processes arbitrary matrix dimensions (like 127 × 131) rather than just clean multiples of 16.

Writing the kernel was only half the project; measuring it accurately was the other. I refactored the codebase to cleanly separate the core CUDA kernels from the testing infrastructure. Relying on CPU wall-clock time is a terrible way to measure GPU performance, so I built a dedicated benchmark harness using CUDA events. The benchmark warms up the GPU, runs 100 iterations of the kernel, and records the GPU-side elapsed time to calculate average latency and GFLOP/s (floating-point operations per second). I integrated all of this into a strict GitHub engineering workflow—handling feature branches, pull requests, and correctness tests just like a production codebase.

The rigorous benchmarking paid off immediately. Testing a 1024 × 1024 × 1024 matrix on a Colab GPU, the naive kernel hit a latency of 4.660 ms (about 460 GFLOP/s). The tiled shared-memory kernel dropped that latency to 2.429 ms (884 GFLOP/s). That is a measured 1.92× speedup purely from optimizing how data moves through the GPU's memory hierarchy.

The most important shift in this project is that I am no longer asking, "Can I write a CUDA kernel?" I am now asking, "Why does this kernel perform this way, and what evidence tells me what to change?" I am intentionally avoiding the temptation to blindly throw warp shuffles or tensor cores at the code. My next steps are driven by data: sweeping matrix sizes from 512 to 4096 to understand scaling behavior, running tile-size experiments, and using profiling tools for roofline analysis. Once the foundational performance methodology is locked in, this pipeline will pave the way for building custom Softmax, LayerNorm, and eventually, a fused FlashAttention implementation.

What is GPU Passthrough?

Mon, 21 Sep 2026 14:40:10 +0000

GPU passthrough is a virtualisation method used to directly assign a physical graphics processing unit to a single VM, ultimately bypassing the virtualisation software layers like the NVIDIA Virtual GPU (vGPU) manager. In this configuration, the physical hardware is accessed exclusively by the driver running inside that specific VM and is not shared with other environments. While a single VM can be assigned multiple physical GPUs depending on hypervisor limits, individual physical GPUs cannot simultaneously host virtual GPUs and perform passthrough.

This architectural limitation creates challenges for local development setups. For instance, I was trying to run a simple Kubernetes cluster through Docker Desktop on my Apple Silicon Macbook and I had encountered a now obvious hardware limitation because macOS container runtimes operates inside a Linux VM via the Hypervisor framework. This setup cannot natively pass through the Apple Silicon GPU using Metal to containers. As a result, running even a simple, small qwen2.5:7b model relied entirely on CPU software inference, which easily maxed-out the processor cores and caused noticeable performance bottlenecks. Like, I was unable to perform even the most basic tasks with this local model.

To address hardware isolation and GPU utilisation constraints in production environments, tools like Edera allow direct NVIDIA and AMD GPU passthrough into isolated Edera zones. By configuring hardware identifiers in the protect daemon, binding devices to the appropriate drivers, and launching zones with dedicated kernel variants, users can run GPU-accelerated workloads and verify hardware access directly through utilities like nvidia-smi.

Bryan Steiner has a really useful tutorial for GPU passthrough, if this is interesting to readers:
https://github.com/bryansteiner/gpu-passthrough-tutorial

The KV Cache Formula That Should Set Your Context Limit

Mon, 07 Sep 2026 16:22:29 +0000

Phone prices are climbing this year largely because memory is scarce - AI data centers are buying DRAM faster than fabs can make it. That same squeeze shows up in your cloud bill, and it changes which knob you turn first when inference gets expensive.

Memory Is the Binding Constraint, Not Compute

When teams hit capacity limits serving a language model, the reflex is to ask for more GPUs. But on most serving setups the shortage isn't raw compute - it's the memory those chips carry. Model weights take a fixed slice. Everything left over goes to the KV cache: the running store of intermediate values for every token the model has already seen in a conversation. That cache grows linearly with context length and with the number of requests you serve at once.

So the real decision isn't "buy more GPUs or not." It's: do we cap context length and retrieve the relevant parts, or do we pay for the memory to keep stuffing everything in the prompt? Long-context models made the second option easy to reach for. Chipflation is making it expensive to keep reaching.

Running the Numbers Before You Ask for Budget

The formula is simple enough to put in a planning doc:

kv_bytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_value

# 7B-class model, GQA: 2 * 32 * 8 * 128 * 2 (fp16) = 131,072 bytes
# = 128 KB per token, per request
# 8k context -> 1.0 GB per request
# 32k context -> 4.0 GB per request

On an 80 GB accelerator with roughly 15 GB going to weights, you have ~65 GB for cache. At 32k context that's about 16 concurrent requests. At 8k, about 65. Same hardware, four times the throughput - purely from a context decision, no new chips required.

That's the trade-off stated honestly: you're spending retrieval quality to buy concurrency. The way to tell whether it's worth it is not vibes. Build a held-out set of 100 - 200 real user questions, answer each one twice - once with the full long context, once with an 8k window plus retrieval - and score them. If the quality gap is inside your tolerance, cap the window. If your users routinely need reasoning across a whole document, it isn't, and the extra memory is a legitimate line item.

There's a middle lever too: quantizing the KV cache to 8-bit halves the per-token cost with a smaller quality hit than truncation. Worth testing before you commit to either extreme.

Key Takeaways

Memory, not compute, usually sets your inference ceiling - and memory prices are rising for reasons outside your control.
Context length and batch size drive KV cache size linearly; a 4x context cut is roughly a 4x concurrency gain on the same hardware.
Decide with a scored comparison on real questions, not intuition - long context is a cost you should be able to justify.

If you ran that comparison on your own traffic, how much of your current context window is actually load-bearing?

Sources referenced: The Verge

Reverse-Engineering NVIDIA: Modifying a CUDA binary

Sun, 06 Sep 2026 18:41:28 +0000

Motivation

For quite a while I wanted to find out how NVIDIA really works under the hood.
One of the greatest unknowns for me was how GPU code is compiled, packaged and
loaded. So I accepted my own challenge: let's try and decode machine code
running on GPU.

Test code

I needed to start really simple and to get as much data as possible. My first
CUDA program to try and decipher was simply:

#include <stdio.h>

__global__ void test(int *a) {
        a[0] = 0x00;
}

int main(int argc, const char* argv[]) {
        cudaError_t err;
        int *a;
        cudaMallocManaged(&a, sizeof(int));
        *a = 0x01;
        test<<<1, 1>>>(a);
        if ((err = cudaGetLastError()) != cudaSuccess)
                fprintf(stderr, "launch: %s\n", cudaGetErrorString(err));

        cudaDeviceSynchronize();
        printf("*a = %x\n", *a);

        return 0;
}

I won't get into too many details here (I am not a CUDA expert), but essentially, the function marked with __global__ is the kernel that will run on the device. The cudaMallocManaged() function takes care of allocating memory that can be accessed by both the host and the device, saving us some additional memory-management code. The test<<<1, 1>>>(a) syntax launches the test kernel with one block containing one thread, while the CUDA runtime takes care of loading the kernel for execution on the GPU. Finally, cudaDeviceSynchronize() waits for the GPU to finish executing the kernel, ensuring that the result written to a is available when we access it from the host.

Build

I also created a reusable build command that will also give me something to
start with:

nvcc -arch=sm_50 -lineinfo --keep --keep-dir out -o test test.cu

Note: for this to work, the out directory needs to be created before running
nvcc compiler. Now I got a folder full of really useful files:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ ls out/
test.cpp1.ii    test.cudafe1.cpp     test_dlink.fatbin    test_dlink.reg.c        test.fatbin.c   test.ptx
test.cpp4.ii    test.cudafe1.gpu     test_dlink.fatbin.c  test_dlink.sm_50.cubin  test.module_id  test.sm_50.cubin
test.cudafe1.c  test.cudafe1.stub.c  test_dlink.o         test.fatbin             test.o

The test.sm_50.cubin file sounded like exactly what I was looking for:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ file out/test.sm_50.cubin 
out/test.sm_50.cubin: ELF 64-bit LSB executable, NVIDIA CUDA architecture,, statically linked, not stripped

First hack

Now I wanted to try changing the value of a in the test function and see how
cubin file will behave. I had no idea what I was dealing with, so I started
checking the file with random small values to get the one with least repetitions
in the CUDA binary. After some random attempts I realized that the best one is:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ hexdump out/test.sm_50.cubin | grep bc
0000680 07f1 fde0 bc00 001f 02ff 0007 2000 eedc
00008f0 0000 0000 0000 0000 04bc 0000 0000 0000

So I decided to change my test function to:

__global__ void test(int *a) {
        a[0] = 0xbc;
}

If everything went well, I would see an additional line here containing my bc.
The reason I started with really small values is because I had no idea what kind
of byte ordering I will encounter. After rebuilding the binary, I now got:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ hexdump out/test.sm_50.cubin | grep bc
0000310 f1bc 06d4 02fc 0000 0209 0000 0000 0000
0000690 f000 0bc7 0000 0100 0002 0507 0780 4c98
00006a0 07f2 fe20 bc00 001f 0003 0517 0780 4c98

Well, I definitely got a bit more different hex dump. So I tried just changing
it again, this time to 0xcc (just randomly) to see what I will get. And I got:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ hexdump out/test.sm_50.cubin | grep cc
0000690 f000 0cc7 0000 0100 0002 0507 0780 4c98

This was promising. Notice that this is the same as address 0x0000690 in the
previous run, except we got 0cc7 instead of 0bc7. Now I could try and find
this kind of pattern in the real binary:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ hexdump test | grep 0cc7
000cc70 528b 4830 e789 5c89 1024 c748 2444 0008
008a700 0001 0087 0780 4c98 f000 0cc7 0000 0100

We can see our value on address 0x008a700. Right now, if we run the binary,
we get:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ ./test 
*a = cc

Now, I used bvi to quickly find the address and changed 0cc7 to 0dd7.
So the result was:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ hexdump test | grep 0dd7
000dd70 01bb 0000 e900 ff76 ffff 0f66 441f 0000
008a700 0001 0087 0780 4c98 f000 0dd7 0000 0100

Well, hopefully, I didn't corrupt the binary. When I tried running it:

stjepan@stjepan-Aspire-F5-573G:~/Develop/NVIDIA$ ./test 
*a = dd

Bingo. We didn't recompile the CUDA source, regenerate the CUBIN, or even touch
the host executable's source code. We changed bytes inside the compiled binary,
and the GPU executed the modified instruction.

Final thoughts

We managed to modify code running on the GPU without recompiling it. More
importantly, we have found what appears to be an instruction field containing
our constant. Given what the kernel does, it is likely part of a mov-like
instruction, although we haven't decoded it yet.

So the next question is obvious: what is the real encoding of the instruction?

CUDA, ROCm, and the GPU Software Stack: Where Do You Even Start?

Tue, 01 Sep 2026 07:59:33 +0000

If you've worked with PyTorch, you've probably written something like model.cuda() and watched your model start running on the GPU. But what actually happened after that? How does a line of Python eventually turn into thousands of threads doing computation on a GPU? What exactly is CUDA? Where does ROCm fit in? And if you're someone who wants to go beyond simply using GPUs and actually understand them, where do you even start?I started asking myself these questions recently, and that slowly pulled me into the world of GPU programming. At first, I thought CUDA was basically just a way of writing code for NVIDIA GPUs. But it's much bigger than that. CUDA is NVIDIA's platform and programming model for general-purpose GPU computing. It gives you the tools and abstractions needed to write GPU kernels, manage memory, launch work, and interact with the hardware. Once you start writing CUDA, you stop thinking of a program as something that simply runs from top to bottom. Instead, you start thinking about how thousands of pieces of work can execute in parallel.

That leads to one of the first important things to understand about GPUs: a GPU isn't simply a faster CPU. CPUs are built around a relatively small number of powerful cores, while GPUs are designed to keep huge amounts of parallel work running at the same time. In NVIDIA's programming model, threads are organized into blocks, and threads execute in groups called warps. Those warps are scheduled onto the GPU's Streaming Multiprocessors, or SMs. You don't need to understand every hardware detail immediately, but once you start caring about performance, these concepts become difficult to ignore.

Then there's memory, which is probably where GPU programming really starts getting interesting. GPU memory isn't one giant pool where every access costs the same. You have registers, shared memory, caches, and global memory, all with different sizes and access characteristics. This means that an operation can be doing relatively little computation but still be slow because it keeps moving data back and forth between global memory and the compute units.

That changed the way I started looking at GPU optimization. Instead of only asking, "How can I make the GPU calculate this faster?", I started asking, "Why am I moving this data in the first place?" If you can load something once, reuse it several times, and avoid unnecessary trips to global memory, you can sometimes get a much larger performance improvement than simply trying to squeeze more computation out of the GPU.

So where does AMD fit into all of this? If CUDA is NVIDIA's GPU computing ecosystem, ROCm is AMD's corresponding open software stack for GPU computing. ROCm provides the compilers, runtimes, libraries, profilers, debugging tools, and other components needed to develop software for AMD GPUs. One of the main programming technologies within ROCm is HIP, which provides a C++ programming model that is deliberately similar to CUDA.

This is useful because the two ecosystems aren't completely unrelated worlds. CUDA code and HIP code share many of the same fundamental ideas: kernels, threads, blocks, memory management, synchronization, and parallel execution. AMD also provides tools such as HIPIFY that can help translate CUDA source code into HIP. That doesn't mean you can blindly convert any CUDA program and expect identical performance, because hardware-specific optimizations still matter, but it makes the ecosystems much closer than they might initially appear.And this is why I think it's useful to understand CUDA and ROCm as part of a larger GPU software stack, rather than treating them as isolated technologies. At the top you might have your ML application and PyTorch. Underneath that are GPU libraries and kernels. Those eventually interact with CUDA on NVIDIA hardware or ROCm/HIP on AMD hardware, which then interacts with the compiler and ultimately the GPU architecture itself.

For someone coming from ML, this is where things get interesting. Most of the time, you don't need to think about any of this. You write torch.matmul(), torch.softmax(), or torch.layer_norm(), and highly optimized implementations take care of the underlying work. That's exactly what frameworks are supposed to do. But eventually you might encounter an operation that's a bottleneck, a sequence of operations that could be fused, or a workload where the default implementation isn't ideal. That's when understanding what happens underneath the framework becomes extremely useful.

It also explains why technologies like Triton are becoming interesting. Triton gives you a higher-level way of writing custom GPU kernels without forcing you to manually deal with every individual thread in the same way CUDA does. You can think more in terms of blocks or program instances, while the compiler handles much of the mapping to the GPU. For me, this became a much more approachable way of experimenting with GPU kernels while still having to think about things like memory access, tiling, fusion, and data reuse.
And this is probably the biggest recommendation I'd give: don't just learn CUDA or ROCm by reading about them. Build something. I'm planning to approach my own learning through a GitHub repository where each kernel becomes an experiment. Start with a naive implementation, benchmark it, identify the bottleneck, optimize it, benchmark it again, and document what changed. Over time, that becomes much more valuable than simply having "CUDA" listed on a resume because you have actual evidence that you understand what the GPU is doing.
And I think that's ultimately why CUDA and ROCm are worth learning, especially if you're interested in ML systems. You start with something as simple as model.cuda(), but underneath that is an entire stack connecting Python frameworks, libraries, compilers, kernels, memory systems, and GPU hardware. Once you start peeling those layers back, things like FlashAttention, custom kernels, inference optimization, ML compilers, and GPU performance stop looking like completely separate topics.

You don't need to become a GPU expert before starting. Pick one kernel. Understand what it does. Write it. Break it. Benchmark it. Make it faster. Then move to the next one.

I Tried Getting Closer to the GPU With Triton

Sat, 29 Aug 2026 16:49:42 +0000

I've been trying to understand what actually happens when a piece of ML code reaches the GPU. Not the usual model.cuda() and GPU go brrrr kind of understanding, but the actual stuff underneath it — threads, warps, memory, kernel launches, and why some operations are ridiculously fast while others suddenly become slow.

That rabbit hole eventually led me to Triton.

At first, I was wondering why anyone would even bother writing custom GPU kernels when PyTorch already gives us highly optimized operations. But the more I looked into it, the more I realized that GPU performance isn't always about how much computation you're doing. A lot of the time, it's about how you're moving data around.

CUDA gives you an insane amount of control over the GPU, but that control comes with a lot of complexity. You have to think about threads, warps, synchronization, memory access, and a bunch of other low-level details. That's great if you're comfortable with CUDA, but if you're coming from Python and PyTorch, it can be a pretty steep jump.

PyTorch makes this much easier through its higher-level abstractions. You can just write something like torch.softmax() and let the framework handle everything underneath. The problem is that when you're executing operations eagerly, individual operations can result in separate kernel launches and repeated reads and writes to global GPU memory. The code is easy to write, but there can be a lot of unnecessary movement of data happening underneath.

And this is where Triton becomes interesting.

Triton is a Python-based language and compiler for writing GPU kernels. What I found interesting about it is that instead of forcing you to think about individual GPU threads, it lets you think in terms of blocks of data. You basically describe the work that one program instance should perform, and Triton takes care of launching many of those program instances across the GPU.

That changed the way I started thinking about GPU programming.

Before getting into Triton, I had to understand the GPU itself a little better. A GPU is built around massive parallelism. Instead of having a small number of powerful CPU cores, GPUs have huge numbers of lightweight threads that can execute work simultaneously. Those threads are organized into warps and blocks, and the actual hardware that executes these warps is organized into Streaming Multiprocessors.

But the part that really matters for performance is the memory hierarchy.

GPU memory isn't just one big pool where everything costs the same to access. You have registers, shared memory, caches, and global memory. Registers are extremely close to the computation and very fast, while global memory is much larger but significantly more expensive to access. So if your algorithm keeps loading data from global memory, performing a tiny amount of work, writing it back, and then loading the same data again, you're potentially wasting a huge amount of time.

The basic idea I started taking away was pretty simple: move data as little as possible and reuse it as much as possible.

My first Triton experiment was just vector addition. Nothing fancy. Given two arrays, add them together. But even this simple example helped me understand the programming model. Instead of launching a thread for every individual element, I could define a block size and let each Triton program instance handle a chunk of the input.

For example, if I have 1024 elements and a block size of 128, I can think of the workload as eight program instances. The first handles elements 0 to 127, the second handles 128 to 255, and so on. Triton gives each program instance an ID, and I can use that ID to calculate which section of memory it needs to work on.

The next thing that confused me initially was pointers. Triton kernels don't work with tensors in the same high-level way normal PyTorch code does. They work with pointers to GPU memory. You calculate offsets from those pointers, load the values you need, perform the computation, and then store the results back.

This is also where masks become important. If your input size isn't perfectly divisible by your block size, the final program instance might be assigned elements that don't actually exist. You obviously don't want the GPU trying to access memory outside the tensor, so you use a mask to make sure only valid elements are loaded and stored.

After vector addition, I moved on to something more interesting: softmax.

A straightforward softmax implementation involves several operations. You find the maximum value, subtract it for numerical stability, calculate the exponential, sum the results, and finally divide by the sum. Each of these operations can involve reading and writing data.

So conceptually, you can end up with something like: load data, calculate something, write it back, load it again, calculate something else, write it again, and repeat.

The data is constantly travelling between global memory and the compute units.

Instead of doing that, we can fuse the operations into one kernel. The idea is to load the data once, keep it on-chip while performing the different operations, and only write the final result back to global memory.

This is one of the things that made Triton click for me. The code isn't necessarily about doing more computation. It's about avoiding unnecessary memory traffic.

Then came matrix multiplication, which made the whole concept of tiling much more obvious.

When multiplying two matrices, a naive implementation can repeatedly load the same pieces of the input matrices from global memory. But if you're computing multiple nearby output values, you're often reusing the same data. So instead of calculating one output element at a time, you divide the matrices into smaller tiles.

You load a small tile of each input matrix into faster on-chip memory, perform as much computation as possible using those tiles, accumulate the result, and then move on to the next tile.

The fundamental idea is basically: load less often, reuse more.

That sounds ridiculously simple, but it is one of the core ideas behind high-performance matrix multiplication on GPUs.

Once I understood tiling, some of the other Triton concepts started making more sense too. Program IDs can be used to determine which output tile a program instance is responsible for. Strides let you calculate where elements actually live in memory. Masks handle boundaries. The accumulator keeps partial results while working through different chunks of the reduction dimension.

Then there is scheduling and cache locality.

If multiple output tiles reuse the same pieces of input data, it makes sense to execute those tiles close together so that the data has a better chance of remaining in cache. Triton provides mechanisms for grouping program instances to take advantage of this kind of locality.

And then you get to autotuning.

Choosing the perfect block size, number of warps, pipeline stages, and other parameters manually isn't always easy. Different workloads and different GPUs can behave differently. Triton's autotuning system lets you provide multiple configurations and benchmark them to find a better configuration for a particular workload.

This is another thing I found pretty interesting. Instead of assuming there is one perfect configuration, you can basically let the system search through several possibilities and keep the one that performs best.

Eventually I started benchmarking the kernels instead of just looking at the code and assuming the optimized version was faster. And that was probably another important lesson.

GPU optimization isn't about writing code that looks complicated.

It's about understanding what the hardware is actually spending time doing.

Sometimes you're compute-bound. Sometimes you're memory-bound. Sometimes kernel launch overhead matters. Sometimes you're repeatedly moving data that could have stayed on-chip. And sometimes the library implementation you're trying to beat is already so optimized that your custom kernel isn't going to magically win.

That's probably the biggest thing I took away from learning Triton.

Before this, I used to think about GPU optimization mostly as “How do I make the GPU perform more computation?”

Now I find myself asking a slightly different question:

“Why am I moving this data in the first place?”

That shift in perspective is probably more valuable than any individual Triton kernel I wrote.

I'm definitely not claiming to be a GPU programming expert after doing this. If anything, the more I learn, the more I realize how much more there is to understand — occupancy, register pressure, memory coalescing, Tensor Cores, persistent kernels, FlashAttention, compiler behavior, and a lot more.

But at least now those topics don't feel completely alien.

And that's honestly why I wanted to write this.

I didn't want this to be another post that throws a complicated Triton kernel at you and says, "look how fast this is." I wanted to understand why the kernel is written that way and what the GPU is actually doing underneath it.

Because once you start thinking about GPUs in terms of data movement, reuse, parallelism, and locality, GPU programming starts making a lot more sense.

My CUDA/GPU Journey: From "What Even Is a GPU?" to Actually Fascinated

Sun, 23 Aug 2026 17:43:11 +0000

A few months ago, if you'd asked me what a GPU actually does, I would've mumbled something about "graphics" and changed the subject. I'm an undergrad, self-taught in most of what I know about programming, and until recently, GPUs were just... background noise. Something gamers cared about. Something that showed up in specs sheets I didn't understand.

The moment it stopped being background noise

It started the way a lot of rabbit holes start: I kept seeing people online talking about RTX this, RTX that, benchmarks, VRAM, tensor cores — and it all went completely over my head. I'd nod along in threads I didn't understand, half-following conversations about why one card was "better" than another, quietly bothered that I had no idea what any of it meant.

At some point that turned into an actual itch. I don't like not knowing things, especially when it feels like everyone else is fluent in something I can't even parse. So I decided I had to figure out what a GPU actually was — not just as a gaming accessory, but as a piece of hardware people were clearly doing serious computational work with.

That's when I started noticing something else: people weren't just gaming on these things. I kept running into posts and videos of people running genuinely heavy workloads on GPUs — training models, running simulations, crunching numbers at speeds that made no sense to me coming from a CPU-only mental model. That was the real hook. Not the marketing, not the specs — the fact that people were using GPUs as general-purpose computing beasts, and I had zero idea how that was even possible.

Falling into CUDA (and discovering it's not the only player)

Once I started digging, I learned that "using a GPU for computation" isn't some universal, plug-and-play thing — it depends entirely on who made your hardware. NVIDIA has its own ecosystem called CUDA, and AMD has its own competing thing called ROCm. That distinction genuinely fascinated me. Two of the biggest hardware companies in the world, each with their own language and toolchain for talking to their own silicon.

A bit of digging confirmed what I'd suspected: CUDA is NVIDIA's proprietary platform, built specifically for NVIDIA GPUs, and it's been around since 2006 — which is a big part of why it has such a mature ecosystem of libraries, tools, and community support today. ROCm, on the other hand, is AMD's open-source answer, built around HIP (Heterogeneous-Compute Interface for Portability), which is designed to closely resemble CUDA's syntax so that code can be ported between the two with less pain. There's even a translation layer called HIPIFY that converts CUDA code into HIP so it can run on AMD hardware.

What struck me most was that this isn't just a technical footnote — it's basically a walled-garden situation. CUDA's maturity and dominance, especially in AI and deep learning workloads, is a big reason NVIDIA has had such a strong grip on the space. AMD is playing catch-up with an open-source strategy, hoping that openness eventually wins over raw maturity. As someone who'd never thought about hardware ecosystems as competing programming languages before, that was a genuinely new way of seeing the tech industry.

Where I'm at now

I'll be honest — I'm still very early. Right now my "journey" is less about writing optimized kernels and more about building the mental model: understanding what a thread, block, and grid even mean in CUDA terms, why memory transfer between CPU (host) and GPU (device) is such a big deal, and why "just throw it on the GPU" is a lot more nuanced than it sounds from the outside.

But that initial confusion — the feeling of being completely lost in a conversation about RTX cards — has turned into something I actually look forward to learning more about. I went from not knowing what a GPU was for, to being genuinely curious about parallel computing as a discipline. That's a bigger shift than I expected.

Why I'm writing this

I'm writing this partly for accountability, and partly because I know there are other self-taught devs out there who've felt that same "everyone else understands this but me" feeling. If that's you — you don't need a CS degree or a research lab to start. You just need the willingness to feel lost for a while before things click.

If you're also early in your GPU programming journey, or you've been down this road already and have advice, tools, or resources that helped things click for you, I'd genuinely love to hear about it in the comments. This is very much a "learning in public" post, not an expert one.

More updates to come as I actually start writing CUDA kernels instead of just reading about them.

Bypassing the VRAM bottleneck: practical local AI inference on restricted hardware

Tue, 09 Jun 2026 12:37:52 +0000

Google's Gemma-4 release changed the game for local AI development. The 26B Mixture of Experts model, combined with Unsloth's QAT quantization and llama.cpp's -cmoe memory-split flag, makes it possible to run a 26-billion parameter model on an everyday 8GB GPU laptop. No cloud API keys. No expensive hardware. Just your development machine running cutting-edge AI.

The promise is compelling. Run inference locally. Keep your data private. Eliminate API costs. Get fast feedback loops during development.

The problem is that running a 26B model on 8GB VRAM creates sustained thermal loads that your laptop's cooling system wasn't designed to handle. After 15-20 minutes of inference, your VRAM junction temperature hits 105°C and the firmware clamps your memory clocks. Your token rate drops from 20 t/s to 5 t/s. Your development velocity tanks.

This article walks you through the complete setup: building llama.cpp, downloading the model, configuring the -cmoe memory split, connecting to your Next.js 16 application, and managing VRAM thermals with VRAM Shield. By the end, you'll have a stable local AI pipeline running on your budget development laptop.

The `-cmoe` memory split explained

The -cmoe flag in llama.cpp is the key to running 26B models on 8GB VRAM. Here's the command:

./llama-cli -m "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" -cmoe -c 248000 -v

The flag splits the model across two memory domains. The attention mechanism and KV Cache stay in GPU VRAM for fast access. The expert weights are loaded from system RAM on-demand via PCIe.

Here's the data flow:

DATA FLOW: LOCAL AI INFERENCE
─────────────────────────────────────────────────────────────────
User Request (Next.js 16)
    │
    ▼
llama.cpp Server (localhost:8080)
    │
    ├──► System RAM (DDR5)
    │    │
    │    ├── Expert Weights (120/128)
    │    │   ~11.5 GB (loaded on-demand)
    │    │
    │    └── Router Network
    │        Expert selection logic
    │        ~200 MB
    │
    └──► GPU VRAM (8GB GDDR6X)
         │
         ├── Attention Layers
         │   Q, K, V, O projections
         │   ~1.2 GB (always resident)
         │
         ├── KV Cache
         │   64K token context
         │   ~800 MB (active)
         │
         └── Shared Experts
             8/128 experts (always active)
             ~720 MB
─────────────────────────────────────────────────────────────────

The key insight is the bandwidth asymmetry. System RAM provides approximately 16 GB/s of read bandwidth. GPU VRAM provides approximately 256 GB/s. The attention mechanism requires the higher bandwidth because it's compute-bound and latency-sensitive. The expert weights tolerate the lower bandwidth because they're memory-bound and the router network can prefetch the next expert while the current expert is being processed.

This split prevents Out-of-Memory errors. Without -cmoe, loading the entire 13.2GB model into 8GB VRAM is impossible. With -cmoe, only the attention mechanism and KV Cache reside in VRAM, consuming approximately 2.7GB. The expert weights remain in system RAM, consuming approximately 12.4GB. The total memory footprint exceeds 8GB, but it's split across two memory domains that operate independently.

The 8 shared experts (activated on every token) are loaded into VRAM at startup and remain resident. This eliminates the swap latency for the most frequently used expert weights. Only the 120 specialized experts are swapped on-demand via PCIe, with each swap completing in under 2 milliseconds.

The thermal saturation problem

The -cmoe memory split solves the capacity problem but creates a thermal problem. The attention mechanism and KV Cache in VRAM create sustained thermal loads on the memory modules.

During inference, the memory bus runs at 90%+ utilization. The GDDR6X chips generate heat faster than the laptop's cooling system can dissipate it. The VRAM junction temperature climbs steadily.

Here's the thermal timeline:

THERMAL PROFILE: 30-MINUTE INFERENCE SESSION
─────────────────────────────────────────────────────────────────
Time        Core Temp    VRAM Junction    Token Rate    Status
─────────────────────────────────────────────────────────────────
0:00        72°C         78°C             20 t/s        Stable
5:00        73°C         85°C             20 t/s        Stable
10:00       74°C         91°C             18 t/s        Creeping
15:00       75°C         98°C             12 t/s        Warning
20:00       75°C         105°C            5 t/s         THROTTLED
30:00       75°C         105°C            5 t/s         THROTTLED
─────────────────────────────────────────────────────────────────

After 15-20 minutes, the junction hits 105°C. The firmware clamps the memory clocks. Your token rate drops from 20 t/s to 5 t/s. The model still runs, but it feels like it's wading through molasses.

The debugging cost is significant too. When your inference session tanks, you spend time investigating. Is it a Python memory leak? A CUDA driver issue? A model loading problem? You chase phantom issues for hours before discovering the thermal explanation. That's time you could spend building features.

The problem is that your laptop's fan curves are designed for bursty workloads, not sustained inference. The fan speed is tied to CPU/GPU core temperature, which stays at 75°C. The VRAM junction temperature, which is climbing to 105°C, is invisible to standard monitoring tools. You're flying blind.

The thermal path is straightforward. The VRAM modules generate heat during sustained memory bus operations. The heat-pipes transport the thermal energy to the heatsink. The fans blow air across the heatsink to dissipate the heat. But the fans are running at moderate speed because the core temperature sensor shows 75°C. The VRAM junction, which is the actual point of failure, is hidden from the cooling system's control logic.

This creates a dangerous blind spot. Your monitoring tools show a healthy system. Your performance says otherwise. You spend hours chasing software ghosts because the hardware telemetry is telling you the wrong story. The debugging rabbit hole is always the same: you notice performance degradation, you check Task Manager, everything looks fine, you assume it's a software issue, you debug Python scripts and check for memory leaks, nothing fixes it, and eventually you stumble upon the thermal explanation.

Setting up the local AI pipeline

Here's the complete setup for running Gemma-4 26B on an 8GB laptop:

Step 1: Build llama.cpp

# Clone the repository
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Build with CUDA support
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j --target llama-cli llama-server

Step 2: Download the model

# Download Unsloth QAT GGUF (13.2GB)
curl -L -o models/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf \
  "https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/resolve/main/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf"

Step 3: Run the inference server

# Start llama.cpp server with -cmoe memory split
./build/bin/llama-server \
    -m "models/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf" \
    -cmoe \
    -c 64000 \
    --host 0.0.0.0 \
    --port 8080

Step 4: Connect to Next.js 16 with Vercel AI SDK

Create a Route Handler that proxies requests to the local llama.cpp server:

// app/api/chat/route.ts
import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';

const localLLM = openai({
  baseURL: 'http://127.0.0.1:8080/v1',
  apiKey: 'not-needed',
});

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: localLLM.chat('gemma-4-26B-A4B-it-qat-UD-Q4_K_XL'),
    messages,
  });

  return result.toDataStreamResponse();
}

Step 5: Test the connection

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-4-26B-A4B-it-qat-UD-Q4_K_XL",
    "messages": [
      {"role": "user", "content": "Explain the -cmoe flag"}
    ],
    "temperature": 0.7,
    "max_tokens": 500
  }'

Your local AI pipeline is now running. The llama.cpp server handles inference, the Next.js Route Handler proxies requests, and your application stays responsive.

The key configuration parameters are:

-cmoe: Activates the Mixture of Experts memory split
-c 64000: Sets the context window to 64K tokens (adjust based on your needs)
--host 0.0.0.0: Binds to all network interfaces (for local development)
--port 8080: Sets the server port (match this in your Next.js Route Handler)

For most development workloads, 32K-64K tokens is sufficient. Only use 248K if you genuinely need the full context for long document analysis or codebase-wide refactoring.

The performance characteristics are predictable. First token latency is around 200 milliseconds on cold start. Sustained throughput stays at 18-20 tokens per second with proper thermal management. The llama.cpp server handles one request at a time, so use a queue if you have multiple concurrent users.

Here's what the complete thermal profile looks like with and without VRAM Shield:

THERMAL PROFILE: 60-MINUTE INFERENCE SESSION
─────────────────────────────────────────────────────────────────
Time    Without VRAM Shield    With VRAM Shield (90% Duty Cycle)
─────────────────────────────────────────────────────────────────
0 min   78°C (stable)          78°C (stable)
10 min  91°C (climbing)        86°C (stable)
20 min  98°C (critical)        88°C (stable)
30 min  105°C (THROTTLED)      89°C (stable)
60 min  105°C (THROTTLED)      89°C (stable)
─────────────────────────────────────────────────────────────────

Without VRAM Shield, your performance tanks at 30 minutes. With VRAM Shield, you get sustained, stable performance for hours. The 10% duty cycle sacrifice is worth it.

The zero-installer safeguard

VRAM Shield solves the thermal problem without modifying firmware or requiring complex hardware modifications. It's a portable Windows utility that manages VRAM thermal loads at the process level.

Download VRAM Shield v2.2.2 from vramshield.com or install via WinGet:

winget install 53Software.VRAMShield

Launch it, configure your target temperature and duty cycle, and let it run in the background. VRAM Shield monitors VRAM junction temperature via NVML and introduces micro-pauses in the GPU compute stream when the temperature crosses your threshold.

The portable design is particularly useful for developers. You can keep VRAM Shield on a USB drive and run it on any Windows laptop. No installation. No system services. No reboot required.

For development workloads, we recommend:

Target VRAM Temperature: 95°C (gives headroom below the 105°C firmware threshold)
Duty Cycle: 90% (Pulse mode for sustained inference)
Panic Threshold: 108°C (emergency halt to prevent hardware damage)

The key insight is that thermal management for local AI isn't about reducing peak performance. It's about preventing the firmware from making a catastrophic intervention. Keep the VRAM junction below 105°C, and your inference session continues uninterrupted.

The 90% duty cycle sacrifices 10% of peak performance to prevent the 75% cliff. Your sustained throughput is 2.25x better over a one-hour session. For development workloads, this trade-off is absolutely worth it.

Join the local-first revolution

Google's Gemma-4 release makes running 26B models on an 8GB laptop a reality. The economics are compelling, the privacy benefits are real, and the development experience is fast.

But managing memory heat is critical. The thermal saturation penalty is real, and it can destroy your hardware if left unchecked. VRAM Shield is the essential utility to keep your development machine stable and protect your hardware's long-term health.

Learn more at vramshield.com.

Get started

Download VRAM Shield: vramshield.com
GitHub Repository: github.com/53-software/vram-shield
Install via WinGet: winget install 53Software.VRAMShield

The tools are open-source. The cost savings are real. Your local AI pipeline can handle sustained workloads. You just need the right thermal management.

Star the repository. Join the local-first AI revolution. Build your applications on your own hardware.

Snowflake integrates NVIDIA CUDA-X libraries so your Python ML code runs up to 200x faster on GPUs.

Tue, 18 Nov 2025 14:04:35 +0000

Snowflake Brings Native NVIDIA GPU Acceleration to ML Workflows—No Code Changes Required

Snowflake announced it's embedding NVIDIA's CUDA-X libraries directly into its ML platform. The integration enables data scientists to run GPU-accelerated machine learning workflows on Snowflake data without requiring any code modifications.

This matters because GPU acceleration has traditionally required developers to rewrite applications, manage complex infrastructure, or move data between systems. Snowflake is removing those barriers.

What's Actually Changing

Snowflake ML now comes preinstalled with NVIDIA cuML and cuDF libraries. These are part of NVIDIA's CUDA-X Data Science ecosystem, which includes open-source tools that enable popular Python frameworks to run on GPUs instead of CPUs.

The integration works with frameworks you're already using: scikit-learn, pandas, UMAP, and HDBSCAN. You don't need to learn new APIs or refactor your code. Your existing Python workflows simply run faster.

The libraries are available through Snowflake's Container Runtime, a pre-built environment for large-scale ML development. You can access them in Snowflake Notebooks or through ML Jobs for remote execution.

The Performance Numbers

NVIDIA's benchmarks show meaningful speedups on A10 GPUs compared to CPUs. Random Forest algorithms run approximately 5 times faster. HDBSCAN clustering can run up to 200x faster.

Those aren't theoretical numbers. They translate to real workflow improvements. Processing and clustering millions of product reviews, which previously took hours on CPUs, now takes minutes on GPUs. Genomics workflows that analyze high-dimensional gene sequences get similar acceleration.

The performance gain matters most when you're working with large datasets. As enterprise data volumes grow, CPU-only processing becomes a bottleneck. GPU acceleration helps maintain productivity without exponentially increasing infrastructure costs.

Why This Approach Works

The key advantage is eliminating the integration work. Most GPU acceleration projects require developers to:

Rewrite code to use GPU-specific APIs
Set up and manage GPU infrastructure
Move data between storage systems and compute environments
Debug compatibility issues between frameworks

Snowflake handles all of that. The CUDA-X libraries are already integrated and configured. Your data stays in Snowflake. You write standard Python code.

This removes a major barrier to GPU adoption. Many data science teams know their workloads would benefit from GPU acceleration, but can't justify the engineering effort required to implement it. Native integration changes that calculation.

Real Use Cases

Two examples show where this makes a practical difference:

Large-scale topic modeling: If you're processing customer feedback, support tickets, or social media data at scale, clustering and categorization workflows become computationally expensive on CPUs. GPU acceleration brings those workflows back to interactive speeds.

Computational genomics: Research teams analyzing genetic sequences deal with massive, high-dimensional datasets. Classification tasks like predicting gene families require significant compute power. The integration lets researchers focus on analysis rather than managing GPU infrastructure.

Both scenarios share a common pattern. The workflows are computationally intensive but don't require custom ML architectures. They use standard algorithms that benefit from parallelization. That's exactly where GPU acceleration provides the most value.

What to Consider

This integration works best when you're already using Snowflake for data storage and ML development. If your data lives elsewhere, you'll need to evaluate whether the performance gains justify moving it to Snowflake.

The integration currently supports specific libraries, including cuML and cuDF. If your workflows depend on other frameworks, you'll need to check compatibility. Snowflake and NVIDIA are continuing their partnership, so expect the list of supported libraries to expand.

You'll also need to understand GPU pricing in Snowflake's environment. While GPU acceleration reduces processing time, it comes with higher compute costs per hour. The value proposition depends on your specific workload patterns.

What This Means for Data Teams

The broader trend here is about reducing friction in ML development. Data scientists spend too much time on infrastructure and not enough time on modeling and analysis.

Native GPU integration is one piece of that puzzle. When acceleration works transparently, allowing you to write normal Python code and achieve GPU performance automatically, it removes a decision point from the development process.

That matters more as ML workflows become standard business tools rather than specialized projects. The easier it is to implement performance optimizations, the more teams can focus on solving actual business problems.

The integration is available now through Snowflake's Container Runtime. If you're running ML workloads on Snowflake, it's worth testing to see how your specific workflows benefit from it.

Revolutionary open storage cuts AI infrastructure costs 90% while boosting GPU performance by 12%.

Tue, 15 Jul 2025 21:00:22 +0000

Breaking Down the Walls: How Open Storage Architecture is Revolutionizing AI Infrastructure

The enterprise AI landscape is at a crossroads. As organizations grapple with soaring infrastructure costs and the complexities of managing massive datasets across hybrid environments, a fundamental question emerges: Should we continue building within proprietary walled gardens, or is it time to embrace open highways that promise greater efficiency, flexibility, and cost-effectiveness?

At Reuters MOMENTUM AI, Hammerspace CEO David Flynn delivered a compelling vision for the future of AI infrastructure, introducing revolutionary concepts that could fundamentally transform how enterprises approach data storage and management. His presentation, "The Great AI Crossroads: Open Highways vs. Walled Gardens," offered more than just technical insights, it presented a roadmap for breaking free from the constraints that have long plagued enterprise data architectures.

The Problem with Proprietary Bottlenecks

Flynn's central thesis challenges a fundamental assumption in enterprise storage: that accessing network data requires going through the network. This seemingly obvious principle has created layers of proprietary systems that act as traffic lights and stop signs in the data highway, forcing information to traverse up to 10 different chip-to-chip transfers before reaching GPU processors.

"Every single time you have to touch the data and send it from one chip to another, you're just burning extra power," Flynn explained. "And it turns out the main reason you're doing that today is to give people a place to put proprietary IP, it just slows it all down."

This inefficiency becomes particularly problematic when training large language models with billions of parameters. The traditional approach not only wastes computational resources, but also dramatically increases power consumption and hardware costs. For enterprises investing millions in GPU infrastructure, these bottlenecks represent a significant drain on both performance and budget.

The Linux-Native Solution

Hammerspace's approach leverages a fundamental shift in the computing landscape: the consolidation around Linux as the dominant operating system for AI workloads. With their CTO serving as the kernel maintainer for the network file system portion of Linux, the company has worked directly with the Linux kernel community to enhance the operating system's native capabilities.

The result is what Flynn calls "Tier Zero" storage—a revolutionary approach that positions data directly within GPU servers, eliminating the need for separate storage arrays entirely. This isn't just about incremental improvements; it's about fundamentally reimagining how data flows through AI infrastructure.

"We can literally remove the proprietary IP out of the data path and have bulk flash serving directly to massive arrays of GPUs, without the need for any IP in the middle or the servers to host it," Flynn noted. This approach reduces data traversals from 10 hops to just 3, dramatically improving efficiency while reducing power consumption.

Unlocking Stranded Capacity

One of the most compelling aspects of this architecture is its ability to unlock what Flynn calls "stranded capacity"—the massive amounts of flash storage that already exist within GPU servers but remain underutilized. Modern DGX and HGX nodes come equipped with 60 terabytes of flash storage each, yet this capacity often sits idle or serves only as isolated silos.

By aggregating this existing storage through parallel NFS and a separate control plane, organizations can transform these isolated resources into shared storage infrastructure. This eliminates the need for separate storage arrays while maximizing the value of existing hardware investments.

"You don't need to load any software. You simply add it to the control plane, which can aggregate it and have it become shared storage," Flynn explained. This approach turns what was previously a sunk cost into a productive asset, improving both utilization rates and return on investment.

The Open Flash Platform: Density Revolution

Perhaps the most ambitious aspect of Hammerspace's vision is the Open Flash Platform—an initiative that promises to deliver 50 times the capacity density of current shared storage systems using existing flash technology. The platform aims to pack an exabyte of usable capacity into a single data center rack, a density that would have been unimaginable just a few years ago.

This dramatic improvement in density comes from eliminating the servers and proprietary IP that typically sit between flash devices and the applications that need them. By using smart NICs and embedded Linux systems, the platform can serve data directly from flash arrays without the overhead of traditional storage controllers.

The implications extend beyond just space efficiency. Flynn highlighted three key benefits: 30% reduction in upfront hardware costs, 90% reduction in ongoing power consumption, and extended hardware lifecycles of eight years instead of the typical five. This last point is particularly important, as organizations often discard functional flash storage simply because the surrounding server infrastructure has reached end-of-life.

Simplifying AI Deployment

Beyond the technical advantages, Flynn emphasized how this architecture addresses one of the most significant challenges in AI deployment: complexity. Traditional storage systems require extensive setup, configuration, and ongoing management. The new approach leverages Linux's native capabilities to dramatically simplify deployment.

"By using something that's standards-based and native to Linux, it's much easier to get it running. In some cases, you already have it there," Flynn noted. This reduction in complexity translates directly to faster time-to-value for AI projects, allowing organizations to focus on developing and deploying models rather than wrestling with infrastructure.

The Migration Challenge Solved

One of the most practical advantages of this approach is its ability to work with existing infrastructure. Rather than requiring expensive and risky forklift upgrades, the architecture can incorporate and accelerate existing shared storage systems while adding new capabilities.

This hybrid approach allows data to move seamlessly between traditional storage, new high-density flash arrays, and GPU-local storage. Organizations can begin realizing benefits immediately while gradually expanding their use of the new architecture. As Flynn put it, "Never do another data migration again," a promise that resonates with any IT professional who has managed large-scale data movements.

Looking Forward: The Open Highway

The vision Flynn presented represents more than just technical innovation; it's a fundamental shift toward open, standards-based infrastructure that can adapt and evolve with changing requirements. By removing proprietary bottlenecks and embracing Linux-native approaches, organizations can build AI infrastructure that's not only more efficient but also more flexible and future-proof.

As enterprises continue to grapple with the challenges of AI deployment at scale, the choice between walled gardens and open highways becomes increasingly critical. The technical capabilities exist today to break free from proprietary constraints and build more efficient, cost-effective AI infrastructure. The question isn't whether this transformation will happen, but how quickly organizations will embrace it.

For those ready to explore this new frontier, the Open Flash Platform represents a concrete step toward a more open, efficient future for AI infrastructure. The highway is open, it's time to start driving.

Comparing GPU and CPU Processing Power for Regular Computer Tasks

Mon, 23 Jun 2025 05:37:24 +0000

The parallel processing power of a Graphics Processing Unit (GPU) is designed to handle massive amounts of data in parallel, making it particularly well-suited for tasks like graphics rendering, scientific simulations, and machine learning. In contrast, the serial processing power of a Central Processing Unit (CPU) is optimized for executing a sequence of instructions as quickly as possible, making it better suited for tasks that require a lot of sequential processing, such as running an operating system, executing most applications, and handling system management tasks.

Key Differences Between GPUs and CPUs

Architecture: GPUs are designed with a large number of cores (often in the hundreds or thousands) that can handle many threads concurrently. This architecture is known as massively parallel processing. CPUs, on the other hand, have fewer cores (typically between 2 and 64 on consumer-grade processors) but are optimized for high clock speeds and efficient execution of serial instructions. The cores in a CPU are also more complex, supporting out-of-order execution, branch prediction, and other features that enhance serial performance.
Processing Type: GPUs excel at parallel processing, where a task is broken down into many smaller sub-tasks that can be executed simultaneously. This is particularly useful for tasks that involve large datasets and can be parallelized. CPUs are better at serial processing, where tasks are executed one after another. This is essential for applications that require a lot of sequential logic.
Task Suitability: GPUs are particularly good at tasks that can be parallelized, such as matrix operations, image and video processing, and certain types of scientific computing. They are also crucial for graphics rendering in games and professional graphics applications. CPUs are generally better for tasks that are inherently serial or require rapid context switching between different tasks. Examples include running operating systems, executing most business applications, and handling system management tasks.

Performance Comparison for Regular Computer Tasks

For most "regular computer tasks" such as web browsing, office applications, and even many games (at the CPU level, not graphics rendering), CPUs are more than sufficient and often perform better due to their higher clock speeds and better support for serial instruction execution. However, certain tasks within these applications can benefit from GPU acceleration, such as video decoding, graphics rendering, and some compute tasks offloaded by applications.

Examples of Tasks and Their Processing Suitability:

Web Browsing: Primarily CPU-bound for page rendering and JavaScript execution, but can use GPU for certain graphics tasks. Modern web browsers can offload tasks like graphics rendering to the GPU, improving performance.
Office Applications: Mostly CPU-bound for document processing and calculations. While GPUs can accelerate certain tasks, most office work doesn't require the kind of parallel processing GPUs excel at.
Gaming: GPU is crucial for graphics rendering, which is a highly parallel task. However, the CPU handles game logic, physics, and other serial tasks. A balanced system with both a good CPU and GPU is ideal for gaming.

Practical Implications of GPU and CPU Performance

Most Users: For general users who primarily engage in web browsing, office work, and similar tasks, the CPU is the more critical component. However, a decent GPU can enhance the experience in tasks that are GPU-accelerated, such as video playback and casual gaming.
Gamers and Graphics Professionals: A powerful GPU is essential for smooth gaming and professional graphics work. The parallel processing power of the GPU is indispensable for rendering complex graphics in real-time.
Specialized Tasks: For tasks like video editing, 3D modeling, and certain types of scientific computing, both CPU and GPU can be critical, depending on how well the specific application is optimized for parallel processing. Some applications are designed to utilize both CPU and GPU resources effectively.

Technical Details and Optimization

The performance difference between GPUs and CPUs for specific tasks is also influenced by how well the task is optimized for the respective hardware. For instance, applications that are optimized for GPU acceleration can significantly outperform CPU-only implementations for certain tasks. This is because GPUs can perform many operations in parallel, reducing the overall processing time.

GPU Acceleration Examples:

Video Encoding/Decoding: Many modern video codecs are optimized for GPU acceleration. GPUs can encode and decode videos much faster than CPUs, thanks to their parallel processing capabilities.
Machine Learning: GPUs are widely used in machine learning for training models. The parallel nature of matrix operations involved in deep learning makes GPUs particularly well-suited for this task.
Scientific Computing: Certain scientific simulations, such as weather forecasting and molecular dynamics, can be significantly accelerated using GPUs.

Example Performance Comparison

To illustrate the difference, consider a simple task like video encoding. A high-end CPU might be able to encode a 4K video in 2 hours, while a high-end GPU can do the same task in 30 minutes, thanks to its massively parallel architecture. This significant performance difference highlights the importance of using the right hardware for the task at hand.

Choosing the Right Hardware for Your Needs

When selecting a computer or upgrading components, understanding the role of both the CPU and GPU is crucial. For most users, a balanced system that includes both a capable CPU and a decent GPU will provide the best overall performance. However, specific needs may dictate a different balance:

For Gamers: A high-performance GPU is crucial, but a fast CPU is also necessary for handling game logic and physics.
For Professionals: Depending on the specific tasks, either a high-end CPU, GPU, or both may be required. For example, video editors and 3D modelers benefit from powerful GPUs, while software developers may prioritize CPU performance.

Future Developments

The landscape of CPU and GPU performance is continually evolving. Advances in technology are leading to more efficient CPUs and more powerful GPUs. Additionally, the lines between CPU and GPU are blurring with the development of heterogeneous computing systems, where tasks can be dynamically allocated to the most suitable processor.

Conclusion

In summary, while GPUs have significantly more raw processing power than CPUs for certain types of tasks, for "regular computer tasks," the CPU remains the primary processor. The GPU's parallel processing power is a valuable asset for specific tasks and applications that can leverage it, enhancing overall system performance and user experience. Understanding the strengths and weaknesses of both CPUs and GPUs is essential for making informed decisions when it comes to computer hardware.

Additional Considerations

When considering upgrading or purchasing a new computer, several factors should be taken into account to ensure that the system meets your specific needs.

Usage Patterns: Understanding how you intend to use your computer is crucial. Different tasks have different requirements.
Budget: Balancing performance with budget is essential. Identify the components that are most critical for your needs and allocate your budget accordingly.
Future Proofing: Technology is constantly evolving. Consider the potential for future upgrades and the compatibility of new components with your existing system.

Summary of Key Points

GPUs excel in parallel processing tasks such as graphics rendering and certain scientific computations.
CPUs are better suited for serial tasks and overall system management.
A balanced system with both a capable CPU and a decent GPU is ideal for most users.
Specific needs, such as gaming or professional graphics work, may require a more powerful GPU.
Understanding the strengths and weaknesses of both CPUs and GPUs is crucial for making informed hardware decisions.

Graid's SupremeRAID uses GPU acceleration to eliminate storage bottlenecks, delivering 28M IOPS for developers.

Sun, 08 Jun 2025 00:01:50 +0000

Graid Technology Revolutionizes Storage Performance with GPU-Accelerated RAID for Developers

The storage landscape for high-performance computing and AI development is undergoing a fundamental transformation. Traditional CPU-based RAID solutions have long created bottlenecks that limit application performance, forcing developers to architect around storage constraints rather than focusing on optimal code design. Graid Technology's SupremeRAID™ represents a paradigm shift that developers have been waiting for—GPU-accelerated RAID that eliminates these performance limitations.

Breaking Free from CPU-Based Storage Constraints

For decades, developers working on data-intensive applications have faced a fundamental challenge: storage performance that couldn't keep pace with computational demands. Traditional RAID controllers consume valuable CPU cycles and create throughput bottlenecks that limit the effectiveness of multi-threaded applications, real-time analytics systems, and AI training workloads.
SupremeRAID™ addresses this limitation by offloading RAID operations entirely to dedicated GPU hardware. This "out-of-path" architecture means storage operations no longer compete with application logic for CPU resources, fundamentally changing how developers can approach performance-critical applications.

The results speak for themselves. Graid's flagship solution achieves up to 28 million IOPS and 260GB/s throughput with a single card supporting up to 32 native NVMe drives. Perhaps more importantly for developers, the company reports achieving over 95% of raw NVMe performance in production environments—performance that was previously impossible with traditional RAID solutions.

Game-Changing Performance for AI Development

AI and machine learning developers face unique storage challenges. Training large models requires massive datasets that must be continuously fed to GPUs at tremendous speeds. Any storage bottleneck directly translates to expensive GPU idle time and extended training cycles.

SupremeRAID AE (AI Edition) specifically addresses these challenges with features such as GPUDirect Storage (GDS) support and an Intelligent Data Offload Engine. GDS enables direct NVMe-to-GPU memory transfers, eliminating CPU bottlenecks entirely. The Intelligent Data Offload Engine can actually put the RAID kernel to sleep during intensive GPU computation phases, freeing up additional GPU resources for AI workloads.

For developers building AI training pipelines, this means they can finally design systems where storage performance matches computational capability. No more architectural compromises to work around storage limitations—just pure performance that scales with your data requirements.

Simplifying Development with Software-Defined Architecture

Traditional RAID controllers require developers to understand complex hardware configurations and work within rigid performance constraints. SupremeRAID™'s software-defined approach provides flexibility that hardware solutions cannot match.

The platform supports seamless scaling across multiple GPU servers through features like NVMe-over-Fabrics (NVMe-oF). Developers can design distributed applications that treat storage as a unified resource pool rather than managing individual server limitations. A single SupremeRAID™ installation can provide RAID protection across multiple servers, dramatically simplifying application architecture.

Integration with existing developer workflows is straightforward. The solution includes RESTful APIs for modern IT automation and supports popular clustering storage systems like BeeGFS, Lustre, and Ceph without requiring data migration.

Real-World Developer Impact

The practical implications for developers are substantial. Consider real-time analytics applications that previously required careful buffering strategies to accommodate storage latency. With SupremeRAID™, developers can design more responsive systems that process data streams without artificial delays.

Game development studios leveraging real-time asset streaming can eliminate loading bottlenecks that previously required complex content management strategies. Scientific computing applications can process larger datasets in memory without the traditional tradeoffs between storage speed and data protection.

Financial trading systems greatly benefit from ultra-low latency capabilities. TelSwitch, a legal data analytics company, has deployed SupremeRAID™ across multiple systems specifically for processing massive datasets with small block sizes, precisely the kind of workload that traditional RAID struggles with.

Enterprise-Grade Reliability for Mission-Critical Code

Despite its performance focus, SupremeRAID™ maintains enterprise-grade data protection features that developers can rely on. The solution supports RAID 6 configurations, protecting against multiple drive failures while maintaining RAID 10-level performance characteristics.

Advanced features, such as journaling, bad block detection, and extended retry mechanisms, enable developers to architect systems with confidence in data integrity. The US Department of Defense has deployed SupremeRAID for edge computing applications that require "military-grade" reliability standards.

The Future of High-Performance Development

NVIDIA CEO Jensen Huang's recent statement, "for the very first time, your storage system will be GPU-accelerated," validates what Graid Technology has been building. As AI workloads become increasingly central to application development, storage architectures that match GPU performance capabilities become essential.

For developers building the next generation of data-intensive applications—whether AI training systems, real-time analytics platforms, or high-performance scientific computing solutions—SupremeRAID™ represents more than just faster storage. It's the foundation for applications that were previously impossible to build efficiently.

The era of designing around storage limitations is coming to an end. With GPU-accelerated RAID, developers can finally focus on building better software rather than working around infrastructure constraints.

Debugging Shaders on macOS: Instruments vs. Gpumkat

Sat, 29 Mar 2025 20:34:03 +0000

Debugging Shaders on macOS: Instruments vs. Gpumkat

Debugging and optimizing Metal shaders on macOS has long been reliant on Apple's Instruments tool. However, a new alternative, gpumkat, has emerged, offering a different approach with enhanced debugging features. In this article, we will compare Instruments and gpumkat, highlighting their differences, strengths, and why gpumkat may be the superior choice for many developers.

Instruments: The Traditional Tool

Features:

Provides detailed profiling of GPU activity
Tracks memory usage and performance metrics
Offers GPU counters for in-depth analysis
Integrated with Xcode
Best suited for hardware-specific insights

Limitations:

No built-in shader debugging tools
Lacks robust breakpoint and step-through debugging for Metal shaders
Requires deep knowledge of GPU internals to interpret data
Limited support for timeline-based debugging and trace analysis

Introducing Gpumkat

Gpumkat is a GPU kernel analysis tool designed for Metal API debugging and performance analysis. Unlike Instruments, which is focused on hardware profiling, gpumkat offers a broader range of tools specifically for shader debugging and optimization.

Key Features:

Performance Analysis: Tracks execution time, cache hit rates, and memory usage.
Shader Debugging: Step-by-step debugging, variable printing, and breakpoints.
Energy Consumption Insights: Estimates GPU power draw and energy efficiency.
Interface Metrics: Monitors how buffers and pipelines interact.
GPU Software States: Tracks command execution flow.
Stack Tracing & Tracing Timelines: Logs and visualizes execution sequences.
Error Handling: Captures warnings, memory errors, and validation issues.
Low-End GPU Simulation: Tests shader performance under constrained hardware conditions.

Why Gpumkat is the Better Alternative

1. Superior Debugging Tools

While Instruments provides GPU counters and profiling, it lacks a dedicated shader debugger. Gpumkat allows developers to set breakpoints, inspect variables, and step through shader execution, making it far more effective for diagnosing shader issues.

2. Comprehensive Performance Analysis

Gpumkat offers in-depth insights into cache hit rates, memory bandwidth, and processing unit availability—data that is difficult to extract from Instruments alone.

3. Better Error Handling

With built-in error detection for memory leaks, shader validation failures, and runtime warnings, gpumkat prevents common pitfalls that are harder to catch using Instruments.

4. Simulated Low-End GPU Testing

Gpumkat allows developers to simulate lower-end GPUs to test how shaders perform under constrained conditions. This feature is crucial for ensuring compatibility with a range of hardware, something Instruments does not provide.

5. Enhanced Trace Logging and Timeline Recording

While Instruments provides time-based tracking, gpumkat offers a more detailed execution timeline, helping developers pinpoint slowdowns and inefficiencies in their shaders.

When to Use Instruments vs. Gpumkat

Use Case	Instruments	Gpumkat
Hardware-Specific Profiling	✅	❌
Shader Debugging	❌	✅
Breakpoints & Step Debugging	❌	✅
Memory & Cache Analysis	✅	✅
Error Detection	❌	✅
Low-End GPU Simulation	❌	✅
Timeline & Trace Recording	✅ (basic)	✅ (detailed)

Installing Gpumkat

Gpumkat is easy to install and requires minimal dependencies:

curl -L -o gpumkat.tar.gz https://github.com/MetalLikeCuda/gpumkat/releases/download/%s/gpumkat.tar.gz \
    && tar -xvzf gpumkat.tar.gz \
    && cd gpumkat \
    && sudo sh install.sh

Replace %s with the latest version number.

Conclusion

For developers focused on deep GPU performance profiling, Instruments remains the go-to tool. However, if you need shader debugging, error handling, and simulated performance testing, gpumkat is the clear winner. Its comprehensive feature set makes it an invaluable tool for Metal developers looking to optimize their shaders efficiently.

You can see the repository here

https://github.com/MetalLikeCuda/gpumkat

Coder Legion - Recent posts tagged gpu

Animating a 3D Gift Box with WebGPU

From Naive CUDA to Performance Engineering: My First GPU Matmul Journey

What is GPU Passthrough?

The KV Cache Formula That Should Set Your Context Limit

Reverse-Engineering NVIDIA: Modifying a CUDA binary

Motivation

Test code

Build

First hack

Final thoughts

CUDA, ROCm, and the GPU Software Stack: Where Do You Even Start?

I Tried Getting Closer to the GPU With Triton

My CUDA/GPU Journey: From "What Even Is a GPU?" to Actually Fascinated

The moment it stopped being background noise

Falling into CUDA (and discovering it's not the only player)

Where I'm at now

Why I'm writing this

Bypassing the VRAM bottleneck: practical local AI inference on restricted hardware

The -cmoe memory split explained

The thermal saturation problem

Setting up the local AI pipeline

The zero-installer safeguard

Join the local-first revolution

Get started

Snowflake integrates NVIDIA CUDA-X libraries so your Python ML code runs up to 200x faster on GPUs.

Snowflake Brings Native NVIDIA GPU Acceleration to ML Workflows—No Code Changes Required

What's Actually Changing

The Performance Numbers

Why This Approach Works

Real Use Cases

What to Consider

What This Means for Data Teams

Revolutionary open storage cuts AI infrastructure costs 90% while boosting GPU performance by 12%.

Breaking Down the Walls: How Open Storage Architecture is Revolutionizing AI Infrastructure

The Problem with Proprietary Bottlenecks

The Linux-Native Solution

Unlocking Stranded Capacity

The Open Flash Platform: Density Revolution

Simplifying AI Deployment

The Migration Challenge Solved

Looking Forward: The Open Highway

Comparing GPU and CPU Processing Power for Regular Computer Tasks

Key Differences Between GPUs and CPUs

Performance Comparison for Regular Computer Tasks

Practical Implications of GPU and CPU Performance

Technical Details and Optimization

Example Performance Comparison

Choosing the Right Hardware for Your Needs

Future Developments

Conclusion

Additional Considerations

Summary of Key Points

Graid's SupremeRAID uses GPU acceleration to eliminate storage bottlenecks, delivering 28M IOPS for developers.

Graid Technology Revolutionizes Storage Performance with GPU-Accelerated RAID for Developers

Breaking Free from CPU-Based Storage Constraints

Game-Changing Performance for AI Development

Simplifying Development with Software-Defined Architecture

Real-World Developer Impact

Enterprise-Grade Reliability for Mission-Critical Code

The Future of High-Performance Development

Debugging Shaders on macOS: Instruments vs. Gpumkat

Debugging Shaders on macOS: Instruments vs. Gpumkat

Instruments: The Traditional Tool

Features:

Limitations:

Introducing Gpumkat

Key Features:

Why Gpumkat is the Better Alternative

1. Superior Debugging Tools

2. Comprehensive Performance Analysis

3. Better Error Handling

4. Simulated Low-End GPU Testing

5. Enhanced Trace Logging and Timeline Recording

When to Use Instruments vs. Gpumkat

Installing Gpumkat

Conclusion

You can see the repository here

The `-cmoe` memory split explained