Thirteen billion of DeepSeek V4 Flash’s 284 billion parameters activate per token. That one number decides the entire hardware conversation. A dense model of the same size would be out of reach for anything short of a GPU cluster, and this one is not.
This guide covers what it takes to run DeepSeek V4 Flash locally: which GGUF quant tier to pull, the real memory floor measured rather than estimated, and the tokens per second you get once it loads. Everything below was benchmarked with llama.cpp build b10273 on Ubuntu 24.04 in August 2026, on a 16 thread Xeon with 8 physical cores, 123 GiB of usable RAM, and no GPU at all.
Why a 284B model fits in 128GB of RAM
Every expert has to be in memory, but only a fraction of them do any work on a given token. Mixture of experts routes each token through a small subset of the network, so this model holds 284 billion parameters resident while multiplying roughly 13 billion of them per token.
The consequence is the whole hardware story. Memory capacity scales with the total parameter count, because any expert can be selected at any moment and all of them have to be loadable. Memory bandwidth and compute scale with the active parameter count, which is about 4.6% of the total here.
So the binding constraint is how much memory you have, not how fast it is. That inverts the usual advice. A machine with a large pool of ordinary DDR5 can hold this model, while a fast 24 GB graphics card cannot touch it. If the split between pooled system memory and dedicated VRAM is new territory, the unified memory versus VRAM breakdown covers it, and the VRAM sizing arithmetic applies the same logic to smaller models.
The same capacity-first reasoning is what ruled out running Kimi K3 locally on this class of machine. Both are sparse MoE models, but that one needs a 610 GB floor across RAM and VRAM combined, which is a data centre problem rather than a desk problem.
Two housekeeping facts before the install. The 0731 release on Hugging Face is the official one and supersedes the earlier preview, so target it and not the plain repository. And the license is MIT, so local use, fine tuning, and commercial deployment are all permitted.
Pick a GGUF quant tier
Unsloth publishes thirteen tiers, from a 1 bit squeeze that lands under 83 GB up to an 8 bit build at 162 GB. What you can run is decided by RAM plus VRAM combined, because llama.cpp will split the model across both. These are the ones worth choosing between.
| Quant tier | Size | Bits | Practical target |
|---|---|---|---|
| UD-IQ1_S | 82.5 GB | 1 | 96 GB machines |
| UD-IQ2_M | 90.9 GB | 2 | 128 GB with room for long context |
| UD-Q2_K_XL | 96.8 GB | 2 | 128 GB |
| UD-IQ3_XXS | 104 GB | 3 | 128 GB, the quality pick |
| UD-IQ3_S | 116 GB | 3 | 128 GB is tight, 192 GB is comfortable |
| UD-IQ4_XS | 137 GB | 4 | 192 GB |
| UD-Q4_K_XL | 155 GB | 4 | 192 GB, near lossless |
| UD-Q8_K_XL | 162 GB | 8 | 192 GB, lossless |
Unsloth’s documentation for this model recommends UD-IQ3_XXS for best results and puts the floor at 110 GB of RAM. That is the tier used for every number in this guide. Its four shards totalled 104,207,848,032 bytes on disk, which llama.cpp reports as 97.05 GiB once loaded, and the gap between those two figures is GB against GiB rather than anything being dropped.
The model size is a floor, not the requirement. KV cache for the context sits on top of it, and the context window here runs to 1,048,576 tokens if you feed it that much. On a 128 GB machine a 2 bit tier buys considerably more usable context than a 3 bit one does, so pick the quant against the context length you actually need rather than the largest one that technically loads.
Run DeepSeek V4 Flash locally with llama.cpp
Three things have to happen: build llama.cpp, pull the shards, point one at the other. Start with the toolchain and a virtualenv for the Hugging Face download client.
sudo apt update
sudo apt install -y build-essential cmake git curl libssl-dev time python3-pip python3-venvKeep the download client out of the system Python:
python3 -m venv ~/hfvenv
~/hfvenv/bin/pip install "huggingface_hub[hf_transfer]"Set the quant and the destination once, since both are referenced repeatedly from here on:
export QUANT=UD-IQ3_XXS
export MODEL_DIR=$HOME/models/v4flashStart the download first, because 104 GB takes longer to arrive than llama.cpp takes to compile:
HF_HUB_ENABLE_HF_TRANSFER=1 ~/hfvenv/bin/hf download \
unsloth/DeepSeek-V4-Flash-0731-GGUF --include "${QUANT}/*" \
--local-dir "${MODEL_DIR}"While that runs, build llama.cpp in a second shell. Re-run the two exports there, because every command from here on uses them. Support for this architecture is in mainline, so a fresh clone needs no patches:
git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp
cd ~/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON
cmake --build build --config Release -j"$(nproc)"GGML_NATIVE=ON is already the default when you compile on the machine that will run the model, so the flag above is a reminder rather than a change. What matters is not turning it off: it compiles for the local CPU’s instruction set, and on a Sapphire Rapids or Zen 4 host that is the difference between using the wide vector units and ignoring them. Older build guides also pass -DLLAMA_CURL=OFF at this point. Skip it. llama.cpp dropped libcurl for a bundled HTTP client, and the option is now deprecated and silently ignored, so it switches nothing off. For a slower walk through the same build, the llama.cpp setup guide covers it in more detail.
Two binaries matter. Confirm both landed before going further:
ls build/bin | grep -E '^llama-(cli|bench)$'Both names print, one per line:
llama-bench
llama-cliNow load it. Point at the first shard and llama.cpp finds the other three itself. The sampling values are the ones Unsloth’s model documentation specifies, and they are wider than what you would use on a dense chat model. One carve out worth knowing before you wire this into anything: Unsloth suggests --top-p 0.95 for agentic workloads and 1.0 for everything else.
~/llama.cpp/build/bin/llama-cli \
--model "${MODEL_DIR}/${QUANT}/DeepSeek-V4-Flash-0731-${QUANT}-00001-of-00004.gguf" \
--threads 16 \
--temp 1.0 --top-p 1.0 --min-p 0.0 \
-st \
-p "In exactly three sentences, explain what a mixture-of-experts model is."That -st is the flag people miss. Without single turn mode, llama-cli drops into an interactive session, and a scripted run with no terminal attached spins on end of file instead of exiting. It wrote a 388 MB log file in two minutes before we caught it.
The startup banner confirms which build and which quant actually loaded:
build : b10273-a6aa6f545
model : /home/ubuntu/models/v4flash/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf
ftype : IQ3_XXS - 3.0625 bpw
modalities : textGeneration then opens with a reasoning block before any answer text appears. Ours began verbatim like this:
[Start thinking]
1. **Understand the User's Request**: The user wants a three-sentence explanation of what a mixture-of-experts (MoE) model is and why only *some* parameters activate per token.Budget for that. Every response spends tokens restating the request before it answers, and at single digit tokens per second those tokens are minutes of wall clock during which the reader sees nothing.
It has an off switch, and on a slow machine it is the single biggest lever you have. --reasoning off drops the thinking block entirely. To keep the reasoning and tune it instead, pass --chat-template-kwargs '{"reasoning_effort":"max"}', which also accepts high. High is the default, which is why an unconfigured run thinks so much before it speaks. Older guides reach for enable_thinking through that same kwargs mechanism, and llama.cpp now warns that spelling is deprecated and points at --reasoning instead.
What it runs at on a CPU-only 128GB box
llama-bench reports prompt processing and generation separately, without a chat session’s overhead in the way. Prefill and decode are different workloads on this model and a single averaged figure hides the interesting part.
/usr/bin/time -v ~/llama.cpp/build/bin/llama-bench \
-m "${MODEL_DIR}/${QUANT}/DeepSeek-V4-Flash-0731-${QUANT}-00001-of-00004.gguf" \
-p 512 -n 128 -t 16 -r 1One repetition, which is why the variance column reads zero. This is what came back:
| model | size | params | backend | threads | test | t/s |
| --------------------------------- | ---------: | ---------: | ------- | ------: | ----: | -----------: |
| deepseek4 ?B IQ3_XXS - 3.0625 bpw | 97.05 GiB | 284.33 B | CPU | 16 | pp512 | 12.33 ± 0.00 |
| deepseek4 ?B IQ3_XXS - 3.0625 bpw | 97.05 GiB | 284.33 B | CPU | 16 | tg128 | 5.61 ± 0.00 |
build: a6aa6f545 (10273)
Maximum resident set size (kbytes): 109503840The last line comes from time rather than llama.cpp, and it is the one to read twice. Peak resident set was 109,503,840 units of what GNU time labels kbytes but actually reports in kibibytes, so the true peak is 104.4 GiB. Converted to the decimal gigabytes model cards use, that is 112.1 GB, against 123 GiB of usable RAM. About 18.6 GiB of headroom left.
That is the measured basis for treating 128 GB as the floor for this tier rather than a number borrowed from a model card, and note which side of Unsloth’s 110 GB guidance it lands on. The real peak came in above it, not under it. Read 110 GB as the point where the weights fit and 128 GB as the point where the machine still has room to run them.
Nothing swapped. During generation the process showed zero swap use, zero block reads, and 93% user CPU with 16 to 18 runnable threads, so the model was resident in page cache and the machine was purely compute bound. A 284B parameter model does generate text on a CPU with no accelerator at all.
Prompt processing is the real bottleneck
5.61 tokens per second of generation is slow but workable for batch jobs you leave running. 12.33 tokens per second of prompt processing is what actually rules this configuration out for interactive work.
Run the arithmetic against a realistic agent prompt. Ten thousand tokens of system prompt, tool definitions, and pasted source at 12.33 tokens per second is 811 seconds, so roughly 13.5 minutes before the first output token appears. Point it at a 100,000 token codebase and you are waiting over two hours for the model to finish reading.
The same 128 GB memory budget with an accelerator able to reach it behaves nothing like this. A published DGX Spark run on llama.cpp using the GB10 with 128 GB of unified memory reports 459 to 462 tokens per second of prompt processing and about 19.1 tokens per second of single stream generation on the 2 bit UD-IQ2_M tier.
| Metric | CPU only, 123 GiB RAM | DGX Spark GB10, 128 GB unified |
|---|---|---|
| Quant tier | UD-IQ3_XXS (104 GB) | UD-IQ2_M (90.9 GB) |
| Prompt processing | 12.33 tok/s | 459 to 462 tok/s |
| Generation, single stream | 5.61 tok/s | about 19.1 tok/s |
| Time to first token, 10k prompt, calculated | about 13.5 min | about 22 s |
| Source | measured here | NVIDIA developer forum report |
Generation improves by 3.4x. Prompt processing improves by 37x. That is not a controlled comparison. The quant differs, the context configuration differs, and the Spark run used llama-server with flash attention on, a 2048 token batch, and a 1024 token microbatch while ours used llama-bench defaults, which flatters prefill on the Spark side specifically. None of that accounts for a factor of 37. Prefill is dense matrix multiplication across the whole batch and it wants parallel compute; decode is memory bound, and a CPU with a wide enough memory pool keeps up far better than most people expect.
One caveat on our own figures. The CPU box was a cloud VM slice (AWS r7i.4xlarge, Xeon Platinum 8488C, 8 physical cores with SMT), and a slice does not guarantee the host’s full memory bandwidth. We also ran 16 threads to match the vCPU count, where llama.cpp usually prefers threads matched to physical cores, so there is probably a little left on the table there too. Treat 5.61 and 12.33 as a floor for the CPU-only class rather than its ceiling.
The hardware that actually works
Roughly 128 GB of memory that a GPU or NPU can address directly. That is the class where this model crosses from technically running to genuinely usable, and there are three ways to get there.
Unified memory machines. DGX Spark, the Ryzen AI Max+ 395 platform, and Apple’s higher end M series all pool one memory bank between CPU and accelerator, which is exactly what a sparse MoE wants. The Ryzen AI Max+ 395 systems compared covers what shipping boxes in that class cost and where they differ, and the Mac mini against mini PC against discrete GPU piece frames the same decision by price. If you are shopping in that bracket, the mini PC picks for local AI guide tracks memory configuration against model size.
Multi GPU rigs. Enough VRAM to hold a 2 bit or 3 bit tier outright is the fastest route and the most expensive per gigabyte, and which GPU to buy for LLM work covers the current cards. Going partway is the more interesting option: llama.cpp’s --n-cpu-moe N keeps the expert weights of the first N layers in system RAM while the hot layers and the KV cache stay on the GPU, and --cpu-moe keeps every expert on the CPU. That hybrid split is the reason MoE models are practical on consumer hardware at all, and it is where a single 24 GB card stops being useless.
CPU only with a lot of RAM. It works, and the benchmark above proves it works. At 12.33 tokens per second of prefill it belongs to overnight batch jobs and nothing that a human waits on. Worth using if the RAM is already in the machine, never worth buying for.
Then there is the option that undercuts all three. DeepSeek lists deepseek-v4-flash at $0.14 per million input tokens on a cache miss, $0.0028 per million on a cache hit, and $0.28 per million output. A peak and off-peak schedule has been announced but has no effective date yet, and it will double the rate during two blocks of the Beijing working day once it lands. A day of heavy agent use costs cents. Running the weights yourself wins when they have to stay on your own hardware, when there is no network, or when you intend to fine tune. If none of those apply, the hosted API is cheaper than the electricity.
What these numbers do not cover
Four gaps, named plainly, because a benchmark that hides its limits is worth less than one that states them.
No GPU offload figure. The rig had no accelerator, so there is no measurement here for the hybrid expert split described above. That is the configuration most readers will actually run, and it is the number missing from this guide.
No Apple silicon result. M series machines run this through a different stack and the figures do not transfer, so treat any Mac number you see as unrelated to these.
One quant only. Everything above is UD-IQ3_XXS. The 2 bit tiers are smaller and faster and we did not measure what accuracy they give up, which on a reasoning model is the question that matters.
No long context throughput. Raising the context to 32,768 slowed the same prompt to a crawl, with the model still fully resident and zero swap, so it was not memory thrashing. We did not isolate the cause, so there is no tokens per second figure here for long context. A proper depth sweep is the next measurement.
If you have a 128 GB machine with an accelerator that can reach the memory, that is the test worth running. It is also the one that decides whether a 284B model has any place in your daily workflow.
