Run Kimi K3 Locally: Hardware, Setup and Real Speed Tested

Moonshot released the weights for a 2.8 trillion parameter model, and the smallest of the mainstream dynamic quantizations is 594 GB. That number decides everything else about running Kimi K3 on your own hardware. It does not fit on a workstation, it does not fit on a single H100, and the machines that do hold it are rented by the hour.

This guide covers what running Kimi K3 locally actually requires: the GGUF quantization sizes, the RAM and VRAM floor for each one, how to pull 594 GB of weights and build a llama.cpp that understands the architecture, and the token throughput we measured on four A100 GPUs backed by 2 TB of system memory. The speed result is the part worth reading before you rent anything. Every number below was measured in August 2026 on Ubuntu 24.04 with driver 580.159.03 and CUDA 12.6, running llama.cpp b10245 and the UD-IQ1_S quant across four A100-SXM4-40GB cards.

Kimi K3 in numbers

The specifications come from the official model card and from what llama.cpp reports when it loads the file.

The mixture-of-experts design is what makes the model theoretically approachable. Only 104B parameters activate per token even though 2.8T sit in memory. That helps compute. It does nothing for the memory bill, because the router can pick any expert at any token, so all of them have to stay resident. Note that llama.cpp labels the GGUF 2.8T.A50B, which is its own accounting of the active weight, so do not be thrown when the loader disagrees with the model card.

Work out the memory you need

Unsloth publishes the dynamic quants, and the sizes are the real constraint. We summed the file sizes for every shard through the Hugging Face API rather than trusting the summary table:

Combined memory is what counts, not VRAM alone. Given the flag covered in the serving section below, llama.cpp keeps the expert weights in system RAM and runs those tensors on the CPU backend, which is the only reason a four-GPU box with 160 GB of VRAM can load a 594 GB model at all. Nothing is streamed across the bus; the experts simply execute where they sit. Whether the result is usable is a separate question, and the benchmark section answers it.

Our test box was four A100-SXM4-40GB cards, two AMD EPYC 7542 processors giving 128 threads, and 2015 GiB of RAM. If you are sizing your own machine, the difference between VRAM and pooled system memory matters more here than in any other local model, and the unified memory versus VRAM breakdown covers why. For single-card budgets and normal model sizes, the VRAM-to-model-size map and the GPU rankings for local LLMs cover the sane end of the market.

Download the weights

Every command here runs as root, which is what rented GPU images hand you. On a machine where you are not root, prefix the package install with sudo and point K3_DIR at a path your user owns. Set the paths once so the rest of the commands stay short:

export K3_DIR=/root/k3
export K3_QUANT=UD-IQ1_S
export K3_MODEL="${K3_DIR}/${K3_QUANT}/Kimi-K3-${K3_QUANT}-00001-of-00014.gguf"

The Hugging Face CLI is the fastest way to move this much data. Transfers now run over the Xet backend, which ships as a base dependency. Plenty of guides still tell you to install the hf_transfer extra and export HF_HUB_ENABLE_HF_TRANSFER; that extra no longer exists in current huggingface_hub releases and the variable is a deprecated no-op. Ubuntu marks its system Python as externally managed, so install into a virtualenv rather than fighting pip:

apt install -y python3-venv
python3 -m venv "${HOME}/.venvs/hf"
"${HOME}/.venvs/hf/bin/pip" install -U huggingface_hub
export PATH="${HOME}/.venvs/hf/bin:${PATH}"
export HF_XET_HIGH_PERFORMANCE=1
hf download unsloth/Kimi-K3-GGUF --include "${K3_QUANT}/*" --local-dir "${K3_DIR}"

Most rented GPU images already put you inside a conda or virtualenv Python, in which case the first three lines are unnecessary. On a datacenter link the full 594 GB landed in 235 seconds, a sustained 2.5 GB/s. Budget far more on a home connection. At full line rate the same transfer needs about 80 minutes over 1 Gbps and past 13 hours over a typical 100 Mbps residential link, and since real throughput lands nearer 85 to 90 percent of that, plan for roughly 90 minutes and 15 hours respectively. Confirm the shard count before moving on:

du -sh "${K3_DIR}/${K3_QUANT}" && ls "${K3_DIR}/${K3_QUANT}" | wc -l

Fourteen shards totalling 554 GiB is a complete UD-IQ1_S set:

554G	/root/k3/UD-IQ1_S
14

Keep the shards where they are. llama.cpp reads the whole set from the first file, so there is no merge step to run.

Build llama.cpp with the Kimi K3 support

Mainline llama.cpp does not read this architecture as of August 2026, although an upstream pull request adding the text model is open and active. Unsloth maintains the branch that works today. It also carries the MoonViT vision tower, which this guide does not cover, because the download above pulls only the quant directory and skips the separate projector file that vision needs. Clone the fork and check out the pull request branch:

git clone https://github.com/unslothai/llama.cpp
cd llama.cpp
git fetch origin pull/48/head:kimi-k3-fullsize-vision
git checkout kimi-k3-fullsize-vision
git checkout daef2b3e1b5b1ac7b575f13b13a9450cb2d02862

That last line is optional but recommended. The pull request head still moves, and daef2b3e is the exact commit, build b10245, that produced every number below.

Build with CUDA on. Setting the architecture to match your cards cuts the compile time by a large margin, and 80 is correct for A100. Use 89 for Ada and 90 for Hopper. Blackwell splits in two: 100 covers the B100, B200 and GB200 datacenter parts, while 120 covers the RTX 50 series and the RTX PRO 6000. Both Blackwell targets need CUDA 12.8 or newer. A bare architecture number compiles PTX alongside the device code, so a card newer than your target still runs the binary by jitting it, but the reverse does not hold: build for 90 and run on an A100 and you get no kernel image available for execution on the device.

cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=80
cmake --build build --config Release -j $(nproc) \
  --target llama-cli llama-server llama-bench llama-gguf-split

That took 248 seconds across 128 cores. Run the download and the build at the same time, because neither one competes for the other’s resources and together they are the bulk of your setup cost. If CUDA is not working yet on the host, our NVIDIA driver and CUDA install guide gets that sorted first.

Load and serve the model

This is the configuration that actually loaded on a 160 GB VRAM box. The MoE weights go to the CPU, everything else goes to the GPUs:

./build/bin/llama-server \
  --model "${K3_MODEL}" \
  --n-gpu-layers 99 --cpu-moe --no-mmap \
  --ctx-size 8192 --temp 1.0 --top-p 0.95 \
  -t $(nproc) --host 127.0.0.1 --port 8080

Two flags carry the weight here. --cpu-moe keeps every layer’s expert tensors in system RAM, which is what lets the model fit. --no-mmap reads the whole file into memory instead of paging it from disk on demand, and with 2 TB of RAM available that is the right call because paging 554 GiB through the page cache on every token is ruinous. The load took about nine minutes.

Two warnings about that second flag. llama.cpp now prints a deprecation notice steering you to --load-mode, and the true equivalent of --no-mmap is --load-mode none, not the --load-mode mmap the message appears to suggest. More important: drop --no-mmap entirely if your combined RAM and VRAM is smaller than the quant. Memory mapping is exactly what lets an undersized machine fall back to reading weights off disk instead of being killed by the OOM reaper. It will be painfully slow, but it will run.

The sampling values are the ones Moonshot recommends. Temperature 1.0 with top-p 0.95 for single-step work, top-p 1.0 for agentic loops. Confirm the server actually answers before spending real time on it:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Say hello."}],"max_tokens":16}'

Two things to expect from that call. At the measured rate those sixteen tokens take roughly two and a half minutes to come back, which is the first honest signal of what the rest of this will feel like. And no answer comes back. K3 is a thinking-only model, so a cap that low is spent entirely on the thought. Depending on whether the fork’s parser recognises the K3 thinking tags, you get an empty content with the thought extracted into reasoning_content, or a content full of raw thinking. Either way there is no reply at sixteen tokens, and the server is working. That behaviour becomes a real problem later, and the quality section returns to it.

What Kimi K3 actually runs at

Prompt processing came in at 12.87 tokens per second under llama-bench, and between 5.5 and 6.2 tokens per second through the server on short prompts. Token generation is where the story ends: 0.10 tokens per second, held steady across the sampled window.

One token every ten seconds is not a usable chat model. A 500 token answer takes an hour and twenty three minutes. We aborted the llama-bench generation test after 17 minutes because it had not finished 128 tokens, and a single short question never produced a complete answer inside the rental budget.

The cause is memory bandwidth, not compute. GPU utilisation sat between 0 and 1 percent for the whole run with only 64 GB of the 160 GB VRAM in use. Every token has to pull its active expert set out of system RAM, and that read is the wall. Adding GPUs does not help unless you add enough of them to hold the entire model.

Two tuning attempts changed nothing worth reporting. Passing --numa distribute on this eight-node NUMA machine produced 5.85 to 6.12 tokens per second on prompt processing against 5.54 to 6.23 without it, which is inside the noise. The kernel also flagged numa_balancing as enabled and performance-impairing, and that file is read-only inside a container, so it could not be turned off.

Error: “cudaMalloc failed: out of memory” allocating 96937.09 MiB on device 3

Trying to keep the last 15 layers’ experts on the GPUs with --n-cpu-moe 78 fails immediately:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 96937.09 MiB on device 3: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA3 buffer of size 101645909504
llama_model_load: error loading model: unable to allocate CUDA3 buffer

The total VRAM was sufficient. The split was not. llama.cpp assigns the highest layer indices to the last device, so all 15 GPU-resident expert layers landed on one 40 GB card and asked it for 97 GB. Balancing this needs explicit -ot tensor placement across the four devices rather than the layer-count shortcut. Given that the fundamental limit is total memory bandwidth, the payoff for getting that regex right is small.

Is the output any good

Judging quality from a 1-bit quant running at a tenth of a token per second is not fair to the model, so we ran the same prompts against the hosted reference model instead. Moonshot trains and serves K3 with MXFP4 weights, so even that is not BF16, but it is the model behaving as its authors intend. Both answers were correct.

On a merge-overlapping-intervals problem it produced the standard sort-then-sweep solution at O(n log n), correctly treated touching intervals like [1,4] and [4,5] as mergeable, returned new lists rather than mutating the caller’s input, and listed empty input, single intervals, unsorted input and fully nested intervals as the cases it handled.

Given a systemd trap, asking why SSH dies after a reboot once ssh.socket is disabled on Ubuntu, it identified socket activation as the mechanism, explained that disabling the socket unit does not enable ssh.service, and offered both the re-enable path and the explicit switch to a standalone daemon. That is the correct answer to a question that catches plenty of administrators.

One behaviour will cost you money if you miss it. Kimi K3 defaults to a very high reasoning effort. At a 1200 token output cap our systemd question returned completely empty content, with all 1200 tokens consumed by hidden reasoning. Raising the cap to 4000 tokens returned empty again after 4088 reasoning tokens, billed past the cap rather than stopped at it. Only lowering the reasoning effort produced an answer, in 1116 tokens. Budget five thousand output tokens or more per call, or turn the reasoning effort down, otherwise you pay for thinking and receive nothing.

What it costs to run Kimi K3 locally

Owning hardware that runs this model well is not a homelab decision. Fitting all 594 GB in VRAM so generation is not bandwidth-starved means eight 80 GB cards at minimum, and Unsloth reports roughly 20 tokens per second on B200 class hardware. The densest workstation card you can order is the 96 GB RTX PRO 6000 Blackwell, and even the 1-bit quant needs seven of them. That is a rack, not a workstation.

Renting is the honest comparison. Our four A100 box ran at $3.87 an hour and the whole exercise, download and build and benchmark included, cost $8.29. It produced a model too slow to hold a conversation with. A box large enough to be quick costs several times that per hour, and hosted Kimi K3 sits at $3 per million input tokens and $15 per million output. The break-even against an API is measured in months of continuous use, and that maths only works if you have a compliance reason to keep the weights in your own building. We rented through vast.ai, where per-second billing keeps an experiment like this cheap.

For most people the practical answer is to keep Kimi K3 as an API model and run something smaller on your own metal. A 30B class model on a single 24 GB card gives you interactive speed for real work, and a renewed RTX 3090 remains the cheapest way into that bracket. Our llama.cpp local LLM guide and Ollama setup both cover that path. If you want K3 specifically without the hardware, the Kimi CLI with OpenRouter reaches it over an API key. For serving models at production concurrency rather than single-user chat, SGLang and vLLM are the engines to compare, and anyone specifying a machine for local inference should start from the local AI workstation build.