Ollama, LM Studio, and Jan all run on the same engine underneath: llama.cpp. It is the C/C++ inference core that made running open models on ordinary hardware practical, and it works on a plain CPU or offloads to a GPU when you have one. Learning it directly means you stop guessing what those friendlier wrappers are actually doing.
Four steps cover the whole workflow. You build llama.cpp once, pull a model in the GGUF format, run it from the command line, and start an OpenAI-compatible server that any existing client can talk to. Everything below is the exact commands, the real output, and honest numbers from both a plain server CPU and an RTX 4090.
If you just want a working local model in one command and do not care about the internals, our Ollama setup guide is the shortcut, and we compare the two head to head at the end. The build, runs, and benchmarks here were done on Ubuntu 26.04 (Intel Core i7-6700, CPU only) and a rented RTX 4090, on the current llama.cpp (the b10075 release series) with Qwen2.5-3B in July 2026.
What llama.cpp actually is
llama.cpp is an inference engine. It loads a quantized model file, runs the math that turns your prompt into tokens, and hands them back. It ships a set of small tools rather than one monolith: llama-cli for interactive chat and one-shot generation, llama-server for an HTTP API, and llama-bench for measuring throughput. Models come in the GGUF format, a single file that bundles the weights, the tokenizer, and the chat template.
The important part for expectations: it runs on CPU out of a plain build, and the same source builds against CUDA to push the model onto an NVIDIA GPU. The model file is identical either way. That portability is why so many desktop apps embed it instead of writing their own runtime.
Prerequisites
Any recent Linux box works. The two things that actually gate you are memory and, optionally, a GPU:
- RAM (CPU inference): the model has to fit in memory, so size RAM to the model file plus the context. A 3B model quantized to Q4 is about 2 GB, an 8B is roughly 5 GB, a 14B around 9 GB. Add headroom for the OS and the KV cache.
- VRAM (GPU inference): to offload fully, the model plus its KV cache must fit in GPU memory. The 3B Q4 model here used about 3.5 GB of VRAM. Our guide on how much VRAM you need to run an LLM has the sizing math for larger models.
- Build tools: a C++ toolchain, CMake, Git, and the CURL development headers (llama.cpp uses them to pull models straight from Hugging Face).
No GPU is fine. The generation numbers later show exactly what an older CPU delivers so you can decide whether it is enough for your use.
Install the build tools
On Ubuntu or Debian, one apt line covers everything the CPU build needs:
sudo apt update
sudo apt install -y build-essential cmake git libssl-devThe libssl-dev package matters more than it looks. llama.cpp dropped libcurl in early 2026 and its Hugging Face downloader now links OpenSSL directly, so without the OpenSSL headers present at build time the -hf flag later fails with 'https' scheme is not supported. Install it before you build. On RHEL, Rocky, or AlmaLinux, the equivalent set is gcc-c++ cmake git openssl-devel.
Build llama.cpp
Clone the repository and build with CMake. The two commands configure a Release build and compile it:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j"$(nproc)"On the four-core i7-6700 test box the compile finished in about five and a half minutes. When it completes, the tools land in build/bin/. Confirm they are there:
ls build/bin/ | grep -E 'llama-(cli|server|bench)'You should see the three tools this guide uses:
llama-bench
llama-cli
llama-serverCheck the version to confirm the binary runs and note the build it reports:
./build/bin/llama-cli --versionIt prints the build number and the commit it was compiled from:
version: 1 (5735e10)
built with GNU 15.2.0 for Linux x86_64A source build reports its own build number (here 1) and the commit hash it was cut from, not the b10xxx release tag, so do not worry that it does not print a release number. The commit is what pins the exact code.
Prefer not to compile? The project also ships prebuilt Linux binaries on its GitHub releases page (the llama-*-bin-ubuntu-x64.tar.gz asset for CPU). Building from source is worth it though, because the compiler tunes for your exact CPU and it is the only path that pulls in CUDA.
Download and run a model
llama.cpp can pull a GGUF straight from Hugging Face with the -hf flag, caching it under ~/.cache/llama.cpp so the next run is instant. This starts an interactive chat with a 3B model quantized to Q4_K_M:
./build/bin/llama-cli -hf Qwen/Qwen2.5-3B-Instruct-GGUF:Q4_K_MType a question at the prompt and it answers, keeping the conversation in context. For a scripted, one-shot generation that prints an answer and exits instead of waiting for input, pass a prompt with -st (single turn):
./build/bin/llama-cli -hf Qwen/Qwen2.5-3B-Instruct-GGUF:Q4_K_M -st \
-p "Explain what a reverse proxy does, in two sentences." -n 160On the CPU-only box it answered in a few seconds and printed its own speed on the last line:
A reverse proxy sits between a server and clients, forwarding requests from clients
to the server and serving responses from the server to the clients, thereby
improving performance and security.
[ Prompt: 35.1 t/s | Generation: 12.0 t/s ]Twelve tokens a second on a 2015-era desktop CPU is slower than a hosted API but perfectly usable for a chat or a background job. If you skip -st in a script or a pipe, llama-cli drops into interactive mode and appears to hang waiting for input that never comes, so always add it for non-interactive use.
The GPU, benchmark, and server commands below point at the model file directly rather than through -hf, and llama-bench in particular only accepts a local path. Pull the same GGUF into your working directory once so those commands have it:
curl -L -o qwen2.5-3b-instruct-q4_k_m.gguf \
https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF/resolve/main/qwen2.5-3b-instruct-q4_k_m.ggufOffload the model to a GPU
A GPU changes the experience entirely. To use one, rebuild with CUDA enabled. The build needs the NVIDIA CUDA toolkit (nvcc) installed; then it is one extra flag:
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j"$(nproc)"Compiling the CUDA kernels is heavier than the CPU build, though on a 32-core host it finished in about two minutes. Once built, add -ngl 99 to push all model layers onto the GPU:
./build/bin/llama-cli -m qwen2.5-3b-instruct-q4_k_m.gguf -ngl 99 -st \
-p "Explain what a reverse proxy does, in two sentences." -n 160The same model, same prompt, on an RTX 4090:
[ Prompt: 2395.4 t/s | Generation: 294.6 t/s ]Generation jumped from 12 to 295 tokens a second. To confirm the model is actually resident on the GPU rather than quietly falling back to the CPU, check nvidia-smi while a model is loaded:
nvidia-smiThe process shows up holding VRAM, which is the proof it offloaded:
The 3B Q4 model takes about 3.5 GB of the card’s 24 GB, leaving plenty of room for a much larger model or a long context. If your GPU cannot hold the whole model, drop the number after -ngl to offload only some layers and keep the rest on the CPU. Picking a card for this is its own decision, covered in our roundup of the best GPUs for local LLMs.
Benchmark CPU against GPU
The [ Prompt | Generation ] line from llama-cli is a rough guide. For repeatable numbers, llama-bench runs a fixed prompt-processing test (pp512) and a generation test (tg128) and reports the mean with a standard deviation. Run it against the model file:
./build/bin/llama-bench -m qwen2.5-3b-instruct-q4_k_m.gguf -ngl 99On the RTX 4090 it reported this:
| model | size | params | backend | ngl | test | t/s |
| ---------------------- | ------: | ------: | ------- | --: | ----: | ---------------: |
| qwen2 3B Q4_K - Medium | 1.95 GiB| 3.40 B | CUDA | 99 | pp512 | 22279.13 ± 3137 |
| qwen2 3B Q4_K - Medium | 1.95 GiB| 3.40 B | CUDA | 99 | tg128 | 327.69 ± 0.77 |Running the same command on the CPU box (drop -ngl, since there is no GPU) tells the other half of the story. Here are both, side by side, for Qwen2.5-3B at Q4_K_M:
| Backend | Prompt processing (pp512) | Generation (tg128) |
|---|---|---|
| CPU (Intel i7-6700, 4 threads) | 43 t/s | 11.8 t/s |
| GPU (NVIDIA RTX 4090) | 22,279 t/s | 327.7 t/s |
Here are both tools running on the RTX 4090, the live generation from llama-cli and the llama-bench figures behind the table:
The prompt-processing gap is the one people underestimate. Generation is memory-bandwidth bound, so the GPU wins by a wide but finite margin. Prompt processing is compute bound and embarrassingly parallel, which is exactly what a GPU eats for breakfast, so the gap there is enormous. If you feed the model long documents, that first number is what you feel.
Serve an OpenAI-compatible API
The command line is fine for testing, but the reason to keep llama.cpp running is llama-server. It exposes an HTTP API that speaks the OpenAI format, so anything built for OpenAI (a script, LangChain, a chat UI) points at it by changing one URL. Start it with the model and, if you have a GPU, the offload flag:
./build/bin/llama-server -m qwen2.5-3b-instruct-q4_k_m.gguf -ngl 99 \
--host 127.0.0.1 --port 8080 --alias qwen2.5-3bIt loads the model and starts listening. The startup log confirms it is up:
srv load_model: initializing, n_slots = 4, n_ctx_slot = 32768
srv llama_server: model loaded
srv llama_server: listening on http://127.0.0.1:8080From another terminal, a quick health check should return ok:
curl -s http://127.0.0.1:8080/healthA healthy server replies with a short JSON status:
{"status":"ok"}Now call it exactly as you would call OpenAI, at the /v1/chat/completions path:
curl -s http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen2.5-3b","messages":[{"role":"user","content":"In one sentence, what is llama.cpp?"}],"max_tokens":64}'The response is the standard OpenAI shape, with a timings block llama.cpp adds so you can see per-request throughput:
{
"choices": [
{ "message": { "role": "assistant",
"content": "llama.cpp refers to a library used for fast and efficient language model inference." },
"finish_reason": "stop" }
],
"model": "qwen2.5-3b",
"usage": { "completion_tokens": 17, "prompt_tokens": 38, "total_tokens": 55 },
"timings": { "prompt_per_second": 28.4, "predicted_per_second": 12.4 }
}That is the whole trick to running local models in an application: point your existing OpenAI client at http://your-host:8080/v1 and it works, no code change beyond the base URL. For heavier, multi-user serving where throughput under concurrency matters more than a single stream, a dedicated engine is the better fit, which is what our SGLang and vLLM comparison and the vLLM production guide cover.
llama.cpp or Ollama?
This is the question everyone asks, so we ran it properly. Ollama installs in one line and manages models for you, but it uses a llama.cpp-derived runner underneath. To see the cost of that convenience, we installed Ollama on the same RTX 4090 and ran the same Qwen2.5-3B, which Ollama also serves at Q4_K_M, so it is a fair fight. Ollama held it fully on the GPU:
ollama psThe output shows the model held entirely in GPU memory:
NAME ID SIZE PROCESSOR CONTEXT UNTIL
qwen2.5:3b 357c53fb659c 3.4 GB 100% GPU 32768 4 minutes from nowOn a warm run (model already in VRAM), ollama run --verbose reported an eval rate of 263 tokens a second. Raw llama.cpp on the identical model and card generated 294 to 328 tokens a second depending on the tool. So llama.cpp is meaningfully faster, but they are the same order of magnitude, which makes sense because they share an engine. The difference is the thin scheduling and API layer Ollama adds on top.
| llama.cpp | Ollama | |
|---|---|---|
| Setup | Build from source or grab a binary | One-line install script |
| Model management | You point at GGUF files | A registry with ollama pull |
| Generation (Qwen2.5-3B, RTX 4090) | 294 to 328 t/s | 263 t/s |
| API | OpenAI-compatible via llama-server | OpenAI-compatible, always running |
| Control | Every flag (threads, layers, cache) | Sensible defaults, less to tune |
Use Ollama when you want models to just work and switch between them casually. Reach for llama.cpp when you want the last bit of performance, control over every runtime flag, an embeddable library, or you are building the thing Ollama itself is built from. If Ollama is your pick, the CPU-only Ollama walkthrough mirrors this guide on the wrapper side. The GPU numbers here were captured on a rented RTX 4090 through vast.ai, which is the cheapest way to benchmark a card you do not own.
Common misconceptions
A few things trip people up when they first reach for llama.cpp, and clearing them makes the tool a lot less intimidating.
“llama.cpp and Ollama are competitors”
They are not. Ollama, LM Studio, Jan, and most desktop “run a local model” apps embed a llama.cpp-derived runner and the GGUF format it defines. Choosing between them is choosing how much of the engine you want to manage yourself, not choosing between rival engines.
“You need a GPU to run a local LLM”
You do not. A 2015 quad-core desktop generated a usable 12 tokens a second on a 3B model with no GPU at all. A GPU makes it roughly 28 times faster, which matters for real work, but “no GPU” means “slower”, not “impossible”. Small quantized models on CPU are genuinely fine for chat and batch jobs.
“Quantization ruins the model”
Q4_K_M, the quant used throughout this guide, cuts a model to under a third of its full size while keeping answer quality close to the original for most tasks. That is why it is the default almost everywhere. You trade a small, often unnoticeable quality drop for a model that fits in a fraction of the memory and runs several times faster. Start there, and only move to a larger quant if you can measure a difference that matters to you.
