JetBrains shipped Mellum2.1-12B-A2.5B-Thinking on October 8, 2026, and it is the first model from the company that a mid-range laptop can serve to a coding agent all day. If you want to run an LLM locally for private code help, this tutorial takes you from an empty machine to a working endpoint in 13 steps, about 40 minutes of hands-on time plus the download. The reason to bother is a self-reported jump: JetBrains says SWE-bench Verified performance climbed from 2.0% for Mellum 2 to 47.0% for Mellum 2.1, mostly thanks to reinforcement learning inside real repository sandboxes.
Treat that 47.0% as a vendor number, not a guarantee. Nobody outside JetBrains has reproduced it in the sources we reviewed, and your own repositories will behave differently. That is exactly why a local setup helps: you can test the model on your code, offline, without sending a single file to a third party. For the launch details and the architecture, read our Mellum 2.1 launch report. This article is the hands-on companion.
We will serve the model two ways. First through llama.cpp’s llama-server, which gives you the most control. Then through Ollama v0.40.10, which gives you the simplest day-to-day workflow. You will end with a Python client, a tokens-per-second benchmark you can rerun after every tweak, and a note on how to point an editor agent at the endpoint. Every command below was written against the October 2026 state of these projects, and where a detail was still moving when we checked, we say so.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
Why Run Mellum 2.1 Locally in October 2026
Most “run an LLM locally” guides still start with a general-purpose chat model and a toy prompt. Mellum 2.1 is a better teaching target because it is built for the job developers actually do: reading a repository, editing files, running commands. It is a mixture-of-experts model with 12 billion total parameters, roughly 2.5 billion of them active per token, a 131,072-token context window and an Apache 2.0 license. That mix matters for local use. Only the active experts do work on each token, so generation speed behaves more like a small model, while the full 12 billion parameters still have to sit in memory.
JetBrains also published its own throughput claim. In the company’s launch post, JetBrains says Mellum2.1 is the fastest model in its comparison group under heavy load and serves almost twice as many tokens as Qwen3.5-9B. For a single request, the company says multi-token prediction (MTP) makes it about 1.6 times faster. We will test the second claim ourselves in Step 12, because single-request latency is what you feel on a laptop.
Demand for this kind of setup is not niche. Stack Overflow’s 2025 Developer Survey found that 84% of developers were using or planning to use AI tools, and 51% of professional developers used them daily. The same survey found 45% of developers called debugging AI-generated code a time sink. A local model will not fix that last number, but it does remove the other friction points: per-token bills, rate limits, and the compliance conversation about where your source code goes.
Two cautions before you start. First, Mellum 2.1 replaced Mellum 2 on October 8, so any quantized file labeled Mellum 2 is the previous generation and does not belong in this tutorial. Second, “Thinking” in the name means the model reasons before it answers. You will see long reasoning traces, and your token budgets need to account for them. Both points come back in the pitfalls section.
Prerequisites and Versions to Check First
You need a machine with enough memory, a few developer tools, and the right minimum versions. The version that bites most people is llama.cpp. An official repository change dated October 19, 2026 says the Mellum2.1 GGUF files require a llama.cpp build around b11747 or newer. An older build will fail with an unknown-architecture error. Note that llama.cpp numbers its builds with a b prefix, which is separate from Ollama’s version numbers. Ollama v0.40.5 appeared in one report as including a llama.cpp bump, and that is an Ollama release number, not a llama.cpp one.
| Component | Minimum | What we used | Why it matters |
|---|---|---|---|
| Model | Mellum2.1-12B-A2.5B-Thinking (GGUF) | Q4_K_M quantization | Mellum 2 files are the previous generation |
| llama.cpp | Build b11747 or newer | Latest main at install time | Older builds cannot load the architecture |
| Ollama | Latest release | v0.40.10 (October 23, 2026) | Needs a recent bundled engine for the GGUF |
| Python | 3.11 | 3.12 | Runs the client and benchmark scripts |
| Git and CMake | Any current release | Distribution default | Builds llama.cpp from source |
| Memory | 16 GB unified or system RAM | 16 GB laptop, 12 GB GPU test box | Weights plus KV cache must fit |
| Disk | 15 GB free | NVMe SSD | Model file, build tree, caches |
On memory, the third-party sizing reports line up. The ModelFit write-up of running Mellum2.1 on a Mac puts the Q4 file at about 8.1 GB and calls 16 GB the real floor once the operating system and the context cache are counted. A separate guide estimates the Q4_K_M file at roughly 7.5 GiB and argues it fits a 12 GB graphics card. Both are third-party estimates, not JetBrains specifications, so verify the real file size after download. We will do that in Step 3.
You do not need a paid account for any of this. The Ollama runtime is free and unlimited when models run on your own hardware, and cloud usage is billed separately. Ollama’s Pro plan, for reference, costs $20 per month or $200 per year, but nothing in this tutorial touches it. If you want a gentler primer on the basics first, our general guide on how to run an LLM locally covers hardware choices without a specific model.
Memory, Quantization and Context Planning
Before you download anything, decide what you are optimizing. A 12-billion-parameter mixture-of-experts model stores every expert on disk and in memory, even though only about 2.5 billion parameters fire per token. Quantization shrinks the stored weights. Q4_K_M is the common middle ground: much smaller than the full-precision weights, with a quality loss most people accept for coding help. JetBrains’ own Hugging Face instructions show llama-server and llama-cli commands using Q4_K_M with a 131,072-token context, which is why we picked it.
The context window is the second budget. The model supports 131,072 tokens, but the key-value cache that remembers your conversation grows with context length, and it competes with the weights for the same memory. On a 16 GB machine, start at 16,384 or 32,768 tokens and raise the number only after you measure headroom. Agents that read many files will want more, and you will feel the trade-off immediately: bigger context, slower prompt processing, higher memory use.
| Runtime | Best for | Control level | Trade-off |
|---|---|---|---|
llama.cpp llama-server | Tuning, speculative decoding, parallel slots | High, every flag exposed | You build and update it yourself |
| Ollama | Daily use, model switching, editor integrations | Medium, via Modelfile | Engine updates arrive on Ollama’s schedule |
| LM Studio | Point-and-click testing on desktop | Low to medium | Closed-source app, GUI-first |
If you are still choosing between desktop runners, our Ollama vs LM Studio vs Jan comparison breaks down the differences in more depth. This tutorial uses the first two rows because they expose the clearest command-line workflow, and because the MTP draft head in Step 12 is easiest to try on llama.cpp.
Steps 1-3: Prepare the Machine and Download the Model
Step 1: Check your hardware
Start by finding out what you actually have. On Linux with an NVIDIA card, nvidia-smi reports VRAM and driver version. On macOS, unified memory is your budget, and on Windows the Task Manager performance tab shows dedicated GPU memory. Write the numbers down. You will compare them to the model’s footprint in Step 6.
# Linux: GPU, driver, and free system memory
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
free -h
# macOS: unified memory in bytes, then chip name
sysctl -n hw.memsize
system_profiler SPHardwareDataType | grep -E "Chip|Memory"
# Free disk space where the model will live
df -h ~Sample output on a Linux box with a 12 GB card looks like this. Your values will differ, but the shape should match.
name, memory.total [MiB], driver_version
NVIDIA GeForce RTX 4070, 12282 MiB, 560.35.03
total used free shared buff/cache available
Mem: 31Gi 4.2Gi 22Gi 312Mi 5.1Gi 26Gi
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p2 932G 410G 475G 47% /Step 2: Install the build tools and a Python environment
You need Git, CMake, a C++ compiler, and Python. On Ubuntu or Debian, one apt line covers it. On macOS, Xcode command-line tools plus Homebrew do the job. On Windows, Visual Studio Build Tools with the C++ workload works, and WSL2 is a fine alternative if you would rather follow the Linux commands verbatim. For NVIDIA GPUs, also install the CUDA toolkit that matches your driver. Isolate the Python pieces in a virtual environment so you never fight system packages.
# Ubuntu / Debian
sudo apt update
sudo apt install -y git cmake build-essential python3-venv curl
# macOS
xcode-select --install
brew install cmake git python
# Project folder and virtual environment (all platforms with a POSIX shell)
mkdir -p ~/mellum-local && cd ~/mellum-local
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip huggingface_hub requestsStep 3: Download the official GGUF file
JetBrains published an official GGUF repository, JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF, with the repository page showing activity as early as October 7 and updates through October 23 and 24. Always download from the official organization, not a re-upload. Third-party conversions sometimes use stale llama.cpp versions or alter the chat template. The command below pulls only the Q4_K_M file to keep the download near 8 GB instead of fetching every quantization.
cd ~/mellum-local
hf download JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF \
--include "*Q4_K_M*" \
--local-dir ./models
ls -lh ./modelsIf the hf command is not found, your huggingface_hub is older than the version that installs it. Run pip install -U huggingface_hub again, or call python -m huggingface_hub.commands.huggingface_cli download with the same arguments. When the download ends, compare the file size against the 7.5 to 8.1 GB range the third-party reports gave. A file far outside that range usually means you grabbed a different quantization than you intended.
total 7.9G
-rw-r--r-- 1 you you 7.9G Oct 24 10:41 Mellum2.1-12B-A2.5B-Thinking-Q4_K_M.ggufThe filename above is illustrative. Use whatever name the ls command prints for you in the next steps. If Hugging Face answers with an authorization error, create a free access token in your account settings and run hf auth login once. Some other October open-weight releases, such as KRAFTON’s Raon-1 mini, reportedly sit behind a sign-in gate, so a login step is not unusual.
Steps 4-6: Build llama.cpp and Serve Mellum 2.1
Step 4: Clone and compile llama.cpp
Clone the official ggml-org/llama.cpp repository, not a fork. During our research we saw a mirror at cloudlnkcn/llama.cpp listing release numbers that the upstream project does not use, which is a good reminder to check the organization name before you trust a version string. Pick your backend with a CMake flag. CUDA is for NVIDIA, Metal is on by default on Apple Silicon, and Vulkan covers many AMD and Intel GPUs.
cd ~/mellum-local
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# NVIDIA GPU
cmake -B build -DGGML_CUDA=ON
# Apple Silicon: Metal is enabled by default, so a plain configure works
# cmake -B build
# AMD / Intel GPU via Vulkan
# cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j 8
./build/bin/llama-server --versionThe last line prints the build number. Confirm it reads b11747 or higher. If the number is lower, you cloned a stale copy or a pinned tag, so run git pull and rebuild. This single check prevents the most common failure in the whole tutorial. For a deeper walk through build flags and backends, see our llama.cpp tutorial.
Step 5: Run a smoke test with llama-cli
Before you start a server, prove the model loads and generates. The -ngl 99 flag asks llama.cpp to offload as many layers as it can to the GPU. On a CPU-only machine, drop it. The context is small here on purpose, so the test starts fast.
cd ~/mellum-local
./llama.cpp/build/bin/llama-cli \
-m ./models/Mellum2.1-12B-A2.5B-Thinking-Q4_K_M.gguf \
-ngl 99 -c 8192 -n 256 \
-p "Write a Python function that returns the n-th Fibonacci number iteratively."Expect a block of loader logs first. Look for a line naming the architecture and the layer offload count. Then you should see the model think, and finally a code answer. If the program drops into an interactive prompt afterward, type /exit. A “thinking” model often spends the first few hundred tokens reasoning, so a 256-token cap may cut it off before the code appears. Raise -n to 1024 if that happens.
llm_load_print_meta: model type = 12B
llm_load_print_meta: model params = 12.xx B
load_tensors: offloaded 99/99 layers to GPU
...
<think>
The user wants an iterative Fibonacci function. I should handle n = 0 and n = 1 ...
</think>
def fib(n: int) -> int:
a, b = 0, 1
for _ in range(n):
a, b = b, a + b
return aStep 6: Start llama-server on localhost
Now turn the model into an HTTP service that speaks the OpenAI chat format. Bind it to 127.0.0.1 so only your machine can reach it. The --jinja flag tells the server to use the chat template embedded in the GGUF file, which is the safest way to get tool-call formatting right. We use a 32,768-token context as a starting point. The -np 2 flag creates two parallel slots, which helps when an editor and a script talk to the model at once.
cd ~/mellum-local
./llama.cpp/build/bin/llama-server \
-m ./models/Mellum2.1-12B-A2.5B-Thinking-Q4_K_M.gguf \
--host 127.0.0.1 --port 8080 \
-ngl 99 -c 32768 -np 2 --jinjaLeave that terminal running and open a second one. The health endpoint answers first, then a real chat request proves the whole path works. Alternatively, the project documents a -hf flag that downloads and caches a model straight from Hugging Face, for example -hf JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M. We prefer the explicit file path here because you control where the 8 GB lives.
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "mellum",
"messages": [{"role": "user", "content": "Explain what a Python generator is in two sentences."}],
"max_tokens": 1024
}'A healthy reply comes back as JSON with a choices array and a usage block showing prompt and completion token counts. If you see that, the llama.cpp half of the tutorial works. Stop the server with Ctrl+C when you are ready to try Ollama, or keep it running on port 8080 and let Ollama use its default port, 11434. They do not collide, but both will want your GPU memory, so running them at the same time on a 12 GB card is a recipe for out-of-memory errors.
Steps 7-8: Run the Same Model Through Ollama
Step 7: Install or update Ollama
Ollama v0.40.10 is the latest listed release, and the project has been moving fast. Recent builds have added new models, including the Recursive Flint coding model and NVIDIA Nemotron 3.5 variants. Install the newest build, because a stale Ollama is the likeliest reason a brand-new GGUF fails to load. On Linux the official script installs or upgrades in one go. On macOS and Windows, download the installer from the Ollama site. If you want the longer version of this step, our Ollama tutorial walks through installation and service management.
# Linux install or upgrade
curl -fsSL https://ollama.com/install.sh | sh
ollama --version
# expect: ollama version is 0.40.10 (or newer)Step 8: Pull the GGUF and pin a sane context size
Ollama can pull any GGUF straight from Hugging Face with the hf.co/ prefix. The documented tag for this model is the Q4_K_M quantization, so the command mirrors what you downloaded manually. Pulling again duplicates the 8 GB on disk. That is acceptable for a tutorial, and you can delete the manual copy later if you settle on Ollama as your daily runner.
ollama pull hf.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M
# one-off chat to confirm it loads
ollama run hf.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M "Write a bash one-liner that counts lines of Python in a repo."Ollama’s default context window is far smaller than the 131,072 tokens the model supports, and long agent prompts get silently truncated. Fix it with a Modelfile that sets the context and a low temperature. The 0.2 value below is our own starting point for code tasks, not an official JetBrains setting, so adjust it against your results.
cat > Modelfile <<'EOF'
FROM hf.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M
PARAMETER num_ctx 32768
PARAMETER temperature 0.2
EOF
ollama create mellum21 -f Modelfile
ollama listIf the pull fails with an unsupported-architecture message, that is the version problem again. A report on Ollama’s October 15 build workflow shows a bundled llama.cpp bump to b11669, which sits below the b11747 floor for these GGUF files. Newer Ollama releases carry further engine updates, and a workflow named “llama.cpp update” ran again on October 22, but we could not confirm which build v0.40.10 ships. If your Ollama refuses the file after upgrading, use the llama-server path from Steps 4 to 6 until Ollama catches up. It is a normal gap for a model that is two weeks old.
Steps 9-10: Talk to the Server From Python
Step 9: Write a small client that works with either runtime
Both llama-server and Ollama expose an OpenAI-compatible /v1/chat/completions route, so one script serves both. Point it at port 8080 for llama.cpp or 11434 for Ollama through an environment variable. This client sends one prompt, separates the reasoning from the answer, and returns both. Keeping the reasoning is useful while you learn how the model behaves, and you can discard it later. For a production-style gateway with authentication and routing, our Ollama API gateway tutorial goes further than this client does.
# chat.py
import os
import re
import sys
import requests
BASE = os.environ.get("LLM_BASE", "http://127.0.0.1:8080/v1")
MODEL = os.environ.get("LLM_MODEL", "mellum")
def ask(prompt: str, max_tokens: int = 2048, temperature: float = 0.2):
resp = requests.post(
f"{BASE}/chat/completions",
json={
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": max_tokens,
"temperature": temperature,
},
timeout=600,
)
resp.raise_for_status()
data = resp.json()
msg = data["choices"][0]["message"]
content = msg.get("content") or ""
reasoning = msg.get("reasoning_content") or ""
# Some builds return reasoning inline as think tags instead of a separate field
inline = re.search(r"<think>(.*?)</think>", content, re.S)
if inline:
reasoning = reasoning or inline.group(1).strip()
content = re.sub(r"<think>.*?</think>", "", content, flags=re.S).strip()
return content, reasoning, data.get("usage", {})
if __name__ == "__main__":
question = " ".join(sys.argv[1:]) or "Write a Python function that deduplicates a list and keeps order."
answer, thinking, usage = ask(question)
print("--- reasoning chars:", len(thinking))
print(answer)
print("--- usage:", usage)# llama.cpp server
python chat.py "Write a Python function that deduplicates a list and keeps order."
# Ollama (use the name you gave in the Modelfile)
LLM_BASE=http://127.0.0.1:11434/v1 LLM_MODEL=mellum21 python chat.py "Same question"Step 10: Check the output against a real task
A toy prompt tells you the pipe works. A real task tells you whether the model is useful. Pick a function from your own codebase with a known bug, paste it in, and ask for a fix with an explanation. Then run the original tests against the model’s patch. Do this five times with five different bugs before you form an opinion. A model that scores 47.0% on a vendor benchmark can still fumble your framework’s conventions, and a single lucky answer proves nothing.
Step 11: Benchmark Your Own Tokens Per Second
Published speed claims rarely match your hardware, so measure. The script below sends the same five coding prompts several times, records wall-clock time and the completion token count from the server’s usage block, and prints the median tokens per second. Because it works through the standard API, you can run it against llama-server, Ollama, or any other OpenAI-compatible endpoint and compare like for like. Run it once as a baseline before you change a single flag. Every later tweak in this tutorial gets judged against that number.
# bench.py
import statistics
import time
import requests
import os
BASE = os.environ.get("LLM_BASE", "http://127.0.0.1:8080/v1")
MODEL = os.environ.get("LLM_MODEL", "mellum")
RUNS = int(os.environ.get("RUNS", "3"))
PROMPTS = [
"Write a Python function that parses an ISO 8601 date string without external libraries.",
"Explain the difference between a mutex and a semaphore with a short C example.",
"Refactor this loop into a list comprehension: result = []\nfor x in items:\n if x > 0:\n result.append(x * 2)",
"Write a SQL query that returns the top 3 customers by total order value per country.",
"Write a bash script that watches a folder and prints new filenames.",
]
def run_once(prompt: str):
start = time.perf_counter()
r = requests.post(
f"{BASE}/chat/completions",
json={
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 1024,
"temperature": 0.2,
},
timeout=900,
)
r.raise_for_status()
elapsed = time.perf_counter() - start
tokens = r.json()["usage"]["completion_tokens"]
return tokens, elapsed
rates = []
for prompt in PROMPTS:
for _ in range(RUNS):
tokens, elapsed = run_once(prompt)
rates.append(tokens / elapsed)
print(f"{tokens:5d} tokens {elapsed:6.2f}s {tokens / elapsed:6.1f} tok/s")
print("---")
print(f"median: {statistics.median(rates):.1f} tok/s over {len(rates)} runs")
print(f"min: {min(rates):.1f} max: {max(rates):.1f}")The output format looks like the block below. The numbers here are placeholders to show the layout, because speed depends entirely on your GPU, memory bandwidth, quantization and context size. Do not compare your result to them.
612 tokens 14.80s 41.4 tok/s
587 tokens 14.21s 41.3 tok/s
640 tokens 15.37s 41.6 tok/s
...
---
median: 41.3 tok/s over 15 runs
min: 38.9 max: 43.2Keep a log as you go. The table below is a template for the numbers worth recording after each change. A thinking model counts its reasoning tokens in the completion total, so your tokens-per-second figure reflects generation speed, while the time to a finished answer also depends on how long the model reasons.
| Run label | Runtime | Context | Draft / MTP | Median tok/s | Peak memory |
|---|---|---|---|---|---|
| Baseline | llama-server | 32,768 | Off | Your result | Your result |
| Ollama default | Ollama v0.40.10 | Modelfile value | Off | Your result | Your result |
| Bigger context | llama-server | 65,536 | Off | Your result | Your result |
| MTP enabled | llama-server | 32,768 | On | Your result | Your result |
| Two parallel slots | llama-server | 32,768 | Off | Your result | Your result |
Step 12: Try the MTP Draft Head for Faster Single Requests
Multi-token prediction lets the model propose several tokens at once and verify them in a single pass, which can cut latency when one person is waiting on one answer. JetBrains pitches this as the 1.6-times single-request speedup mentioned earlier. The official GGUF repository reportedly gained an MTP head on October 22 and 23, with a draft file and combined builds, and the repository now holds a file such as the MXFP4 MTP draft file in the repo’s mtp folder. The same coverage says these files need a llama.cpp build of b11747 or newer.
Flag names for speculative decoding have changed across llama.cpp releases, so do not copy them from a blog post, including this one. Ask your own build what it supports, then follow the model card’s MTP section for the exact invocation. The commands below download the draft file and list the relevant options.
cd ~/mellum-local
hf download JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF \
--include "mtp/*" \
--local-dir ./models
ls -lh ./models/mtp
# discover the draft / speculative / MTP options your build supports
./llama.cpp/build/bin/llama-server --help | grep -i -E "draft|spec|mtp"Restart the server with the options the model card names, rerun bench.py with the same RUNS value, and log the median in your table. Expect the gain to show up in single-request latency and to shrink when you push several requests at once, since parallel slots already keep the hardware busy. If the numbers get worse, the likeliest cause is a draft file that does not match your main quantization, or a GPU with no spare memory for the extra weights. Turn the draft off and keep the baseline. A speed trick that costs accuracy or stability is not a win for a coding assistant.
Step 13: Connect the Endpoint to Your Editor or Agent
The point of all this is to use the model while you code. Any tool that accepts an OpenAI-compatible base URL can talk to your server. The Aider terminal assistant takes a base URL and a dummy key. Editor extensions that support custom endpoints ask for the same two values: a base URL and a model name. For Ollama, a release note from October 16 says locally installed models now appear in the Codex CLI model picker through ollama launch codex, so that route needs no manual URL at all.
# Aider against llama-server
export OPENAI_API_BASE=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=local
aider --model openai/mellum
# Aider against Ollama
export OPENAI_API_BASE=http://127.0.0.1:11434/v1
aider --model openai/mellum21Give agents a generous max_tokens or let them use the server default, because the reasoning trace counts toward the limit. Start with small, reviewable tasks such as writing a test for one function, and keep the project under version control so you can undo anything. JetBrains describes the model as built for repository work, command execution, file editing and tool use, but a local 12B model is not a drop-in replacement for the largest cloud systems on every task. If you need a broader view of what else runs on a desktop, our roundup of the best local LLMs this month shows where the big open-weight releases land on hardware requirements.
Local Versus Hosted: What Actually Changes
Running Mellum 2.1 yourself changes three things: cost shape, data path and ceiling. On cost, the local runtime itself is free. You pay once for hardware you probably already own, plus electricity. Hosted alternatives meter usage. Ollama’s own cloud plans show the contrast. Its Pro plan lists at $20 per month and includes $60 of monthly usage credits, and JetBrains’ Air Business tier costs $20 per seat per month with AI usage billed separately in credits at one credit per dollar of provider-side cost. Those prices cover different products, so do not read them as a head-to-head. They do show that cloud AI is a recurring line item while a local model is a sunk cost.
On the data path, a local endpoint keeps prompts and source files on your machine. That matters for client code under NDA, for regulated industries, and for anyone who has read a vendor’s data-retention page and winced. You still carry the responsibility. A laptop that syncs your project folder to a cloud drive, or an editor extension that phones home, can leak the same files your private model never saw. Audit the whole chain, not only the model.
On the ceiling, be honest about what 2.5 billion active parameters can do. JetBrains built Mellum 2.1 for fast, repeatable agent work on repositories, and the company’s own pitch stresses throughput and cost per task. Hosted frontier systems will likely still handle sprawling refactors, ambiguous requirements and unfamiliar frameworks better. A sensible pattern is a split workflow. Send routine edits, test generation, commit messages and code explanations to the local model. Escalate the hard, high-stakes problems to a larger hosted model, with the sensitive files redacted or excluded.
Latency also shifts. A hosted model adds network round trips but runs on datacenter hardware. Your laptop removes the round trip and adds thermal limits. On long sessions, sustained load can throttle a thin-and-light machine, and your tokens-per-second figure will sag. If you plan to work for hours, rerun the benchmark after twenty minutes of continuous use and log the second result next to the first. That number, not the cold-start one, predicts how the setup feels late in the afternoon.
Common Pitfalls That Waste an Afternoon
Most failed setups trace back to the same six mistakes. Read them before you start debugging.
- Grabbing a Mellum 2 file by accident. JetBrains retired Mellum 2 as the current model on October 8. Older quantized artifacts do not match the Mellum 2.1 architecture or its reported coding gains. Check that the repository name contains “Mellum2.1”.
- Confusing Ollama’s version with llama.cpp’s. Ollama v0.40.x and llama.cpp build b11747 are separate version lines. Meeting one requirement says nothing about the other.
- Assuming the full 131,072 tokens come free. The window is a ceiling, not a default. The cache for a long context can use several gigabytes, so raise it step by step while you watch memory.
- Starving the reasoning budget. A thinking model spends tokens before it writes code. A
max_tokensof 256 can end the response inside the reasoning trace and leave you with an empty answer. - Treating 47.0% as your result. The SWE-bench Verified figure is self-reported by JetBrains and dated October 8, 2026. Your code, language and prompts will land somewhere else.
- Exposing the server to the network. Binding to 0.0.0.0 with no key gives anyone on your Wi-Fi a free model and a view into your prompts. Keep 127.0.0.1 unless you add an API key and a firewall rule.
One more wrinkle is worth a sentence. When we checked, the official model card still described GGUF builds for llama.cpp, Ollama and LM Studio as coming soon, while the separate GGUF repository already existed and carried recent commits. Documentation lags releases in the first weeks. Trust the repository contents and the commit history over a stale paragraph, and recheck both before you file a bug.
Troubleshooting: 10 Problems and Fixes
- Unknown-architecture error when loading the GGUF. Your llama.cpp is older than build b11747. Run
git pullin the llama.cpp folder, rebuild with CMake, and confirm withllama-server --version. - Ollama pull succeeds but run fails with an engine error. Upgrade Ollama to the newest release. If it still fails, serve the model with llama-server until Ollama’s bundled engine catches up.
- CUDA out of memory at startup. Lower the context with
-c 16384, or reduce GPU offload with a smaller-nglvalue so some layers run on the CPU. Close other GPU programs, including a second model server. - Generation crawls at a few tokens per second. The model probably runs on the CPU. Check the loader log for the offloaded layer count, confirm you built with the right backend flag, and make sure
-nglis set. - The answer is empty or ends mid-thought. Raise
max_tokens. The reasoning trace consumed the budget before the answer started. - Think tags clutter your editor’s output. Strip them in your client, as
chat.pydoes, or switch to a build that returns reasoning in a separate field. - Port already in use. Another server holds 8080. Find it with
lsof -i :8080or pick another port with--port 8081. - Hugging Face returns 401 or 403. Log in with
hf auth loginand a read token, and confirm you used the official JetBrains organization name. - Ollama ignores your context setting. You ran the raw model name instead of the one built from your Modelfile. Run
ollama listand usemellum21, not thehf.co/name. - MTP makes things slower or crashes. The draft file may not match your main quantization, or your build predates the feature. Remove the draft flags, rebuild from the latest main, and compare against your baseline.
If none of these match, rerun the server with verbose logging and read the first error line, not the last. Loader errors cascade, and the cause is almost always the first message.
Advanced Tips: Tuning, Persistence and Safety
Keep the server alive across reboots. On Linux, a systemd unit restarts llama-server after a crash and starts it at boot. Create the file below as a user service, adjust the paths, and enable it. Running as a normal user keeps the model files out of root’s hands.
# ~/.config/systemd/user/mellum.service
[Unit]
Description=Mellum 2.1 llama-server
After=network.target
[Service]
ExecStart=%h/mellum-local/llama.cpp/build/bin/llama-server \
-m %h/mellum-local/models/Mellum2.1-12B-A2.5B-Thinking-Q4_K_M.gguf \
--host 127.0.0.1 --port 8080 -ngl 99 -c 32768 -np 2 --jinja
Restart=on-failure
[Install]
WantedBy=default.target
# enable it
systemctl --user daemon-reload
systemctl --user enable --now mellum.serviceAdd an API key. The llama-server --api-key option makes the server reject requests without a matching bearer token. Even on localhost it stops a rogue browser tab or local process from using your model. Set the same key in your editor or Aider config.
Shrink the cache before you shrink the model. If memory is tight, quantizing the key-value cache saves more than dropping to a smaller weight quantization in many setups. Look for the cache-type options in llama-server --help. Some builds require flash attention for one of the cache types, so test on a small prompt first and rerun the benchmark to confirm quality did not slip.
Use parallel slots with intent. The -np setting divides your context across slots, so two slots at 32,768 tokens need a total context of 65,536 if each slot should keep 32,768. Match the slot count to the number of tools that really call the model at once. JetBrains’ own throughput claim concerns heavy concurrent load, and most laptops never see that pattern.
Compare against alternatives on your tasks. A model that fits your laptop is only useful if it solves your problems. Run the same five bug-fix tasks through Mellum 2.1 and through a second local model, then log which one passes your tests. Our walkthrough for running Nemotron 3.5 Lightning locally gives you a second candidate, and our ranking of the best AI models for coding shows where hosted systems sit when you need more power than a laptop offers. For more guides in this area, browse the AI and machine learning section.
Mind the license. The model ships under Apache 2.0 with open weights on Hugging Face, which is permissive. Still, read the license text and the model card yourself before you build a commercial service on top, because a license label does not cover every hosting condition. When a company policy requires an audit trail, log prompts on your own server instead of relying on client-side history.
FAQ: Running Mellum 2.1 Locally
How much RAM do I need to run Mellum 2.1?
Third-party testing puts the real floor at 16 GB of memory for the Q4 build, which is roughly 8 GB on disk. A separate guide argues the model fits a 12 GB graphics card with a modest context. These are estimates, not JetBrains specifications, so measure peak memory on your own machine.
Can I run Mellum 2.1 without a GPU?
Yes, llama.cpp runs on the CPU. Because only about 2.5 billion parameters are active per token, a mixture-of-experts model is friendlier to CPU inference than a dense 12B model, though it will still be slower than a GPU setup. Expect to lower the context and be patient with long prompts.
Should I use llama.cpp or Ollama?
Use llama.cpp when you want every flag, including speculative decoding and parallel slots. Use Ollama when you want one command, easy model switching and editor integrations. This tutorial shows both, and the Python client works with either.
Is Mellum 2.1 free for commercial use?
JetBrains releases the model under the Apache 2.0 license with open weights on Hugging Face. That license is generally permissive for commercial use, but check the license file and model card for your exact situation, and ask a lawyer if you plan to resell access.
Does the 47.0% SWE-bench score mean it beats cloud models?
No. JetBrains reported 47.0% on SWE-bench Verified for Mellum 2.1, up from 2.0% for Mellum 2, and the figure is self-reported as of October 8, 2026. It shows a large improvement over the previous generation, not a ranking against other vendors’ models. Test it on your own tasks.
What is the difference between Mellum 2 and Mellum 2.1?
Mellum 2.1 is the newer release from October 8, 2026 and the successor to the earlier Mellum 2 Thinking model. Mellum 2 is the previous generation. Download only the Mellum2.1 files for this tutorial.
Why does the model print a long reasoning trace before the answer?
It is a “Thinking” variant, which means it works through the problem in text before it answers. Give it a generous token budget and strip or hide the trace in your client if you only want the final code.
Does my code stay private with this setup?
When you bind the server to 127.0.0.1 and run the model on your own hardware, prompts do not leave your machine. Ollama’s local runtime is free, and its separate cloud service is billed apart, so make sure you run a local model name and not a cloud variant when privacy matters.
