Run DeepSeek Harness With a Local Model: Ollama, vLLM, llama.cpp

dsh boots wired to DeepSeek’s hosted API, and on a fresh install that is the only route it knows. Pointing DeepSeek Harness at a local model is a configuration change rather than a patch: one provider block in settings.yaml and the agent loop runs against Ollama, vLLM, or anything else that speaks the OpenAI chat-completions protocol.

Original content from computingforgeeks.com - post 170966

This guide walks the whole DeepSeek Harness local model route: the four fields a hand-declared provider needs, the credential trap that kills the first run, the compat switches that decide what actually goes on the wire, and the server-side flags vLLM needs before an agent can call a single tool. Every run, error, and timing below came off one Ubuntu 24.04 box with a single RTX 4090, on dsh 0.1.1-rc.2, Ollama 0.32.15, vLLM 0.27.1, and llama.cpp build b10618, in August 2026. The one section sourced from documentation rather than that box says so where it starts.

Prerequisites

You need a host with dsh already working. If it is not installed yet, the steps in getting started with dsh cover the Node floor and the first run. A current Node is not optional: the Ubuntu 24.04 archive still ships 18.19.1, which fails at launch.

You also need one inference server the dsh host can reach over HTTP. Sizing is driven by the model, not by the harness: weights plus KV cache is what has to fit in VRAM, and an agent loop pushes long prompts through it, so budget for a KV cache big enough to hold your context window. An 8B model at Q4 needs roughly 5 GB for weights, an unquantised 4B needs about 8 GB, and both leave room on a 24 GB card. The lab here used one RTX 4090, which is a comfortable floor rather than a requirement. CPU-only works for a chat, and the llama.cpp section below shows why it does not work for an agent.

On versions, the harness declares 22.19 through the end of the 22 line, or 24 and later. Node 23 is excluded outright by that range, and the binding constraint is the harness’s own LLM adapter dependency rather than anything in dsh itself, so a 22.16 box sits below the line even though the CLI looks like it starts. NodeSource 22.x clears it comfortably. The lab box here was a rented GPU container rather than a desktop, which matters only for the CPU numbers later on.

Two small tools are used below. jq is not installed by default, and zstd only comes along reliably on a full install, so take both now:

sudo apt update
sudo apt install -y jq zstd

Declare the local provider in settings.yaml

dsh keeps user configuration in one document at $DSH_HOME/settings.yaml, which defaults to ~/.dsh/settings.yaml. The file does not exist until something writes it, and it is not created by the installer. Two sections matter here: llm-pi-ai holds a dict of provider routes, and agent-default-model decides which route new sessions start on.

mkdir -p ~/.dsh
vim ~/.dsh/settings.yaml

Add a route for the Ollama endpoint. The provider key (ollama here) is the id requests select. Keep it lowercase and hyphenated if you ever want the Web UI to store a key for it, because credential records only accept that form; a route authenticating through apiKeyEnv is not bound by it:

llm-pi-ai:
  providers:
    ollama:
      displayName: Ollama
      apiKeyEnv: OLLAMA_API_KEY
      api: openai-completions
      baseURL: http://127.0.0.1:11434/v1
      models:
        - id: qwen3:8b
          name: Qwen3 8B
          contextWindow: 40960
          maxTokens: 8192
agent-default-model:
  provider: ollama
  model: qwen3:8b

Four things are mandatory on a route the shipped catalog knows nothing about: api, baseURL, a non-empty models list, and unique model ids inside it. Miss one and the section is refused where it is written instead of failing later at request time.

The id has to match what the server itself advertises, character for character. Ollama tags carry a colon, so qwen3:8b is the id, not qwen3. Check against the server before saving:

curl -s http://127.0.0.1:11434/v1/models | jq -r '.data[].id'

Whatever that prints is what belongs in the config. A mismatch surfaces as a 404 straight from the server, wrapped by dsh:

dsh: PI_AI_ERROR: 404: {"message":"model 'hf.co/Qwen/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M' not found","type":"not_found_error","param":null,"code":null}

If Ollama itself is not up yet on this host, the Ollama install steps and the model management commands both apply unchanged; dsh only ever talks to its OpenAI-compatible surface at /v1.

Match the context window to what the server really serves

This is the trap with the highest cost and the lowest visibility, and it is the one section here taken from Ollama’s source and documentation rather than run on the lab box. contextWindow tells dsh how much room it has; nothing interrogates the server, so the number is taken as fact. Ollama meanwhile picks its own default from a VRAM tier at server start, and those tiers are coarse: 262,144 tokens at 47 GiB and above, 32,768 at 23 GiB and above, and 4,096 below that. A 12 GB or 16 GB card therefore serves 4,096 tokens by default while the harness cheerfully builds the 14.7K-token requests measured later in this guide.

Set it explicitly instead of inheriting a tier. The next two blocks are alternatives, not a sequence: pick the foreground server for a quick test, or the drop-in for anything you keep. Foreground means stopping the packaged unit first, or the bind fails with an address already in use:

sudo systemctl stop ollama
OLLAMA_CONTEXT_LENGTH=40960 ollama serve

For the packaged install, stop that foreground server if you started one, keep the service, and put the value in a drop-in so it survives restarts:

sudo systemctl edit ollama.service

Add the environment line under a service section:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=40960"

Then confirm against a resident model rather than trusting the config. The CONTEXT column in ollama ps is the server’s own answer, and that is the number which has to match contextWindow:

sudo systemctl daemon-reload
sudo systemctl restart ollama
ollama run qwen3:8b 'hi' >/dev/null
ollama ps

What happens when the two numbers disagree depends on the model, and neither outcome is good. Most architectures support context shift, so the server quietly drops a block out of the middle of the prompt and the agent keeps answering, slightly dumber, with nothing in the response to say so. DeepSeek-architecture GGUFs do not support shift at all, so the same overflow becomes a hard error telling you the prompt is longer than the available context. The truncation is at least logged, which gives you a free detector on the service path (a foreground server writes the same warning to its own terminal):

journalctl -u ollama | grep "truncating input prompt"

Any hits there mean the agent has been running on a clipped conversation. vLLM behaves better in this respect because --max-model-len is explicit and the engine refuses to start when the KV cache cannot cover it.

Give the route a credential it can resolve

A local server needs no API key, which makes apiKeyEnv look like a field you can leave out. Leave it out and the first run dies.

Error: “PI_AI_ERROR: No API key for provider: ollama”

A route that names no credential defers to the underlying library’s ambient discovery, and for the chat-completions protocol that discovery expects a bearer token. There is nothing to discover, so the request never leaves the box and dsh exits 1. The fix is a reference plus any non-empty value, because nothing on the far side checks it:

export OLLAMA_API_KEY=ollama

Keep the variable name in apiKeyEnv and the exported name identical. Configuration carries the reference, never the literal, so a route naming a variable that resolves to nothing fails with MISSING_CREDENTIAL naming the variable rather than falling back to whatever unrelated key the environment happens to hold. That failure is the friendly one; the ambient-discovery failure above is the one that tells you nothing. For a long-lived setup, put the export in the unit file or shell profile that starts dsh, or type any placeholder into the key field on the Web UI Models page and let the credential service hold it.

Run the first task on the local model

The headless profile is the fastest way to prove the route end to end. It answers one task and exits, so there is no UI state to reason about. Work inside a directory with a file worth reading:

mkdir -p ~/dsh-work
cd ~/dsh-work
printf 'name,qty\nwidget,3\nbolt,12\n' > parts.csv
dsh --profile headless 'Read parts.csv and tell me the total quantity.'

A working local route reads the file with the harness read tool and answers from its contents:

The total quantity from the `parts.csv` file is **15** (3 + 12).

Every run leaves a compressed append-only log under ~/.dsh/sessions/<workspace>/session-<uuid>/session.jsonl.zstd. That log is the fastest way to see which route a session actually used, since the request/context event records the provider, the model, and the context window the harness believed it had.

LATEST=$(ls -t ~/.dsh/sessions/*/session-*/session.jsonl.zstd | head -1)
zstd -dc "$LATEST" | jq -c 'select(.type=="request/context").data' | head -1

Sort by modification time rather than globbing blind, because glob order will happily hand you a session from last week. Every log line is a full event wrapper, so pull the payload instead of grepping raw text. What comes back names the route and the context the harness believed it had, which settles any argument about whether a session really ran locally:

{"provider":"ollama","model":"qwen3:8b","contextWindow":40960}

Set the compat switches

This is the part that decides whether a strict server accepts the request at all, and it is invisible unless you look at the wire. dsh shapes each request from the provider id and the base URL. Your gateway’s URL tells it nothing, so an endpoint it does not recognise is addressed as though it were OpenAI itself.

To see it, put a logging proxy between dsh and the server and read the JSON bodies. Three configurations, same task, same box:

Request field	Plain route	reasoningEfforts plus reasoning: high	Both, with compat set
System prompt role	system	developer	system
Output cap field	max_completion_tokens	max_completion_tokens	max_tokens
Thinking control	absent	reasoning_effort: high	reasoning_effort: high plus thinking: {type: "enabled"}
OpenAI-only extras	store: false	store: false	store: false
Tool schemas sent	25	25	25

Two things stand out. The developer role appears the moment a model declares selectable thinking levels, whether or not an effort is actually being sent, and store goes out on every request whether or not the far end has any concept of stored completions. The reasoning_effort value comes from the route’s reasoning: high default rather than from the levels themselves.

The switches live under compat, settable per route or per model:

llm-pi-ai:
  providers:
    ollama:
      displayName: Ollama
      apiKeyEnv: OLLAMA_API_KEY
      api: openai-completions
      baseURL: http://127.0.0.1:11434/v1
      reasoning: high
      compat:
        supportsDeveloperRole: false
        maxTokensField: max_tokens
        thinkingFormat: deepseek
      models:
        - id: qwen3:8b
          name: Qwen3 8B
          contextWindow: 40960
          maxTokens: 8192
          reasoningEfforts:
            off:
            high: high

supportsDeveloperRole: false sends the system prompt as a system message. maxTokensField: max_tokens swaps the output cap back to the field every OpenAI-compatible server has implemented since 2023. thinkingFormat: deepseek switches the thinking control to the DeepSeek dialect, which is what adds the thinking: {type: "enabled"} object. Under reasoningEfforts, the key is the level a selector offers and the value is the spelling sent on the wire, so max: ultra renames a level for a gateway with its own vocabulary. A bare off: offers the level and sends no effort parameter, except under the DeepSeek dialect this config selects, where it serialises an explicit thinking: {type: "disabled"} instead.

Worth being honest about the result: current Ollama and current vLLM both accepted the unfixed shape and completed the task, developer role and max_completion_tokens included. The switches are insurance against a stricter gateway, and they are the only way to control the thinking dialect, so set them and stop guessing. A switch a protocol does not accept fails resolution and names what that protocol does offer, which makes the failure mode cheap.

Serve the same agent from vLLM

vLLM is the more interesting target because it enforces things Ollama waves through. Start with the flags that matter for agent work, not just the model:

vllm serve Qwen/Qwen3-4B-Instruct-2507 \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.85 \
  --enable-auto-tool-choice --tool-call-parser hermes

Then point a second route at it. Only the endpoint and the model id change:

    vllm:
      displayName: vLLM
      apiKeyEnv: VLLM_API_KEY
      api: openai-completions
      baseURL: http://127.0.0.1:8000/v1
      compat:
        supportsDeveloperRole: false
        maxTokensField: max_tokens
      models:
        - id: Qwen/Qwen3-4B-Instruct-2507
          name: Qwen3 4B Instruct
          contextWindow: 32768
          maxTokens: 8192

Both routes can live in the same file, and the model picker then lists both. Two failures are worth knowing before you hit them.

Error: ‘”auto” tool choice requires –enable-auto-tool-choice and –tool-call-parser to be set’

Leave those two flags off the server and every single agent turn fails. The harness never names a tool choice; vLLM is the one that defaults tool_choice to auto whenever a request carries tools, and then rejects its own default. The error arrives as a 400 that dsh passes through verbatim:

dsh: INVALID_REQUEST: 400: {"message":"\"auto\" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set","type":"BadRequestError","param":null,"code":400}

The parser is model-specific, and picking the wrong one produces an agent that talks about calling tools without ever calling one. hermes is what worked for Qwen3 instruct here, though the vLLM tool calling reference only lists it against the Qwen2.5 family and routes the Qwen3-Coder models to qwen3_xml. Check that table for your own model family before assuming. The full server-side setup lives in the vLLM production install guide.

Error: “4.5 GiB KV cache is needed, which is larger than the available KV cache memory”

Running Ollama and vLLM on one GPU is what causes this. Ollama holds its model resident after a request, vLLM then pre-allocates against what is left, and the engine refuses to start rather than serving a context it cannot honour:

ValueError: To serve at least one request with the model's max seq len (32768), (4.5 GiB KV cache is needed, which is larger than the available KV cache memory (3.57 GiB).

Unload the other model first. Ollama frees the card immediately and reloads on the next request, which took VRAM in use from 9,986 MiB down to 1 MiB on the test box:

ollama stop qwen3:8b
nvidia-smi --query-gpu=memory.used --format=csv,noheader

With the card to itself, vLLM loaded 7.64 GiB of weights in 5.06 seconds and reported a GPU KV cache of 86,752 tokens, which is 2.65 concurrent requests at the configured 32,768-token context. The API took roughly 70 to 100 seconds from launch to answering on /v1/models, so scripts that start the server and immediately fire a task need to poll rather than sleep. If the GPU driver stack is not in place yet, start with the NVIDIA driver and CUDA setup.

llama.cpp works, but not on a CPU build

llama-server speaks the same protocol, so the route is identical apart from the port. Getting the binary is where one packaging detail costs people time: llama.cpp tags a release per build, so the GitHub releases/latest endpoint does not point at the newest one. Ask for the first entry in the list instead, and note the assets are .tar.gz, not zip files:

LC_TAG=$(curl -sL "https://api.github.com/repos/ggml-org/llama.cpp/releases?per_page=1" \
  | grep -m1 '"tag_name"' | grep -o 'b[0-9]\+')   # https://github.com/ggml-org/llama.cpp/releases
echo "$LC_TAG"
curl -sLO "https://github.com/ggml-org/llama.cpp/releases/download/${LC_TAG}/llama-${LC_TAG}-bin-ubuntu-x64.tar.gz"

The tarball unpacks into a directory named after the build, with the binaries and shared objects at its top level. Extract, change into it, and start the server from there:

tar -xzf "llama-${LC_TAG}-bin-ubuntu-x64.tar.gz"
cd "llama-${LC_TAG}"
./llama-server -hf ggml-org/Qwen3-1.7B-GGUF --host 127.0.0.1 --port 8080 -c 16384 --jinja

That --jinja flag is not optional for agent work. Without it, tool calls do not come back in OpenAI shape, and an agent with no tools is a chatbot. With it, the server returns a proper tool call, finish reason included:

finish_reason: tool_calls
tool_calls: [{"type": "function", "function": {"name": "read_file", "arguments": "{\"path\": \"parts.csv\"}"}, "id": "ECiouDPXhQvue8v5dQOFiQBYRwq2L8Gc"}]

The CPU build is where this stops being practical. The prebuilt ubuntu-x64 tarball carries no GPU backend, and on the share-limited container CPU used here, generation ran at 2.23 tokens per second with 7,830 tokens of context. A single agent turn never finished inside a 300-second timeout, because the harness sends the whole tool catalogue on every step. A dedicated desktop CPU will do better than that number, and still not well enough. Use a GPU build, which the llama.cpp guide covers, or keep llama.cpp for interactive chat and give the agent Ollama or vLLM.

The binary’s own version string explains the tag confusion, since it reports a semantic version and the build number together:

version: 0.2.0-dev (build 10618, commit eb25b7263)
built with GNU 11.4.0 for Linux x86_64

Pick the model before you pick the server

Same task and same box for the two GPU-served rows, twice each, with the CPU row included only to show where it lands. Read the caveat before the numbers, because this is not a clean server benchmark:

Server and model	Run 1	Run 2	Notes
Ollama, qwen3:8b (Q4, thinking on)	52.4 s	65.9 s	Correct answer both runs
vLLM, Qwen3-4B-Instruct-2507 (bf16, no thinking)	4.38 s	4.38 s	Correct answer both runs
llama.cpp CPU build, Qwen3-1.7B GGUF	timeout	not run	2.23 tok/s, killed at 300 s

The models differ in size, quantisation, and whether they think, so the gap is not the serving stack alone. That is the actual lesson: a thinking model spends most of a turn on reasoning tokens the agent loop then throws away, and for tool-driven work an instruct model of half the size finishes an order of magnitude faster and gets the same answer. Reach for reasoning when the task needs it, not by default. One disclosure on the Ollama row: those runs predate the context step above, so the server was on its 32,768-token VRAM-tier default while the route declared 40,960, which is exactly the silent mismatch that step exists to close.

The Web UI puts the numbers next to each answer, which makes this easy to check on your own hardware. A two-step task on the 8B model measured 61 tokens per second with a 7.1-second time to first token, 27.7 seconds of LLM time against 0.2 seconds of tool time, and 14.7K input tokens for reading one three-line CSV.

Switch models from the Web UI

Once a route is in the config, the composer’s model selector groups the models by provider, and local routes sit beside the hosted ones with no visual distinction beyond the group label. Picking one also makes it the default for new sessions. A session that has already sent a request keeps the model recorded in its own log, so a switch mid-conversation lands on the next session rather than the current one.

Settings then Models is where the routes report their state. A hand-declared route carries a Custom badge, and the status dot is the quickest health check you have: green means the harness resolved a credential for that route, red means it did not, which is what the stock DeepSeek row looks like on a box with no API key.

You can skip the YAML for the first route entirely. “Add a custom provider” asks for a provider id, a display name, a base URL, the API protocol, and a key, then writes the same section you would have typed. It also insists on at least one model, and offers to fetch the list from the endpoint rather than making you type ids, which is the safest way to avoid the 404 above. Each model row expands to hold its own context window and output cap, so the numbers from the earlier section are reachable here. What the form does not expose is compat and reasoningEfforts, so the wire-shaping work stays in the file. On a loopback connection the panel header carries an “Open configuration file” button as the shortcut; reach the UI across the network and that button is gone.

Edits to settings.yaml are picked up while dsh is running, from either side. The file is watched and writes land atomically, so there is no restart step after adding a route. Comments survive on any map node the write did not touch, with one exception worth knowing before you annotate a config: a changed array is replaced wholesale, and models is an array, so a UI edit to a model list takes the comments inside it with it.

Where the local route bites

The tool catalogue is the hidden cost. Standard mode sent 25 tool schemas on every request on this box, with no MCP servers attached, which is how a three-line CSV becomes 14.7K input tokens; add MCP servers or skills and the count climbs. On a local model you pay that in prompt-processing time rather than in dollars. Trimming the preset is the lever if turns feel slow, and it matters far more than the odd hundred tokens in a system prompt. Comparing dsh with Claude Code covers what the presets contain.

Prompt caching is the other one. The Ollama session measured a 0% cache hit rate start to finish, because the harness reads cached_tokens out of the usage block and Ollama does not report it. vLLM can report it, but only when the server is started with --enable-prompt-tokens-details, which is off by default and absent from the serve command above, so expect the same 0% there until you add it. Nothing breaks either way; a repeated prefix on the Ollama route is simply not free the way it is on the hosted one.

One more trap worth pinning down before you rely on the setup: a model with no declared capacity falls back to a route default of 262,144 tokens, which is a guess, so declare contextWindow on every model even when the server is generous. Images also stay off unless you say otherwise, since a route without defaultInput is text-only. That one we did not exercise; the text path is what every number here covers.