📢 แจ้งเตือน: ollama api tutorial local endpoint gateway - สรุปดราม่าล่าสุด

October 22, 2026

25 min read

The Ollama API is what turns a model sitting on your disk into something every script, notebook and internal tool can call. That matters more this month than it did in the spring. DeepSeek published open weights for V4.1-Flash-1017 on October 17, 2026, and one hardware breakdown puts the download at about 510 GB, reportedly. Almost nobody will load that on a desk. Your code, though, should not care whether a model runs on your GPU or on a rented one, and a clean API layer is how you get there.

Developers already lean on Ollama for this kind of plumbing. The 2025 Stack Overflow Developer Survey reported that Ollama is among the tools developers building AI agents use for orchestration or framework-related work. This tutorial gives you the full path in 13 steps: raw curl calls, the official Python library, the OpenAI-compatible endpoint, structured JSON output, and a small FastAPI gateway that adds API keys, rate limits and request logs in front of Ollama. Every line of the gateway was run against a mock upstream while writing this piece.

You do not need a GPU for the first half. A 4 GB model on a laptop CPU is enough to exercise every endpoint. If you have not installed anything yet, skim our Ollama self-hosting walkthrough first, then come back here for the API layer. Budget about 60 minutes end to end.

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

What the Ollama API Offers in October 2026

Ollama exposes the same HTTP interface in two places. The local server listens at http://localhost:11434 and needs no authentication. The hosted service at https://ollama.com requires an API key sent as a Bearer token. Ollama’s documentation lists three client-facing flavors on each side: the native /api routes, an OpenAI-compatible /v1 surface, and an Anthropic-style base URL for clients that append /v1/messages themselves.

That symmetry is the whole point. Write your application against the local base URL today, and switching to a bigger hosted model later means changing a host and adding a header. Ollama says it now serves 8.9 million developers and has raised $88M, which tells you the hosted side is not a side project. The company’s own Ollama blog tracks the cloud-model releases, including the October 14 release (v0.40.5) that made decision models such as clef:cloud available without a local download.

Here are the routes you will touch in this tutorial, and whether the gateway we build later will let them through.

Method and path	What it does	Gateway exposes it?
POST /api/generate	Single-prompt completion, streams NDJSON by default	Yes
POST /api/chat	Multi-turn chat with a messages array	Yes
POST /api/embed	Returns embedding vectors for one or more inputs	Yes
GET /api/tags	Lists models stored on this machine	Yes
GET /api/ps	Lists models currently loaded in memory	Yes
POST /api/show	Returns model details, template and parameters	No (info leak)
POST /api/pull	Downloads a model from the registry	No (disk and bandwidth abuse)
DELETE /api/delete	Removes a local model	No (destructive)
GET /api/version	Returns the server version	No (not needed)
POST /v1/chat/completions	OpenAI-compatible chat endpoint	Yes

Notice the split. Six rows are safe to hand to a colleague or a front-end service. The other rows can fill your disk, delete models or leak configuration, so the gateway will refuse them even for authenticated callers. If you are still deciding between runtimes, our comparison of Ollama, LM Studio and Jan covers where each one fits.

One more current detail shapes the examples. Ollama’s August 31, 2026 pricing change moved its Pro, Max and Team plans to per-token pricing with usage included on every plan. Hosted models are therefore metered, and your gateway logs become a cost tool as well as a security tool. We come back to that in the cloud section.

Prerequisites and Versions for the Ollama API Tutorial

Everything below was tested on Ubuntu with Python 3.12.3. The Ollama server steps also apply to macOS and Windows, only the service management commands differ. Pin your versions the first time so that a later upgrade does not change behavior under you.

Component	Version used	Why you need it
Ollama	v0.40.8 (latest stable)	The server and CLI. v0.40.9 appeared October 22 as a pre-release, so skip it for now
Python	Not listed	Runs the client scripts and the gateway
FastAPI	Latest release (tested 0.159.1)	Web framework for the gateway
Uvicorn	Latest release (tested 0.54.0)	ASGI server that runs the gateway
httpx	Latest release	Async HTTP client that relays streams to Ollama
ollama (Python library)	Latest release (tested 0.6.3)	Official client with streaming and Pydantic support
openai (Python library)	Latest release	Proves the OpenAI-compatible route works
curl and jq	Any recent build	Raw requests and JSON inspection
RAM or VRAM	8 GB minimum for a small model	Rough rule: parameters in billions times bits per weight divided by 8 gives gigabytes of weights

That last row deserves a worked example. An 8-billion-parameter model at 4-bit precision needs about 4 GB for weights (8 x 4 / 8), plus extra for the context cache. A 30-billion-parameter model at the same precision needs roughly 15 GB. This is arithmetic, not a benchmark, and real files run slightly larger because of metadata and mixed-precision layers. Apply the same math to a 552-billion-parameter model and you see why the October DeepSeek weights are a data-center download rather than a laptop one.

If this is your first time with local inference, the guide to run an LLM locally explains quantization and hardware in more depth than we can here. We keep the focus on the API.

Part 1: Install Ollama, Pull a Model, Make Your First Call (Steps 1-4)

Step 1: Install Ollama and confirm the server is running

On Linux the official installer script does the work. Pin the version through the OLLAMA_VERSION variable so you land on the stable build rather than whatever tag is newest. Our install steps for each operating system cover the macOS and Windows installers.

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.40.8 sh

ollama --version
curl -s http://localhost:11434/
curl -s http://localhost:11434/api/version

A healthy install prints a version line, then a plain-text banner, then a small JSON object. Sample output:

ollama version is 0.40.8
Ollama is running
{"version":"0.40.8"}

If the second command fails with a refused connection, the service is not running. On Linux, check systemctl status ollama. On macOS and Windows, open the Ollama app once so it starts the background server.

Step 2: Pull a model and read its tag

A model reference has two parts, a name and a tag, separated by a colon. The tag usually encodes the parameter count, for example qwen3:8b. Ollama’s own API examples use that exact tag, and it should fit in 8 GB of memory at the default quantization, so it works as a safe starting model. Swap in a newer tag from the Ollama library any time, since nothing in this tutorial depends on a specific model.

ollama pull qwen3:8b
ollama list

NAME         ID              SIZE      MODIFIED
qwen3:8b     a1b2c3d4e5f6    5.2 GB    12 seconds ago

The listing above is sample output. Your ID and size will differ. Pay attention to tags that end in -cloud or :cloud. Those run on Ollama’s servers, not on your machine, and they show up in ollama list with almost no disk footprint. Reports say the newest DeepSeek V4.1 Flash builds appeared in Ollama as cloud tags that run on Ollama’s servers. Treat any tag with that suffix as a remote call that sends your prompt off the box.

Step 3: Make your first call with curl

Start with /api/generate, the simplest route. It takes a model name and a prompt. By default it streams, which means the server sends one JSON object per line as tokens arrive.

curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Why is the sky blue? Answer in one sentence."
}'

Ollama documents this stream as newline-delimited JSON with the application/x-ndjson content type. Each line carries a response fragment and a done flag. Sample output, trimmed:

{"model":"qwen3:8b","created_at":"2026-10-22T09:14:02.411Z","response":"Sunlight","done":false}
{"model":"qwen3:8b","created_at":"2026-10-22T09:14:02.428Z","response":" scatters","done":false}
{"model":"qwen3:8b","created_at":"2026-10-22T09:14:02.445Z","response":" off","done":false}
...
{"model":"qwen3:8b","created_at":"2026-10-22T09:14:03.902Z","response":"","done":true,"done_reason":"stop"}

Now the same idea with chat. The /api/chat route takes a messages array, which is how you carry conversation history. Set stream to false to get one tidy JSON object back instead of a stream.

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "stream": false,
  "messages": [
    {"role": "system", "content": "You answer in plain English."},
    {"role": "user", "content": "Explain an API key in two sentences."}
  ]
}'

Step 4: Read the response fields and compute tokens per second

The final object in a stream, or the single object when streaming is off, carries timing and token counts. All durations are in nanoseconds, a detail that trips up nearly everyone the first time.

Field	Meaning	Unit
message.content	The assistant reply (chat route)	text
response	The generated text (generate route)	text
done	True on the last object	boolean
done_reason	Why generation stopped, usually stop or length	text
total_duration	Whole request, including model load	nanoseconds
load_duration	Time spent loading the model into memory	nanoseconds
prompt_eval_count	Tokens in your prompt	tokens
prompt_eval_duration	Time spent reading the prompt	nanoseconds
eval_count	Tokens the model generated	tokens
eval_duration	Time spent generating	nanoseconds

Divide eval_count by eval_duration in seconds and you have generation speed. Let jq do the math:

curl -s http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "stream": false,
  "messages": [{"role": "user", "content": "List three uses of embeddings."}]
}' | jq '{tokens: .eval_count, seconds: (.eval_duration / 1e9), tokens_per_second: (.eval_count / (.eval_duration / 1e9))}'

{
  "tokens": 212,
  "seconds": 4.9,
  "tokens_per_second": 43.27
}

Those figures are illustrative, not a benchmark of any particular machine. Run it yourself and keep the number. When something feels slow later, this is your baseline.

Part 2: Control Output, Stream in Python, Use the OpenAI Client (Steps 5-8)

Step 5: Tune context, temperature and keep-alive per request

Every request can carry an options object. The one that bites hardest is num_ctx. Ollama’s FAQ states that the default context window is 4096 tokens, and you can raise it per request or server-wide with the OLLAMA_CONTEXT_LENGTH environment variable. If you paste a long document into a chat and the model seems to forget the start of it, you probably hit that ceiling.

curl -s http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "stream": false,
  "keep_alive": "30m",
  "options": {
    "num_ctx": 8192,
    "temperature": 0.2,
    "seed": 42
  },
  "messages": [{"role": "user", "content": "Summarize what a context window is."}]
}' | jq -r '.message.content'

Three settings in one request. A larger context costs memory, because the key-value cache grows with it. A low temperature makes answers more repeatable. A fixed seed makes them reproducible on the same model and hardware. The keep_alive field controls how long the model stays in memory after the call. The default is five minutes, so a quiet endpoint pays a cold-start penalty every time someone returns after lunch.

Step 6: Stream tokens with the official Python library

Install the library inside a virtual environment, then iterate over the stream. The library mirrors the REST API closely, so everything you learned with curl transfers.

python3 -m venv venv
source venv/bin/activate
pip install ollama

from ollama import Client

client = Client(host="http://localhost:11434")

stream = client.chat(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Give me a one-line definition of REST."}],
    stream=True,
)

for part in stream:
    print(part["message"]["content"], end="", flush=True)
print()

The flush=True matters. Without it, terminal buffering hides the streaming effect and you will think the call hangs until the end. The library also ships an async client for web servers, which you would use inside a FastAPI route that calls Ollama directly.

Step 7: Call Ollama with the OpenAI client library

Ollama implements a subset of the OpenAI API, so existing code written for hosted models can point at your machine. Set base_url to the /v1 path and pass any string as the key. Ollama’s docs say the client requires a key value but the local server ignores it. If you already followed our OpenAI API key setup for GPT-6.1 Sol, the switch is a two-line change.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",  # required by the client, ignored by the local server
)

reply = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Say this is a test."}],
)
print(reply.choices[0].message.content)

Expect gaps. A subset means some parameters and endpoints will not behave exactly like the hosted original. Test the specific features you depend on, such as tool calls or log probabilities, before you move production traffic.

Step 8: Force valid JSON with a schema

Free-form text is a poor contract between a model and a program. Ollama’s structured outputs feature accepts a JSON schema in the format field and constrains the reply to match it. The documentation recommends also describing the schema in the prompt to ground the model. Pydantic makes the round trip painless.

from ollama import chat
from pydantic import BaseModel


class Ticket(BaseModel):
    title: str
    severity: str
    needs_human: bool


response = chat(
    model="qwen3:8b",
    messages=[{
        "role": "user",
        "content": "Triage this report as JSON: 'Checkout page returns 500 for all EU users.'",
    }],
    format=Ticket.model_json_schema(),
)

ticket = Ticket.model_validate_json(response.message.content)
print(ticket)

title='Checkout page returns 500 for EU users' severity='high' needs_human=True

Validation is your safety net. If the model drifts, model_validate_json raises an error you can catch and retry. One caveat from Ollama’s documentation: the hosted cloud service does not currently support structured outputs, so this step works against local models only.

Part 3: Lock Down the Network Before You Expose Anything (Step 9)

Step 9: Keep Ollama on loopback and check what is listening

Ollama’s authentication page is blunt: the local API at port 11434 does not require authentication. That is fine while the server binds to 127.0.0.1. It becomes a problem the moment someone sets OLLAMA_HOST=0.0.0.0 to make a phone or a second PC work. Anyone who can reach that port can run prompts on your GPU, list your models and, if you did not filter anything, pull or delete them.

Configure the server through environment variables. On Linux with systemd, Ollama’s FAQ describes using systemctl edit to add them.

sudo systemctl edit ollama.service

# In the editor, add:
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Then verify with the socket table instead of trusting your config:

ss -ltnp | grep 11434

LISTEN 0 4096 127.0.0.1:11434 0.0.0.0:* users:(("ollama",pid=1423,fd=3))

Look at the address column. A loopback address on port 11434 is private. 0.0.0.0:11434 or *:11434 means every network interface answers. The table below lists the environment variables you are most likely to adjust.

Variable	What it controls	Typical value
OLLAMA_HOST	Bind address and port of the server	Not listed
OLLAMA_KEEP_ALIVE	How long an idle model stays loaded	A duration, or -1 to pin
OLLAMA_CONTEXT_LENGTH	Default context window in tokens (default 4096)	Not listed
OLLAMA_MODELS	Directory where model files live	/data/ollama/models
OLLAMA_NUM_PARALLEL	Requests handled at once per loaded model	2 to 4
OLLAMA_MAX_LOADED_MODELS	Models kept in memory at the same time	1 on small GPUs
OLLAMA_ORIGINS	Browser origins allowed to call the API (CORS)	https://app.example.com

Now you see why a gateway makes sense. Ollama stays on loopback, the gateway listens where you choose, and only the gateway checks keys, counts requests and decides which routes exist.

Part 4: Build the Gateway Project (Steps 10-11)

Step 10: Create the project and a key generator

The gateway is a single FastAPI file plus a helper that mints keys. We never store raw keys, only their SHA-256 hashes, so a leaked keys.json does not leak working credentials. Keys are long random strings generated with Python’s secrets module, which is why a fast hash is enough here and a slow password hash is not required.

mkdir ollama-gateway && cd ollama-gateway
python3 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn httpx ollama openai
pip freeze | grep -i -E "fastapi|uvicorn|httpx|ollama|openai" > requirements.txt

# mkkey.py
import hashlib
import json
import secrets
import sys
from pathlib import Path

KEYS_FILE = Path("keys.json")


def main() -> None:
    if len(sys.argv) != 2:
        sys.exit("usage: python mkkey.py <client-name>")
    name = sys.argv[1]
    keys = json.loads(KEYS_FILE.read_text()) if KEYS_FILE.exists() else {}
    token = "olk_" + secrets.token_urlsafe(32)
    keys[name] = hashlib.sha256(token.encode()).hexdigest()
    KEYS_FILE.write_text(json.dumps(keys, indent=2))
    KEYS_FILE.chmod(0o600)
    print(f"Key for {name} (shown once): {token}")


if __name__ == "__main__":
    main()

Mint a key for your first client. Copy it now, because the script prints it exactly once.

python mkkey.py alice

Key for alice (shown once): olk_dZZIs8efkOZSOvGR636i-73kWaKUq0vDWyWdrvkOECw

That key is a throwaway from a test run, so do not reuse it. Export your own real one into a shell variable called GATEWAY_KEY for the tests in Step 12.

Step 11: Write gateway.py

Read the code in four passes. First, the ALLOWED set is a strict allowlist of method and path pairs. Anything else gets a 404, even for valid keys. Second, authenticate hashes the presented token and compares it against every stored hash with hmac.compare_digest, which avoids timing differences. Third, check_rate keeps a sliding one-minute window per client name. Fourth, the relay generator streams bytes from Ollama to the caller as they arrive, so token-by-token output survives the extra hop. If FastAPI itself is new to you, our FastAPI REST API tutorial covers routing and dependency basics.

# gateway.py
import hashlib
import hmac
import json
import logging
import os
import time
from collections import defaultdict, deque
from pathlib import Path

import httpx
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse, StreamingResponse

UPSTREAM = os.getenv("OLLAMA_URL", "http://127.0.0.1:11434")
KEYS_FILE = Path(os.getenv("GATEWAY_KEYS_FILE", "keys.json"))
RATE_LIMIT = int(os.getenv("GATEWAY_RPM", "30"))
MAX_BODY = int(os.getenv("GATEWAY_MAX_BODY", str(1_000_000)))

ALLOWED = {
    ("POST", "/api/chat"),
    ("POST", "/api/generate"),
    ("POST", "/api/embed"),
    ("GET", "/api/tags"),
    ("GET", "/api/ps"),
    ("POST", "/v1/chat/completions"),
    ("POST", "/v1/embeddings"),
    ("GET", "/v1/models"),
}

logging.basicConfig(level=logging.INFO, format="%(message)s")
log = logging.getLogger("gateway")
logging.getLogger("httpx").setLevel(logging.WARNING)

app = FastAPI(title="Ollama gateway")
client = httpx.AsyncClient(
    base_url=UPSTREAM,
    timeout=httpx.Timeout(connect=5.0, read=300.0, write=30.0, pool=5.0),
)
hits: dict[str, deque] = defaultdict(deque)


def load_keys() -> dict[str, str]:
    return json.loads(KEYS_FILE.read_text()) if KEYS_FILE.exists() else {}


def authenticate(request: Request) -> str:
    header = request.headers.get("authorization", "")
    scheme, _, token = header.partition(" ")
    if scheme.lower() != "bearer" or not token:
        raise HTTPException(401, "Missing bearer token")
    digest = hashlib.sha256(token.encode()).hexdigest()
    for name, stored in load_keys().items():
        if hmac.compare_digest(digest, stored):
            return name
    raise HTTPException(401, "Invalid API key")


def check_rate(name: str) -> None:
    now = time.monotonic()
    window = hits[name]
    while window and now - window[0] > 60:
        window.popleft()
    if len(window) >= RATE_LIMIT:
        raise HTTPException(429, f"Rate limit of {RATE_LIMIT} requests per minute reached")
    window.append(now)


@app.get("/healthz")
async def healthz():
    return {"ok": True}


@app.api_route("/{path:path}", methods=["GET", "POST"])
async def proxy(path: str, request: Request):
    full = "/" + path
    name = authenticate(request)
    if (request.method, full) not in ALLOWED:
        raise HTTPException(404, "Endpoint not exposed by this gateway")
    check_rate(name)

    body = await request.body()
    if len(body) > MAX_BODY:
        raise HTTPException(413, "Request body too large")

    upstream_request = client.build_request(
        request.method,
        full,
        content=body or None,
        headers={"content-type": request.headers.get("content-type", "application/json")},
    )
    started = time.perf_counter()
    try:
        upstream = await client.send(upstream_request, stream=True)
    except httpx.ConnectError:
        raise HTTPException(502, "Ollama is not reachable")

    async def relay():
        sent = 0
        try:
            async for chunk in upstream.aiter_bytes():
                sent += len(chunk)
                yield chunk
        finally:
            await upstream.aclose()
            log.info(json.dumps({
                "client": name,
                "path": full,
                "status": upstream.status_code,
                "seconds": round(time.perf_counter() - started, 3),
                "bytes": sent,
            }))

    return StreamingResponse(
        relay(),
        status_code=upstream.status_code,
        media_type=upstream.headers.get("content-type"),
    )


@app.exception_handler(HTTPException)
async def http_error(_: Request, exc: HTTPException):
    return JSONResponse({"error": exc.detail}, status_code=exc.status_code)

One design choice deserves a defense. Errors use the same {"error": "..."} shape that Ollama itself returns, so client libraries that parse Ollama errors keep working through the proxy.

Part 5: Run, Test, and Deploy the Gateway (Steps 12-13)

Step 12: Start the gateway and test every failure path

Start the gateway on port 8000. For the rate-limit test, run it with a deliberately low limit of 5 requests per minute.

GATEWAY_RPM=5 uvicorn gateway:app --host 127.0.0.1 --port 8000

In a second terminal, walk through the paths in order. Health first, then every way a request can be refused, then a success.

export GATEWAY_KEY="paste-your-key-here"

# 1. health needs no key
curl -s localhost:8000/healthz

# 2. no key, bad key
curl -s -w ' [%{http_code}]\n' localhost:8000/api/tags
curl -s -w ' [%{http_code}]\n' -H "Authorization: Bearer nope" localhost:8000/api/tags

# 3. valid key, allowed route
curl -s -w ' [%{http_code}]\n' -H "Authorization: Bearer $GATEWAY_KEY" localhost:8000/api/tags

# 4. valid key, blocked route
curl -s -w ' [%{http_code}]\n' -H "Authorization: Bearer $GATEWAY_KEY" \
  -X POST localhost:8000/api/pull -d '{"model":"qwen3:8b"}'

{"ok":true}
{"error":"Missing bearer token"} [401]
{"error":"Invalid API key"} [401]
{"models":[{"name":"qwen3:8b", ...}]} [200]
{"error":"Endpoint not exposed by this gateway"} [404]

Now confirm streaming survives the proxy, then trip the rate limiter with a loop of seven calls:

curl -sN -H "Authorization: Bearer $GATEWAY_KEY" localhost:8000/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Count to three."}]
}'

for i in 1 2 3 4 5 6 7; do
  curl -s -o /dev/null -w '%{http_code} ' -H "Authorization: Bearer $GATEWAY_KEY" localhost:8000/api/tags
done; echo

200 200 200 429 429 429 429

The earlier tags call and the streamed chat already used two of the five slots in the window, so only three loop requests pass before the 429s start. Blocked-route 404s do not count, because the allowlist check runs before the limiter. Wait a minute and the window clears.

Finally, run the Python client that exercises both library styles through the gateway.

# client.py
import os

from ollama import Client
from openai import OpenAI

BASE = os.getenv("GATEWAY_URL", "http://127.0.0.1:8000")
KEY = os.environ["GATEWAY_KEY"]
MODEL = os.getenv("MODEL", "qwen3:8b")

native = Client(host=BASE, headers={"Authorization": f"Bearer {KEY}"})
print("Native client, streaming:")
for part in native.chat(
    model=MODEL,
    messages=[{"role": "user", "content": "Name one benefit of local inference."}],
    stream=True,
):
    print(part["message"]["content"], end="", flush=True)
print()

compat = OpenAI(base_url=f"{BASE}/v1", api_key=KEY)
reply = compat.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print("OpenAI client:", reply.choices[0].message.content)

Native client, streaming:
Your data never leaves your machine.
OpenAI client: Hello! How can I help you today?

The official Ollama client accepts a headers argument, which is how the Bearer token rides along. The OpenAI client sends the same header automatically from api_key. The server log shows one JSON line per call:

{"client": "alice", "path": "/api/chat", "status": 200, "seconds": 2.214, "bytes": 1840}
{"client": "alice", "path": "/v1/chat/completions", "status": 200, "seconds": 1.032, "bytes": 612}

Step 13: Run it as a service with TLS in front

A terminal session is not a deployment. Create a dedicated user, copy the project to /opt/ollama-gateway, and give it a systemd unit. Bind the gateway to loopback and let a reverse proxy handle TLS, so keys never cross the network in clear text.

# /etc/systemd/system/ollama-gateway.service
[Unit]
Description=Ollama gateway
After=network-online.target ollama.service

[Service]
User=ollama-gw
WorkingDirectory=/opt/ollama-gateway
Environment=OLLAMA_URL=http://127.0.0.1:11434
Environment=GATEWAY_RPM=30
ExecStart=/opt/ollama-gateway/venv/bin/uvicorn gateway:app --host 127.0.0.1 --port 8000
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now ollama-gateway
journalctl -u ollama-gateway -f

Caddy is the shortest path to TLS. This site block terminates HTTPS and turns off response buffering so streamed tokens reach the client immediately:

llm.example.com {
    reverse_proxy 127.0.0.1:8000 {
        flush_interval -1
    }
}

Because the gateway logs one JSON line per request, usage reporting needs no database. This one-liner totals requests and bytes per client from the journal:

journalctl -u ollama-gateway -o cat --since today \
  | grep '^{' \
  | jq -s 'group_by(.client) | map({client: .[0].client, requests: length, bytes: (map(.bytes) | add)})'

Reach Ollama Cloud Models Through the Same API

Local hardware has a ceiling, and the cloud route lifts it without changing your code. Ollama’s documentation describes two ways in. The first is direct: create an API key, export it as OLLAMA_API_KEY, and call https://ollama.com/api/chat with an Authorization: Bearer header. No local install is required. The second is indirect: run ollama signin once, then use a cloud-suffixed model name against your local server, and Ollama forwards the request.

# Direct, no local Ollama needed
export OLLAMA_API_KEY="your_api_key"
curl https://ollama.com/api/chat \
  -H "Authorization: Bearer $OLLAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma4:31b","stream":false,"messages":[{"role":"user","content":"Why is the sky blue?"}]}'

# Indirect, through your local server
ollama signin
curl http://localhost:11434/api/generate -d '{"model":"gpt-oss:120b-cloud","prompt":"Why is the sky blue?"}'

The second form works through the gateway too, because the gateway forwards to your signed-in local Ollama. That also means your cloud credits fund every client who holds a gateway key, so keep the rate limit tight.

Pricing is the other half of the decision. Ollama’s pricing page lists Pro at $20 a month, or $200 billed annually, and states usage-credit tiers of $60, $300 and $1,000 per month across Pro, Max and Team. Team starts at $500 per month with unlimited users and credits shared across the team. Paid plans send an email reminder at 90% of included monthly usage. Secondary roundups such as the October 2026 local LLM roundup from LLMCheck also flag the community debate about the move toward paid cloud, so read the current pricing page before you commit.

Compare that with going straight to a model vendor. DeepSeek’s API keeps the identifier deepseek-flash for V4.1-Flash-1017, and its DeepSeek API change log says calling method and pricing stayed unchanged with the October 17 update. Reported peak pricing is $0.30 per million input tokens and $1.20 per million output tokens, with lower off-peak rates. Our DeepSeek API key setup for V4.1 Flash shows that direct route, and the Claude API tutorial shows the same Bearer pattern against Anthropic’s hosted API.

A privacy note belongs here. A tag with a cloud suffix sends your prompt to a remote service. Treat it like any hosted API for compliance. The local-only tags are the ones that keep data on your machine. Also keep expectations current on model families. Reports say Alibaba’s Qwen 4 is still in training with no published weights, so any tutorial claiming a Qwen 4 tag today is guessing.

Common Pitfalls When Calling Ollama

Exposing the server to the network without a gateway. The local API has no authentication. Exposing it hands strangers your GPU and your model library. Keep OLLAMA_HOST on loopback and verify with ss -ltnp.
Treating a stream as one JSON document. The default response is newline-delimited JSON. Calling response.json() on it fails with an extra-data error. Set "stream": false or parse line by line.
Forgetting the 4096-token default context. Long prompts get truncated without an error. Set num_ctx per request or OLLAMA_CONTEXT_LENGTH on the server, and budget memory for the larger cache.
Mistaking a cloud tag for a local model. A -cloud or :cloud suffix means the prompt leaves your machine and may consume paid usage credits. Check ollama list and your invoice, not the model name alone.
Reading nanoseconds as milliseconds. Every duration field is in nanoseconds. A total_duration that looks enormous is usually just a few seconds, not millions of anything useful.
Mixing up /api and /v1. The native library wants the bare host. The OpenAI client wants the /v1 base URL. Putting /v1 on the native host or leaving it off the OpenAI base URL produces 404 errors that look like missing models.
Installing a pre-release. Ollama v0.40.9 appeared on October 22, 2026 marked as a pre-release. Pin the stable v0.40.8 with OLLAMA_VERSION until the newer build is promoted.
Ignoring cold starts. After the keep-alive window expires, the next call reloads the model from disk. Raise OLLAMA_KEEP_ALIVE or send a warm-up request on a schedule.

Troubleshooting Ollama and the Gateway

Work through this table top to bottom. Most failures come from the first five rows.

Symptom	Likely cause	Fix
curl: connection refused on 11434	The Ollama service is not running or is bound to another address	Run `systemctl status ollama`, start it, and check `OLLAMA_HOST`
{“error”:”model ‘x’ not found”} with 404	The tag was never pulled, or has a typo	Run `ollama list`, then `ollama pull` the exact tag
Gateway rejects the request for a missing bearer token	The client sent no Authorization header	Add `-H "Authorization: Bearer $GATEWAY_KEY"` or set the client `api_key`
Gateway rejects the request for an invalid API key	Key typo, or `keys.json` was regenerated	Mint a new key with `mkkey.py` and confirm `GATEWAY_KEYS_FILE` points to the right file
429 Rate limit reached	The client exceeded the per-minute window	Wait 60 seconds or raise `GATEWAY_RPM` for trusted clients
Gateway reports that Ollama is not reachable	The gateway cannot connect to `OLLAMA_URL`	Start Ollama, then test `curl http://127.0.0.1:11434/api/version` from the gateway host
First request takes 20 seconds or more	Cold model load from disk	Raise `keep_alive`, use a smaller model, or put models on an SSD
Replies are slow and `ollama ps` shows CPU	The model does not fit in VRAM and spilled to system memory	Pick a smaller tag or lower `num_ctx`, then re-check the PROCESSOR column
Tokens arrive all at once	A proxy is buffering the response	Use `flush_interval -1` in Caddy or `proxy_buffering off;` in nginx
JSON parse error: extra data	The client parsed a stream as one object	Set `"stream": false` or read the body line by line
Gateway reports that the request body is too large	The payload exceeds `GATEWAY_MAX_BODY`	Trim the prompt or raise the limit deliberately
Browser blocks the call with a CORS error	The origin is not allowed by Ollama or the gateway	Call from a backend instead, or add a CORS middleware to the gateway and set `OLLAMA_ORIGINS` if you hit Ollama directly

When none of that explains the failure, read the server log. On Linux, journalctl -u ollama -f shows model loads, memory errors and GPU detection messages as they happen. Mid-stream errors are special. Ollama’s documentation notes that a failure after streaming begins arrives as an object with an error property, while the HTTP status stays 200 because the response already started. Check each line for that key.

Advanced Tips for a Production Ollama Endpoint

Per-key model allowlists and budgets

Extend keys.json so each client maps to a hash plus a list of permitted models. In proxy, parse the JSON body for the model field and reject requests for models outside the list. That lets a public demo key reach a small model while your internal key reaches the biggest one. Add a daily request counter next to the per-minute window if you want hard budgets.

Count tokens from non-streaming replies

For calls with stream set to false, the response body is one JSON object that includes prompt_eval_count and eval_count. Parse those in the gateway and write them to the log line. You then know which client burned how many tokens, which matters once cloud usage credits are involved. Streaming replies put the same counts on the final line, so a small line buffer can capture them too.

A short checklist of other upgrades worth making once the basics run:

Raise OLLAMA_NUM_PARALLEL only after measuring memory. Each parallel slot multiplies cache use.
Pin a busy model with keep_alive set to -1 so interactive users never see a cold start.
Batch embedding inputs in one /api/embed call. The route accepts several inputs, which beats one request per sentence.
Move rate-limit state to Redis if you run more than one gateway process, since the in-memory deque is per process.
Rotate keys on a schedule. The mint script makes replacing one a ten-second job.
Point your retrieval pipeline at the endpoint. A LangChain RAG chatbot built on this gateway needs only a base URL and a key.

For wider context on where local inference fits among hosted models, browse our AI and machine learning coverage for model comparisons and release tracking.

Complete Project Recap and Smoke Test

You now have a small, complete project. This is the final layout:

ollama-gateway/
  gateway.py        FastAPI proxy with auth, allowlist, rate limit, logs
  mkkey.py          Mints keys and stores SHA-256 hashes
  client.py         Native and OpenAI-style client demo
  keys.json         Created by mkkey.py (mode 600, never commit it)
  requirements.txt  Pinned dependencies

Keep this smoke test next to the project. Run it after every upgrade of Ollama, the gateway or the model:

#!/usr/bin/env bash
set -u
BASE="${GATEWAY_URL:-http://127.0.0.1:8000}"
: "${GATEWAY_KEY:?set GATEWAY_KEY first}"

check() {
  local want="$1" got="$2" label="$3"
  if [ "$want" = "$got" ]; then echo "PASS $label"; else echo "FAIL $label (wanted $want, got $got)"; fi
}

check 200 "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/healthz")" "health"
check 401 "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/api/tags")" "no key refused"
check 200 "$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $GATEWAY_KEY" "$BASE/api/tags")" "tags"
check 404 "$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $GATEWAY_KEY" -X POST "$BASE/api/pull" -d '{}')" "pull blocked"
check 200 "$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $GATEWAY_KEY" "$BASE/api/chat" -d '{"model":"qwen3:8b","stream":false,"messages":[{"role":"user","content":"ok"}]}')" "chat"

PASS health
PASS no key refused
PASS tags
PASS pull blocked
PASS chat

Five passes mean the server is up, the key check works, the allowlist holds and a real generation completes. From here you can point any OpenAI-compatible tool at https://llm.example.com/v1 with a gateway key and it will behave.

Frequently Asked Questions

Does the Ollama API need an API key?

Not for local use. Ollama’s documentation says the local API at port 11434 requires no authentication. Direct calls to the hosted service at ollama.com do need an API key sent as a Bearer token. The OpenAI client library demands a key string even for local calls, so pass any placeholder.

What is the default port for the Ollama API?

The server listens on port 11434 and, by default, binds to the loopback address 127.0.0.1. Change OLLAMA_HOST to alter either, and verify the result with ss -ltnp.

Is the Ollama API compatible with the OpenAI API?

Partly. Ollama implements a subset of the OpenAI API under the /v1 path, including chat completions. Existing OpenAI client code usually works after you change base_url. Features outside the subset may behave differently, so test the ones you rely on.

How do I stream responses from the Ollama API?

Streaming is the default on routes such as /api/generate and /api/chat. The server sends newline-delimited JSON, one object per line, ending with an object where done is true. Send "stream": false to receive a single JSON response instead.

Can I expose the Ollama API to my network safely?

Not directly. Because the local API has no authentication, put a gateway or authenticating reverse proxy in front of it, keep Ollama on loopback, and terminate TLS at the proxy. The gateway in this tutorial adds keys, rate limits, an endpoint allowlist and request logs.

Which Ollama version should I run for this tutorial?

Version 0.40.8, the latest stable release as of October 21, 2026. A 0.40.9 build appeared on October 22 as a pre-release. Pin the stable one with OLLAMA_VERSION during installation and upgrade once the newer build is promoted.

Can the API run the newest DeepSeek model locally?

Practically, no. DeepSeek published V4.1-Flash-1017 weights on October 17, 2026, and reports put the download at roughly 510 GB. Ollama’s cloud-suffixed tags run such models on remote servers instead. Use a smaller local model for private work and the cloud route for the giants.

How do I count tokens per client?

Read prompt_eval_count and eval_count from the final response object, then add them to the gateway log line. Non-streaming replies carry them in the single JSON body, and streaming replies put them on the last line.

Nadia Dubois

AI & Innovation Editor

Nadia Dubois is the AI & Innovation Editor at Tech Insider, where she tracks the rapid evolution of artificial intelligence, from foundation models to real-world enterprise deployment. She previously covered AI and startups for La Tribune and contributed to MIT Technology Review's European coverage. Nadia specializes in generative AI, AI regulation, and the intersection of technology and European industrial policy. She holds a dual degree in Computational Linguistics and Journalism from Sciences Po Paris.

View all articles