Running a model on your own machine solves the privacy and cost problem, but it creates a visibility problem. Once you self-host the model, nobody tells you why a prompt took 40 seconds or which retrieval step fed it the wrong context. The fix is LLM observability, and the setup that fits a local stack best is self-hosted Langfuse: the current compatibility path is Langfuse v4 on the server with Python SDK v5, which needs a self-hosted server at v4.59.0 or newer.
This tutorial walks through 13 steps, about 50 minutes of work, from an empty Ubuntu box to a traced, scored, locally hosted LLM app. You will boot the full Langfuse stack with Docker Compose, point a local model server at it, record traces with decorators, attach scores, and finish with a complete working project you can drop into your own repo. If you have not set up a model server yet, start with this guide to run a local LLM with Ollama, then come back here.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
Why LLM Observability Matters for Local Models and What Changed in Langfuse v4
Hosted APIs give you a dashboard for free. Local models give you a terminal and a hope. A traced request answers the questions that actually come up in week two of a project: how long did the retrieval step take, how many tokens went into the prompt, did the model hit its output limit, and did last Tuesday’s prompt edit make answers worse or just different?
The demand is not theoretical. McKinsey’s 2025 State of AI survey found that “62% of respondents said their organizations were at least experimenting with AI agents.” Agents multiply the problem because one user request becomes five or ten model calls. Stack Overflow’s 2025 developer survey also reported that 82% of developers used OpenAI GPT models for development work in the past year, per its technology results. Many teams therefore run a hosted model and a local one side by side. You want one place that records every call, whichever runtime serves it. For a wider view of how the model market is moving, see our AI and machine learning coverage.
Langfuse is an open-source, self-hostable platform for debugging and iterating on LLM applications, and it changed a lot in the past quarter. Four changes matter for this tutorial:
- Server v4 is the current generation. Langfuse says v4 is generally available for self-hosted deployments, and Cloud is scheduled to become v4-only on November 16, 2026. Self-hosted users pick their own migration date.
- SDK majors moved on October 14, 2026. Python SDK v5 and JS/TS SDK v6 shipped together. Both require Langfuse Cloud or a self-hosted server at v4.59.0 or newer.
- Deprecated APIs are gone. The v5 and v6 clients dropped interfaces such as
api.legacy.*andapi.ingestion, and the release added experiment links, nested metadata filters, and gzip-compressed span batches. - The data model is observation-first. According to the v4 documentation summary, correlation attributes such as user ID, session ID, metadata, and tags now live on every observation instead of only on the trace.
If you still run self-hosted v3, you cannot use the new SDK majors. Stay on Python SDK v4 (or JS/TS SDK v5) until you upgrade the server. This tutorial targets the current path, so it assumes a fresh v4 install.
What You Will Build
The finished project, which we call TraceLab, has five moving parts. Docker Compose runs Langfuse. A local OpenAI-compatible model server (Ollama or llama.cpp’s llama-server) answers prompts. A small Python package wraps each request in a trace with a retrieval span and a generation span. A test suite replays a prompt file and writes pass/fail and latency scores back to Langfuse. A check script verifies the whole chain before you trust any chart.
Here is what each layer records and why you care:
| Layer | What gets recorded | Question it answers |
|---|---|---|
| Trace | One user request, user ID, session ID, tags | Which conversation went wrong? |
| Retrieval span | Query, returned snippets, duration | Did the model see the right context? |
| Generation | Prompt, completion, model name, token usage | What did it cost in tokens and time? |
| Score | Numeric or boolean label on a trace | Is quality moving up or down between prompt versions? |
| Session | Traces grouped into one conversation | How does a multi-turn chat drift? |
Langfuse also reworked its session view on October 13, 2026 to show user and assistant messages across traces as one continuous conversation, which makes the last row of that table far more useful than it was a month ago.
Prerequisites and Versions
You need a Linux or macOS machine with enough memory to run the Langfuse containers and a model at the same time. Langfuse itself is a web service, a worker, PostgreSQL, ClickHouse, Redis, and object storage, so budget generous headroom before you load a large model. If you are shopping for hardware, an AI laptop such as the Surface Laptop Ultra can run both pieces, though a small model keeps the demo comfortable on anything.
| Component | Version to use | Notes |
|---|---|---|
| Operating system | Ubuntu 24.04 LTS or macOS | Windows works through WSL 2 |
| Docker Engine | 27 or newer, with the Compose plugin | Check with docker compose version |
| Langfuse server | v4.59.0 or newer | Required by Python SDK v5 and JS/TS SDK v6 |
| Langfuse Python SDK | v5 line (current major) | Needs Python 3.11 or newer |
| Python | 3.11 or newer | Python 3.10 is outside the v5 requirements |
| Local model server | Ollama v0.40.8 or newer, or llama.cpp v0.7.0 | Both expose an OpenAI-compatible API |
| Git and OpenSSL | Any current release | Used for cloning and generating secrets |
Two version rules deserve a second look before you start. First, the SDK and server must agree: a v4 server older than v4.59.0 rejects SDK v5. Second, the OpenAI integration no longer supports pre-1.0 OpenAI SDKs, so install a current openai package. If a pinned dependency in your existing project blocks either upgrade, keep that project on Python SDK v4 and run a separate environment for this tutorial.
You also need a model installed locally. Any chat model works, and the tutorial code finds the first model your server lists, so you do not have to hard-code a tag. For a sense of which small open models suit a laptop, compare the options in this small open model comparison.
Steps 1-3: Prepare the Host, Clone Langfuse, and Generate Secrets
The first three steps are boring on purpose. Most failed self-hosted installs trace back to a missing tool or a secret left at its placeholder value, so spend ten minutes here and save an hour later.
Step 1: Check your tooling
Run these five commands and compare the output with the table in the previous section. Anything older than the listed minimum gets upgraded before you continue.
docker --version
docker compose version
python3 --version
openssl version
git --versionTypical output on a prepared Ubuntu 24.04 machine looks like this (your patch numbers will differ):
Docker version 27.x.x, build abcdef1
Docker Compose version v2.x.x
Python 3.12.x
OpenSSL 3.0.x
git version 2.43.xIf python3 --version prints 3.10 or lower, install a newer interpreter first. Python SDK v5 will not install on 3.10, and the error message for that is easy to misread as a network problem.
Step 2: Clone the Langfuse repository
Langfuse publishes a Docker Compose file in its main repository. A shallow clone is enough because you only need the compose definition, not the full history.
mkdir -p ~/tracelab && cd ~/tracelab
git clone --depth 1 https://github.com/langfuse/langfuse.git
cd langfuse
docker compose config --servicesThe last command lists the service names. You should see the web service, the worker, PostgreSQL, ClickHouse, Redis, and an S3-compatible object store. Langfuse’s own guidance published on October 22, 2026 says self-hosters should give each of those pieces an explicit owner, and the service list is a good place to start that conversation with your team. Write the names down, because the override file in Step 3 refers to the web service by name. At the time of writing the web and worker services are called langfuse-web and langfuse-worker, so adjust the override below if your output differs.
Step 3: Replace every placeholder secret
The shipped compose file marks every value you must change with a CHANGEME comment. Find them all:
grep -n "CHANGEME" docker-compose.ymlGenerate fresh values with OpenSSL. Langfuse expects the encryption key to be 64 hexadecimal characters, which is exactly what openssl rand -hex 32 prints.
# 64 hex characters for ENCRYPTION_KEY
openssl rand -hex 32
# Random strings for NEXTAUTH_SECRET, SALT, and database passwords
openssl rand -base64 32
openssl rand -hex 24Open docker-compose.yml and replace each flagged value. Where the same password appears twice (the PostgreSQL service and the connection string in the web and worker services, for example), use the identical value both times. Also change any published port from 3000:3000 to 127.0.0.1:3000:3000 if the machine is reachable from other hosts, because the UI will hold your prompts and completions.
Next, create a .env file beside the compose file for the three application secrets, plus a docker-compose.override.yml that creates your organization, project, API keys, and first user on boot. This headless initialization saves you from clicking through the sign-up flow and lets you script the whole setup later.
cat > .env <<EOF
NEXTAUTH_SECRET=$(openssl rand -base64 32)
SALT=$(openssl rand -base64 32)
ENCRYPTION_KEY=$(openssl rand -hex 32)
LF_PUBLIC_KEY=pk-lf-tracelab-$(openssl rand -hex 8)
LF_SECRET_KEY=sk-lf-tracelab-$(openssl rand -hex 16)
LF_ADMIN_PASSWORD=$(openssl rand -base64 18)
EOF
chmod 600 .env# docker-compose.override.yml
services:
langfuse-web:
environment:
NEXTAUTH_SECRET: ${NEXTAUTH_SECRET}
SALT: ${SALT}
ENCRYPTION_KEY: ${ENCRYPTION_KEY}
LANGFUSE_INIT_ORG_ID: tracelab-org
LANGFUSE_INIT_ORG_NAME: TraceLab
LANGFUSE_INIT_PROJECT_ID: tracelab
LANGFUSE_INIT_PROJECT_NAME: tracelab-local
LANGFUSE_INIT_PROJECT_PUBLIC_KEY: ${LF_PUBLIC_KEY}
LANGFUSE_INIT_PROJECT_SECRET_KEY: ${LF_SECRET_KEY}
LANGFUSE_INIT_USER_EMAIL: [email protected]
LANGFUSE_INIT_USER_NAME: Admin
LANGFUSE_INIT_USER_PASSWORD: ${LF_ADMIN_PASSWORD}
langfuse-worker:
environment:
SALT: ${SALT}
ENCRYPTION_KEY: ${ENCRYPTION_KEY}Keep a copy of the key pair and the admin password in a password manager, because you will paste the keys into the Python project in Step 7. Never commit this .env file to Git.
Steps 4-5: Boot the Stack and Verify It Is Healthy
With secrets in place, the stack starts in one command. The first boot pulls several images and runs database migrations, so give it a few minutes.
Step 4: Start the containers
docker compose up -d
docker compose ps
docker compose logs -f langfuse-webPress Ctrl+C to leave the log view once you see a line announcing that the web server is ready. A healthy docker compose ps shows every service as running, and the databases as healthy. The web container often restarts once or twice while ClickHouse and PostgreSQL finish their migrations. That is normal. If a container keeps restarting for more than five minutes, jump to the troubleshooting section for the matching symptom.
Check which server version you actually pulled. The compose file pins an image tag, and Python SDK v5 needs v4.59.0 or newer:
docker compose images langfuse-web
curl -s http://localhost:3000/api/public/healthStep 5: Verify health and sign in
The health endpoint returns a small JSON document that includes the server version:
{"status":"OK","version":"4.x.y"}If the version starts with 3, the compose file still points at the old generation. Stop the stack with docker compose down (without -v, so you keep data), change the web and worker image tags to the v4 line, and start again. GitHub release notes list v4.68.0 as published on October 22, 2026, so anything near that is current.
Open http://localhost:3000 and sign in with [email protected] and the password from your .env file. You should land inside the tracelab-local project with an empty traces table. Open Settings and confirm the API keys match the values you generated. If the sign-in page offers account creation but no project exists, the initialization variables did not reach the web container, so run docker compose config | grep LANGFUSE_INIT to see what Compose actually resolved.
Langfuse added API key expiration on October 7, 2026, with presets of 30 days, 60 days, 90 days, one year, a custom date, or no expiration. Keys created through initialization variables behave like any other key, so decide now whether you want to rotate them. For a throwaway lab, leaving them alone is fine. For a shared team server, create fresh keys in the UI with an expiry and stop relying on the bootstrap pair.
Steps 6-7: Start a Local Model Server and Set Up the Python Project
Langfuse records what your code sends. It does not care who answers, as long as your code can talk to a server that speaks the OpenAI chat format. Both mainstream local runtimes do.
Step 6: Run a local OpenAI-compatible model server
Option A is Ollama. As of this week the latest release is v0.40.9, published October 22, 2026, and the v0.40 series added automatic MLX execution for supported architectures on Apple silicon. Local serving needs no account and no API key.
ollama --version
ollama pull YOUR_MODEL_TAG
ollama list
curl -s http://localhost:11434/v1/modelsReplace YOUR_MODEL_TAG with any chat model from the library. If you want a current agent-oriented option and have the memory for it, Ollama’s August 11 announcement describes NVIDIA Nemotron 3.5 Lightning as a 30B-parameter model with 3B active parameters. A small model is perfectly fine for this tutorial, since you are testing the plumbing and not the answers. Our Ollama, LM Studio, and Jan comparison covers the other desktop runtimes if you prefer a GUI.
Option B is llama.cpp. Its server exposes an OpenAI-compatible API and a web UI for GGUF files, and the project’s latest tagged release is v0.7.0 from October 15, 2026, with rolling prerelease builds in between.
llama-server -m ./models/your-model.gguf --port 8080 --jinja
curl -s http://localhost:8080/v1/modelsIf you need higher throughput for many concurrent users later, the same tracing code works unchanged against a faster engine, such as the one in our SGLang serving setup. Only the base URL changes.
Step 7: Create the Python project
Back in ~/tracelab, create a virtual environment with Python 3.11 or newer and install the pinned dependencies.
cd ~/tracelab
python3.11 -m venv .venv
source .venv/bin/activate
cat > requirements.txt <<'EOF'
langfuse>=5,<6
openai>=1.50
httpx>=0.27
python-dotenv>=1.0
EOF
pip install -r requirements.txt
pip show langfuse | head -n 2The pip show output should report a 5.x version. If it reports 4.x, your interpreter or an old cache is interfering, and Step 7 is where to fix that rather than three steps later. Now write the project’s .env, copying the keys from the Langfuse .env you created in Step 3:
cat > .env <<'EOF'
LANGFUSE_PUBLIC_KEY=pk-lf-tracelab-REPLACE_ME
LANGFUSE_SECRET_KEY=sk-lf-tracelab-REPLACE_ME
LANGFUSE_HOST=http://localhost:3000
LLM_BASE_URL=http://localhost:11434/v1
LLM_API_KEY=local
LOCAL_MODEL=
EOFLeave LOCAL_MODEL empty and the code will use the first model the server lists. Set it only when you want to pin a specific tag. If the SDK in your installed version prefers a differently named host variable, the v4-to-v5 upgrade notes list the exact environment names.
Steps 8-9: Write the Config and Check Script, Then Record Your First Trace
Two small modules carry the whole project: one that reads settings and builds the traced OpenAI client, and one that proves the chain works. Write them before any business logic, so a broken key shows up as one clear message instead of a silent empty dashboard.
Step 8: Create the config, client, and stack check
mkdir -p tracelab && touch tracelab/__init__.py# tracelab/config.py
import os
from dotenv import load_dotenv
load_dotenv()
LLM_BASE_URL = os.getenv("LLM_BASE_URL", "http://localhost:11434/v1")
LLM_API_KEY = os.getenv("LLM_API_KEY", "local")
LOCAL_MODEL = os.getenv("LOCAL_MODEL", "").strip()
PROMPT_VERSION = os.getenv("PROMPT_VERSION", "v1")# tracelab/llm.py
from .config import LLM_API_KEY, LLM_BASE_URL, LOCAL_MODEL
import httpx
from langfuse.openai import OpenAI
client = OpenAI(base_url=LLM_BASE_URL, api_key=LLM_API_KEY, timeout=180)
def resolve_model() -> str:
if LOCAL_MODEL:
return LOCAL_MODEL
resp = httpx.get(f"{LLM_BASE_URL}/models", timeout=10)
resp.raise_for_status()
models = resp.json().get("data", [])
if not models:
raise RuntimeError("The model server lists no models. Pull or load one first.")
return models[0]["id"]The import from langfuse.openai is the drop-in wrapper. It subclasses the normal OpenAI client, so every chat.completions.create call you make is recorded as a generation with its prompt, completion, model name, and token usage. Nothing else in your code has to know Langfuse exists. Pointing base_url at a local server is the only change from a hosted setup.
Now the check script, which tests the three links in the chain: server version, API keys, and model server.
# check_stack.py
import os
import sys
import httpx
from langfuse import get_client
from tracelab.config import LLM_BASE_URL
from tracelab.llm import resolve_model
MIN_SERVER = (4, 59, 0)
def parse(version: str) -> tuple:
return tuple(int(p) for p in version.split("-")[0].split(".")[:3])
def main() -> int:
host = os.getenv("LANGFUSE_HOST", "http://localhost:3000")
failures = 0
health = httpx.get(f"{host}/api/public/health", timeout=10).json()
version = health.get("version", "0.0.0")
if parse(version) >= MIN_SERVER:
print(f"[ok] langfuse server {version}")
else:
print(f"[fail] langfuse server {version} is older than 4.59.0")
failures += 1
if get_client().auth_check():
print("[ok] langfuse API keys accepted")
else:
print("[fail] langfuse rejected the API keys")
failures += 1
try:
print(f"[ok] model server answers at {LLM_BASE_URL}, model: {resolve_model()}")
except Exception as exc:
print(f"[fail] model server: {exc}")
failures += 1
return failures
if __name__ == "__main__":
sys.exit(1 if main() else 0)python check_stack.py[ok] langfuse server 4.x.y
[ok] langfuse API keys accepted
[ok] model server answers at http://localhost:11434/v1, model: your-model-tagThree green lines mean the rest of the tutorial will behave. A red line tells you which of the three layers to fix, which beats reading a stack trace from the SDK.
Step 9: Record your first traced call
Write the smallest possible traced request. It exercises the wrapper, the background exporter, and the flush call that short scripts need.
# hello.py
from langfuse import get_client
from tracelab.llm import client, resolve_model
model = resolve_model()
reply = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Explain what a trace is in one sentence."}],
temperature=0,
max_tokens=128,
name="hello-local",
)
print(reply.choices[0].message.content)
get_client().flush()If you only need the runtime basics before this step, our Ollama walkthrough for local LLMs covers them. Run the script, then refresh the Traces table in the UI. Within a few seconds you should see one row named hello-local with the model name, input, output, and token counts. Click it. The generation view shows the exact messages, the sampling settings, and the latency. That single row is the whole value proposition: every later step only adds structure around it.
The flush() call matters. The SDK batches spans and exports them in the background, and Python SDK v5 compresses those batches with gzip before sending. A script that exits right after the last request can quit before the final batch leaves. Long-running servers do not need the call. Scripts, notebooks, and test runners do.
Step 10: Add Spans, Sessions, Streaming, and Metadata
A flat list of generations does not explain a multi-step request. This step adds a retrieval span, a user and session ID, and a streaming path that measures time to first token, which is the latency number users actually feel.
# tracelab/pipeline.py
import time
from langfuse import get_client, observe
from .config import PROMPT_VERSION
from .llm import client, resolve_model
MODEL = resolve_model()
SYSTEMS = {
"v1": "Answer using only the supplied context. If the context does not contain "
"the answer, say you do not know. Keep answers under three sentences.",
"v2": "You are a careful assistant. Use only the supplied context. Quote the "
"relevant phrase, then answer in one sentence. If the context is silent, "
"say you do not know.",
}
def _messages(question: str, context: list[str]) -> list[dict]:
joined = "\n".join(f"- {c}" for c in context)
return [
{"role": "system", "content": SYSTEMS[PROMPT_VERSION]},
{"role": "user", "content": f"Context:\n{joined}\n\nQuestion: {question}"},
]
@observe(name="retrieve")
def retrieve(question: str, docs: list[str], k: int = 2) -> list[str]:
words = set(question.lower().split())
ranked = sorted(docs, key=lambda d: len(words & set(d.lower().split())), reverse=True)
return ranked[:k]
@observe(name="answer-question")
def answer(question: str, docs: list[str], user_id: str = "demo-user",
session_id: str | None = None) -> dict:
lf = get_client()
lf.update_current_trace(
user_id=user_id,
session_id=session_id,
tags=["tracelab", "local-model", f"prompt-{PROMPT_VERSION}"],
metadata={"model": MODEL, "prompt_version": PROMPT_VERSION},
)
context = retrieve(question, docs)
started = time.perf_counter()
resp = client.chat.completions.create(
model=MODEL,
messages=_messages(question, context),
temperature=0.2,
max_tokens=300,
name="local-answer",
)
latency = time.perf_counter() - started
lf.score_current_trace(name="latency_s", value=round(latency, 3))
return {
"answer": resp.choices[0].message.content or "",
"trace_id": lf.get_current_trace_id(),
"latency_s": latency,
}
@observe(name="stream-answer")
def stream_answer(question: str, docs: list[str], session_id: str | None = None) -> str:
lf = get_client()
lf.update_current_trace(session_id=session_id, tags=["tracelab", "streaming"])
context = retrieve(question, docs)
started = time.perf_counter()
first_token = None
parts: list[str] = []
stream = client.chat.completions.create(
model=MODEL,
messages=_messages(question, context),
temperature=0.2,
max_tokens=300,
stream=True,
stream_options={"include_usage": True},
name="local-stream",
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
if first_token is None:
first_token = time.perf_counter() - started
parts.append(chunk.choices[0].delta.content)
if first_token is not None:
lf.score_current_trace(name="ttft_s", value=round(first_token, 3))
return "".join(parts)Three details are worth understanding. The @observe decorator turns each function into a span and nests them automatically, so retrieve appears as a child of answer-question with its own duration. The tags and metadata make runs filterable later, and because Langfuse v4 stores these attributes on every observation, a filter on the prompt-v2 tag matches the generation as well as the trace. Finally, stream_options asks the server to send token usage in the last chunk. If your runtime ignores it, the generation still records, only without a token count.
Note that update calls like update_current_trace and score_current_trace belong to the older, widely used surface of the Python SDK. Version 5 removed APIs that had been marked deprecated, so if either name raises an AttributeError on your install, check the v4-to-v5 migration page for the renamed call. The shape of the data stays the same.
Step 11: Score Quality With a Replayable Test Suite
Traces tell you what happened. Scores tell you whether it was good. A score is a named number attached to a trace, and Langfuse charts it over time, so a prompt change that quietly breaks answers shows up as a dip instead of a rumor.
# run_suite.py
import time
from langfuse import get_client
from tracelab.pipeline import MODEL, answer
KNOWLEDGE = [
"The TraceLab web UI listens on port 3000 on the host machine.",
"Traces are stored in ClickHouse, and PostgreSQL holds projects and users.",
"The local model server exposes an OpenAI-compatible API under /v1.",
"API keys can expire after 30 days, 60 days, 90 days, one year, or a custom date.",
"Python SDK v5 needs Python 3.11 or newer and a Langfuse server at v4.59.0 or newer.",
]
CASES = [
{"id": "port", "q": "Which port does the web UI use?", "expect": ["3000"]},
{"id": "storage", "q": "Where are traces stored?", "expect": ["clickhouse"]},
{"id": "python", "q": "Which Python version does SDK v5 need?", "expect": ["3.11"]},
{"id": "expiry", "q": "How long can an API key live before it expires?", "expect": ["30", "90"]},
{"id": "unknown", "q": "Who won the 2022 football World Cup?", "expect": ["know"]},
]
def main() -> None:
lf = get_client()
session = f"suite-{int(time.time())}"
passed_total = 0
print(f"model: {MODEL}")
print(f"{'case':<10}{'passed':<8}{'latency_s':<10}")
for case in CASES:
out = answer(case["q"], KNOWLEDGE, user_id="suite-runner", session_id=session)
text = out["answer"].lower()
ok = all(word.lower() in text for word in case["expect"])
passed_total += int(ok)
lf.create_score(
trace_id=out["trace_id"],
name="contains_expected",
value=1.0 if ok else 0.0,
comment=f"case={case['id']}",
)
print(f"{case['id']:<10}{'yes' if ok else 'NO':<8}{out['latency_s']:<10.2f}")
print(f"{passed_total}/{len(CASES)} cases passed")
lf.flush()
if __name__ == "__main__":
main()python run_suite.py
PROMPT_VERSION=v2 python run_suite.pyIllustrative output (your latencies depend entirely on your hardware and model):
model: your-model-tag
case passed latency_s
port yes 1.84
storage yes 1.52
python yes 1.71
expiry NO 2.03
unknown yes 1.39
4/5 cases passedThe second command runs the same suite with the second system prompt. Because every trace carries a prompt-v1 or prompt-v2 tag, you can now filter the Traces table by tag and compare the contains_expected and latency_s scores side by side. This is a crude keyword check, and deliberately so: it runs offline, costs nothing, and catches regressions in the retrieval step as well as the model. For fuzzier answers, add an LLM judge later, ideally a fast one, such as the diffusion-style LLMs we compared on speed. The check also demonstrates a point about local models in particular: small models drift more between prompt wordings than large hosted ones, so having a repeatable suite is worth more here than anywhere.
Step 12: Read the Dashboards and Compare Prompt Versions
With a few dozen traces in the system, the UI starts paying you back. Spend ten minutes on the views below before you write more code.
| View | Where to find it | What to look for |
|---|---|---|
| Trace tree | Open any trace | Which child span ate the time: retrieval or generation? |
| Generations table | Tracing, filtered to generations | Prompt length against latency, to spot oversized contexts |
| Sessions | Sessions, then pick one | Whether a long chat degrades after the model’s context fills up |
| Scores | Filter by score name | Rows where contains_expected is 0, your regression list |
| Tag filter | Filter by prompt-v1 or prompt-v2 | Which prompt wins on quality and on speed |
| Experiments | Experiments section | Charts comparing metrics across runs, with baseline statistics |
The Experiments interface got charts comparing metrics across runs, baseline summary statistics, and item-level comparisons filtered by score thresholds on October 19, 2026. If you keep a dataset of questions, run each prompt version as an experiment and read the comparison there instead of eyeballing tag filters. The SDK release also added links to experiments, so a script can print a clickable URL after a run.
Latency deserves a careful read on local hardware. The first request after a model loads includes load time, which can dwarf normal generation, so discard it when you compare runs. Also compare ttft_s and latency_s separately. A model with a slow first token and fast decoding feels sluggish in chat even if its total time looks fine.
Langfuse also ships an in-app assistant, called Halo after a rename on October 19, 2026 according to the v4.64.0 release notes. Halo is in public beta for self-hosted deployments at v4.28.0 or newer, and it needs an instance-wide Langfuse AI model plus a worker queue consumer configured. It is optional, and you can run this entire tutorial without it. If you enable it on a private server, remember that it needs a model connection, so decide which model you are comfortable sending trace content to.
Step 13: Back Up Data, Rotate Keys, and Plan Upgrades
A lab setup becomes a real service the first time someone asks “can we look at last month’s traces?” Four habits keep that conversation short.
First, back up the databases. ClickHouse holds trace data and PostgreSQL holds projects, users, and prompts, so you need both. Stop writes briefly or use each database’s native tooling, and copy the Docker volumes to storage outside the host. Test a restore once, because an untested backup is a hope.
docker compose ps --format '{{.Name}} {{.Status}}'
docker volume ls | grep -i langfuse
docker compose exec postgres pg_dump -U postgres postgres > langfuse-pg-$(date +%F).sqlThe service name and database user in that pg_dump line follow the repository’s compose file at the time of writing, so confirm them with docker compose config first.
Second, rotate API keys. Create project keys with an expiry through the UI and move your apps to them. Third, pin image tags. A floating tag lets a routine docker compose pull change your server generation overnight, and a mismatch with your SDK major produces confusing ingestion errors. Fourth, plan upgrades on your own schedule. Self-hosted users control their v4 migration timeline, but Langfuse’s compatibility notes schedule the removal of legacy APIs, features, and ingestion paths for November 16, 2026, the same day Cloud becomes v4-only. If you stayed on an old SDK against a v4 server, finish the move to SDK v5 before that date.
Ownership matters as much as tooling. Langfuse’s October 22 guidance distinguishes Cloud, where the vendor operates the infrastructure, from self-hosting, where you also own SDK rollout, credentials, network paths, exported data, and internal support. Self-hosting is free of license fees but not free of work, as any guide to self-hosting an LLM will tell you. A third-party comparison of five tools, costed at 2M traces by ecorpit, is a useful reality check if you are deciding between running it yourself and buying a plan.
The Complete Working Project
Everything you wrote in Steps 7 through 11 fits into one small repository. Here is the final layout and the one-shot script that boots, checks, and exercises the whole chain.
tracelab/
├── .env # Langfuse keys and model server settings
├── requirements.txt
├── check_stack.py # verifies server version, keys, model server
├── hello.py # first traced call
├── demo_stream.py # streaming trace with time to first token
├── run_suite.py # replayable test suite with scores
├── run_all.sh # boot, check, and test in one go
├── tracelab/
│ ├── __init__.py
│ ├── config.py
│ ├── llm.py
│ └── pipeline.py
└── langfuse/ # cloned repo with compose file and override# demo_stream.py
from langfuse import get_client
from run_suite import KNOWLEDGE
from tracelab.pipeline import stream_answer
print(stream_answer("Which port does the web UI use?", KNOWLEDGE, session_id="demo-stream"))
get_client().flush()#!/usr/bin/env bash
# run_all.sh
set -euo pipefail
cd "$(dirname "$0")"
( cd langfuse && docker compose up -d )
until curl -fs http://localhost:3000/api/public/health > /dev/null; do
echo "waiting for langfuse..."
sleep 5
done
source .venv/bin/activate
python check_stack.py
python run_suite.py
PROMPT_VERSION=v2 python run_suite.py
python demo_stream.py
echo "Open http://localhost:3000 and compare the prompt-v1 and prompt-v2 tags."chmod +x run_all.sh && ./run_all.shWhen it finishes, the Traces table should hold roughly a dozen rows: two suite runs of five cases each, plus the streaming demo and your earlier hello trace. Every suite trace has a latency_s and a contains_expected score, the streaming trace has a ttft_s score, and the Sessions view groups each suite run into one session. If any of that is missing, the troubleshooting table below maps the symptom to a fix.
Common Pitfalls
These are the mistakes that cost people the most time, in rough order of how often they appear.
- Mismatched SDK and server. Python SDK v5 and JS/TS SDK v6 need a server at v4.59.0 or newer. A v3 server or an older v4 build rejects them. Run
check_stack.pyafter every upgrade. - Forgetting to flush in short scripts. Spans export in the background. A script that exits immediately can lose the last batch, which looks like missing traces. Call
get_client().flush()before exit. - Benchmarking the cold start. The first request after a model loads includes load time. Warm the model with a throwaway call, or drop the first row when you compare latency between prompt versions.
- Low token limits on reasoning models. Some models spend their whole budget thinking and return an empty message. Raise
max_tokensbefore you decide the model is broken, and check the generation view for the raw output. - Changing the encryption key after data exists. Langfuse encrypts stored credentials with
ENCRYPTION_KEY. Regenerating it later makes existing encrypted values unreadable, so treat it like a database password and back it up. - Running
docker compose down -v. The-vflag deletes the named volumes, and with them every trace. Use plaindownunless you want a clean slate. - Exposing the UI by accident. Traces contain prompts and completions, which often contain customer text. Bind the web port to localhost or put it behind an authenticating proxy, and apply the same care you would to any self-hosted LLM setup.
- Logging secrets and personal data. Whatever you send to the model gets recorded. Mask sensitive fields before they reach the client, using the SDK’s masking hook, and keep the Pydantic secret masking that shipped in the v4 line in mind as a safety net rather than a policy.
Troubleshooting: 11 Problems and Fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Web container restarts in a loop | Database not ready, or a password differs between services | Wait five minutes, then compare the password values across the compose file and check docker compose logs langfuse-web |
| Health endpoint reports version 3.x | The compose file pins the old image generation | Stop with docker compose down, switch the web and worker tags to the v4 line, start again |
auth_check() returns false or you see 401 errors | Wrong keys or wrong host | Copy the keys from .env again. Initialization keys exist only if the first boot had the init variables set |
| Sign-in works but no project exists | Initialization variables never reached the container | Run docker compose config, search its output for LANGFUSE_INIT, fix the override, and recreate the containers |
| Traces never appear | Missing flush, wrong host, or the worker is down | Add flush(), confirm LANGFUSE_HOST, and read docker compose logs langfuse-worker |
| Traces appear minutes late | The worker or Redis queue is backed up | Check worker and Redis health, then restart the worker service |
pip install langfuse picks a 4.x release | Python 3.10 or older interpreter | Rebuild the virtual environment with Python 3.11 or newer |
| Startup error about the encryption key length | ENCRYPTION_KEY is not 64 hexadecimal characters | Generate it with openssl rand -hex 32 |
| Connection refused on port 11434 | The model server is not running, or the app runs inside a container | Start Ollama, or use host.docker.internal as the host from inside Docker |
| Empty completions in traces | Reasoning model hit max_tokens, or the template is wrong | Raise the limit and, for llama-server, keep the --jinja flag |
| Streaming traces lack token counts | The runtime ignores stream_options | Upgrade the runtime, or accept timing-only data for streamed calls |
Advanced Tips
Once the basics work, five upgrades pay off quickly.
Trace every runtime through OpenTelemetry. Langfuse moved its first-party SDKs toward OpenTelemetry protocols during 2025, which means non-Python services, such as a Go gateway in front of your model server, can send spans to the same project. Other self-hosted AI services, like a self-hosted text-to-speech API, can report into one place the same way.
Run a local judge. The keyword check in Step 11 is cheap but blunt. Add an LLM judge that scores relevance from 0 to 1, and run it on a schedule against yesterday’s traces. Langfuse’s September update added TypeSafe’s Jev as a hosted judge priced at $0.042 per million input tokens, so compare that price with the electricity and time cost of judging on your own GPU before choosing.
Sample in production, never in experiments. High-traffic services do not need every trace. The SDK supports sampling, and the documentation’s experiment guidance says to run with sample_rate=1.0 so no experiment trace is dropped. Sample production traffic and keep experiments at full rate.
Script everything. Langfuse released CLI 1.0 with typed flags, machine-readable exit codes, and automatic API-version detection for servers from v3 onward. Pair it with the headless initialization variables from Step 3 and you can rebuild the whole environment in CI.
Know when to buy instead. Langfuse Cloud lists 100k units per month included, with additional units at $8 per 100k and lower rates with volume. It also lists a 50% first-year discount for early-stage startups. If nobody on your team wants to own ClickHouse, that arithmetic may beat the self-hosting work. ClickHouse, which reportedly acquired Langfuse on January 16, 2026, remains the analytical store in the self-hosted design.
FAQ
What is LLM observability? It is the practice of recording every model call, its inputs, outputs, timing, token usage, and quality scores, so you can debug and compare behavior over time. For local models it fills the gap left by the usage dashboards that hosted APIs provide.
Is self-hosted Langfuse free? The open-source platform can be self-hosted without buying a Cloud plan. You pay in hardware and maintenance instead. Langfuse Cloud is the paid option, and it lists 100k units per month included before the $8 per 100k rate applies.
Does Langfuse work with llama.cpp and Ollama? Yes. Both expose an OpenAI-compatible API, and the Langfuse OpenAI wrapper only needs a different base_url. Our Ollama tutorial shows the runtime side.
Do I need Python 3.11? For SDK v5, yes. Python 3.10 is outside its requirements. If you cannot upgrade, stay on Python SDK v4 and make sure your server is a compatible release.
Can I stay on Langfuse v3? You can, but only with Python SDK v4 and JS/TS SDK v5. The new SDK majors will not work against v3. Cloud becomes v4-only on November 16, 2026, so plan the move.
Does tracing slow down inference? Spans are batched and exported in the background, so the cost on the request path is small compared with generating tokens on local hardware. Measure it on your own setup by running the suite with and without the Langfuse keys set.
How is this different from Grafana or Datadog? General observability tools track infrastructure metrics. Langfuse understands prompts, generations, token usage, and scores as first-class objects, which is what you need to compare prompt versions and models.
