Coder Legion - Recent posts tagged opentelemetry

How to evaluate agents with one trace across MCP, A2A and Realtime

Wed, 14 Oct 2026 16:22:41 +0000

A support agent passes every eval in staging. In production the same agent is reached three ways: from an IDE over MCP, from the web app over the OpenAI Responses API, and from a triage agent over A2A. The dashboard stays green for weeks. Then a customer escalation shows the triage agent handed over a ticket without the customer's region, and the support agent quoted the refund policy for the wrong country. No evaluator flagged it, because no evaluator ever looked at the hop.

The reason is structural. Eval is a property of the path a request took, and the protocol defines the path. Most eval stacks were written against one surface, usually chat completions, so they score whatever still looks like a chat completion and quietly skip the rest. A protocol-neutral eval stack scores the path instead, whichever protocol carried the request.

Every protocol adds seams to the path

A plain chat completion has one path: prompt in, response out. One rubric can grade it. Each protocol you add forks that path into more decision points, and each decision point needs its own check.

MCP adds a tool catalog. The path becomes prompt, tool descriptor list, tool selection, call arguments, tool result, final answer. Score each tool call against policy, and score the catalog as a whole, because a poisoned descriptor looks fine at call time and only shows up against the full catalog.
A2A turns the path into a tree. Every hop needs three answers: did the calling agent pass enough context, did the called agent stay in scope, and did the returned artifact match the task that was asked for.
The Responses API puts three outputs in one envelope. Structured JSON gets graded against a schema, streamed text gets graded as it arrives, and a tool call gets graded against a per-tool policy. One end-to-end pass mixes three things that share nothing except the wrapper.
Realtime has no final message. Audio streams in both directions, so checks have to run mid-stream, every N tokens or N audio chunks, with a stop or disclaimer action when a rule fires. The end-to-end budget sits below 500ms, which caps how heavy each check can be.
Inline vision needs a second judge. When images ride in the same message as tool blocks, the text portion and the image portion need different rubrics, and the image rubric needs a judge that can open the image.
Session state spans turns. On a framework with explicit session state, the evaluator has to read state across turns instead of grading each turn alone.

Trace fidelity caps eval fidelity

An evaluator can only score what the trace can represent. If your trace has no way to express an agent-to-agent hop or a streaming chunk, no rubric downstream will catch a failure inside one.

So the first fix is in instrumentation, before any rubric work:

Model both sides of a hop. Emit one span kind on the calling agent and another on the called agent, and carry the trace ID across the wire so the whole multi-agent path lands in one trace.
Give each modality its own span kind. Voice, image and computer-use spans should be distinguishable from text spans, so each one reaches an evaluator built for that modality.
Record guardrail and evaluator results as spans. When a guardrail decision and an evaluator score sit in the same trace as the model call, you can see which check ran on which step.

The MCP spec helps here. The 2026-07-28 revision documents trace context propagation through _meta (traceparent, tracestate, baggage), so MCP tool calls can join the same trace as the rest of the agent loop.

Normalize the spans, then write each rubric once

The same agent identity shows up differently on every protocol. If you write a groundedness check against raw MCP payloads and another against raw Responses payloads, you now maintain two checks that drift apart.

Normalize first. Map every span to one semantic convention, such as the OpenTelemetry GenAI conventions, before any evaluator reads it. Then write rubrics against the normalized trace. One groundedness rubric can then grade a RAG answer that arrived over the Responses API and a RAG answer that arrived over MCP, and one task-completion rubric can grade the support agent no matter which client reached it.

This is the same move observability teams made with OpenTelemetry. Teams that locked to one vendor's tracing format rebuilt their instrumentation every time they switched vendors. Agent protocols are repeating that pattern one layer up.

Gate releases per protocol

A single overall score hides which surface regressed. A drop in the MCP tool-audit rubric and a drop in the Realtime guardrail are different failures with different fixes.

Keep a regression dataset per protocol surface. Run the matching rubric set against it on every change.
Set a threshold per protocol. Fail the release when any one surface drops below its own threshold, even if the blended number looks fine.
Key each suite on protocol and version. MCP's 2026-07-28 revision removed the initialize handshake, so every request now carries its protocol version in _meta, and the new deprecation policy gives every deprecated feature a minimum twelve-month window. Tag each regression case with the revision it ran on, and keep the tests for deprecated features until their window closes.
Keep the rubric sets in the repo as code. Reviewers should be able to read exactly what gets scored on each surface.

A five-step readiness audit

Inventory every surface in production. MCP servers, A2A endpoints, streaming, voice, vision and computer-use, with the protocol version and client library for each. Most teams find one they did not plan for, usually an MCP server a developer installed last sprint.
Instrument and pull one real trace per surface. Confirm the hop spans appear on both sides with the propagated trace ID, and that voice and image calls emit their own span kinds. Fix gaps before writing rubrics.
Write the rubric set for each protocol. Per-tool audit on MCP, per-hop audit on A2A, split rubrics on the Responses API, mid-stream checks on Realtime, an image-capable judge for vision.
Run a per-protocol regression suite and gate on per-protocol thresholds.
Cluster production failures by protocol. Failures that look like noise one at a time often form a pattern once you group them by surface.

If you only have half a day, do step 1. The list is usually longer than the team expects, and it tells you which evaluators are scoring the wrong path today.

Pitfalls that keep showing up

Evaluators hard-coded to the chat completions shape. Every new protocol triggers a rewrite.
Scoring each agent in isolation. The end-to-end view looks green while a handoff in the middle dropped context.
Post-call checks only on streaming surfaces. By the time the evaluator sees the output, the risky tokens are already on the wire.
Accepting tool descriptors from any MCP server without a scan. An untrusted server can change what your agent does, and the eval layer never sees it.
Scoring voice and computer-use on text alone. Errors at the ASR layer and the vision layer stay invisible.

The span kinds, the rubric set for each of the six protocols and the full audit are in the longer protocol-neutral eval playbook.

New protocols and revisions will keep arriving. Score the path the request took, and each new surface plugs into a trace model and a rubric library you already maintain.

Stop Paying $0.05 Per Trace: Building an OTel-Native LLM Cost Attribution Gateway

Fri, 09 Oct 2026 09:49:24 +0000

Stop Paying $0.05 Per Trace: Building an OTel-Native LLM Cost Attribution Gateway

When deploying generative AI into production, the second line item on your cloud bill isn't inference—it's the telemetry tax. Commercial LLM observability tools charge steep monthly per-seat licenses alongside predatory micro-billing for every trace, span, and log ingested. Worse, they require proprietary SDKs that pollute your clean codebase with vendor-specific client wrappers.

In this guide, we will design and deploy a zero-lock-in, self-hosted LLM proxy using Python (FastAPI + AsyncIO) that natively adopts OpenTelemetry GenAI Semantic Conventions (v1.26+). It intercepts inbound OpenAI and Anthropic inference calls, attributes micro-cent usage to individual tenants, and exports standard OTLP spans directly to Grafana Tempo, ClickHouse, or Jaeger.

1. The Bottleneck: Proprietary Silos vs. Standard OTel

Most commercial LLM observability platforms function as data traps. They rely on monkey-patching libraries like openai or langchain with closed-source telemetry hooks. When your application scales to millions of requests:

Data Ingestion Tax: Charging $0.005–$0.05 per trace turns a $2,000 OpenAI bill into an additional $1,500 observability invoice.
Egress & Compliance Friction: Sending raw prompts and completions through a closed third-party SaaS breaks strict enterprise compliance requirements (SOC2, HIPAA, GDPR).
Broken Tracing Context: When an LLM trace lives in a walled-garden dashboard, you cannot correlate slow generation latency with upstream database queries or downstream background queue workers in your primary APM.

The solution is simple: Decouple the proxy from the vendor. By running an asynchronous proxy at the edge of your infrastructure that speaks native OpenTelemetry protocol (OTLP), your LLM telemetry becomes just another distributed trace in your existing observability stack.

2. Gateway Architecture

The proxy sits as a transparent, high-throughput intermediary between your internal applications (n8n instances, microservices, AI agents) and upstream model providers.

[ Client / Agent / n8n ] 
       │ (Bearer Token + Custom Tenant Headers)
       ▼
[ OTel Reverse Proxy Gateway (FastAPI / uvloop) ]
       ├── 1. Parse Context (Tenant, Workspace, Workflow ID)
       ├── 2. Stream Request to Upstream (OpenAI / Anthropic)
       ├── 3. Token Count & Cost Engine (Micro-cent resolution)
       └── 4. Build OpenTelemetry GenAI Span
       │
       ├───────────────┬─────────────────┐
       ▼               ▼                 ▼
[ OTLP / Tempo ]  [ ClickHouse ]  [ Upstream LLM ]
 (Distributed)     (Cost Analytics) (Returns Response)

Core Invariants

Zero Overhead Streaming: Responses are piped as Transfer-Encoding: chunked byte streams to minimize Time-to-First-Token (TTFT) degradation (< 1.5ms added latency).
Exact Cost Attribution: Token calculations run against real-time model pricing tables, factoring in cached inputs, standard inputs, and completions.
OTel GenAI Standard: Complies with semantic convention attributes such as gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.cost.

3. The Implementation: Reverse Proxy with Native OTLP Telemetry

Below is a working implementation using FastAPI, httpx, and the OpenTelemetry Python SDK.

Step 1: Pricing Engine and Cost Matrix

Save this as pricing.py. It tracks costs down to 6 decimal places per 1,000 tokens.

# pricing.py
from decimal import Decimal

MODEL_PRICING = {
    "gpt-4o-mini": {
        "input_per_1k": Decimal("0.00015"),
        "output_per_1k": Decimal("0.00060"),
    },
    "gpt-4o": {
        "input_per_1k": Decimal("0.00500"),
        "output_per_1k": Decimal("0.01500"),
    },
    "claude-3-5-sonnet-20241022": {
        "input_per_1k": Decimal("0.00300"),
        "output_per_1k": Decimal("0.01500"),
    },
}

def calculate_cost(model: str, input_tokens: int, output_tokens: int) -> float:
    pricing = MODEL_PRICING.get(model, {"input_per_1k": Decimal("0.0"), "output_per_1k": Decimal("0.0")})
    input_cost = (Decimal(input_tokens) / Decimal(1000)) * pricing["input_per_1k"]
    output_cost = (Decimal(output_tokens) / Decimal(1000)) * pricing["output_per_1k"]
    return float(input_cost + output_cost)

Step 2: Proxy Server with OTel Instrumentation

Create main.py. This service receives standard OpenAI-compatible completions, passes them upstream, counts stream tokens, and issues telemetry spans.

# main.py
import json
import time
import httpx
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from pricing import calculate_cost

# Configure OpenTelemetry Provider
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://tempo:4317", insecure=True))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm.observability.gateway", "1.0.0")

app = FastAPI()
UPSTREAM_BASE = "https://api.openai.com/v1"

@app.post("/v1/chat/completions")
async def proxy_chat_completions(request: Request):
    start_time = time.time()
    body = await request.json()
    headers = dict(request.headers)
    
    # Extract Context Attribution
    tenant_id = headers.get("x-tenant-id", "anonymous")
    workspace_id = headers.get("x-workspace-id", "default")
    model = body.get("model", "unknown")
    stream = body.get("stream", False)

    # Setup upstream client
    client = httpx.AsyncClient(base_url=UPSTREAM_BASE, timeout=60.0)
    auth_header = headers.get("authorization", "")
    
    # Forward to Upstream
    upstream_req = client.build_request(
        method="POST",
        url="/chat/completions",
        headers={"Authorization": auth_header, "Content-Type": "application/json"},
        content=json.dumps(body)
    )
    upstream_resp = await client.send(upstream_req, stream=stream)

    if not stream:
        raw_data = await upstream_resp.aread()
        await client.aclose()
        data = json.loads(raw_data)
        
        # Extract token usage
        usage = data.get("usage", {})
        prompt_tokens = usage.get("prompt_tokens", 0)
        completion_tokens = usage.get("completion_tokens", 0)
        total_cost = calculate_cost(model, prompt_tokens, completion_tokens)
        
        # Record OTel Trace Span
        with tracer.start_as_current_span("gen_ai.chat") as span:
            span.set_attribute("gen_ai.system", "openai")
            span.set_attribute("gen_ai.request.model", model)
            span.set_attribute("gen_ai.usage.input_tokens", prompt_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", completion_tokens)
            span.set_attribute("gen_ai.usage.cost", total_cost)
            span.set_attribute("tenant.id", tenant_id)
            span.set_attribute("workspace.id", workspace_id)
            span.set_attribute("http.status_code", upstream_resp.status_code)
            span.set_attribute("latency.total_seconds", time.time() - start_time)

        return StreamingResponse(iter([raw_data]), status_code=upstream_resp.status_code, media_type="application/json")

    # Streaming logic
    async def stream_generator():
        completion_tokens = 0
        prompt_tokens = body.get("prompt_tokens_hint", 0)  # Heuristic fallback if not provided
        try:
            async for chunk in upstream_resp.aiter_bytes():
                # Track approximate completion chunks
                if b'"content":' in chunk:
                    completion_tokens += 1
                yield chunk
        finally:
            await upstream_resp.aclose()
            await client.aclose()
            
            total_cost = calculate_cost(model, prompt_tokens, completion_tokens)
            
            with tracer.start_as_current_span("gen_ai.chat.stream") as span:
                span.set_attribute("gen_ai.system", "openai")
                span.set_attribute("gen_ai.request.model", model)
                span.set_attribute("gen_ai.usage.input_tokens", prompt_tokens)
                span.set_attribute("gen_ai.usage.output_tokens", completion_tokens)
                span.set_attribute("gen_ai.usage.cost", total_cost)
                span.set_attribute("tenant.id", tenant_id)
                span.set_attribute("workspace.id", workspace_id)
                span.set_attribute("stream.chunks", completion_tokens)
                span.set_attribute("latency.total_seconds", time.time() - start_time)

    return StreamingResponse(stream_generator(), status_code=upstream_resp.status_code, media_type="text/event-stream")

4. Deployment, Rate Limits, and CRM/Billing Routing

Running with Uvicorn and Gunicorn

To run this under heavy multi-agent traffic (e.g., hundreds of concurrent n8n pipelines), deploy via Docker with uvloop:

gunicorn main:app \
  --workers 4 \
  --worker-class uvicorn.workers.UvicornWorker \
  --bind 0.0.0.0:8080 \
  --backlog 2048 \
  --timeout 120

OTLP Export to ClickHouse or Tempo

Configure standard environment variables for your container to target your backend without code changes:

OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.internal.net:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_SERVICE_NAME=production-llm-gateway

Client-Side Configuration (n8n or Microservices)

In your n8n workflows or LangChain services, point your client configuration to the self-hosted proxy instead of OpenAI directly:

Base URL: http://proxy-gateway.internal:8080/v1
Headers:
- x-tenant-id: customer-acme-corp
- x-workspace-id: prod-lead-enrichment-v2

Your existing code requires zero architectural modifications. It simply gains comprehensive tracing, cost tracking, and micro-cent billing attribution.

5. Conclusion & Ready-to-Use Workflow Package

By deploying this self-hosted proxy, you instantly eliminate vendor telemetry lock-in and cut SaaS monitoring bills to zero. Every token processed by your workflows—whether triggered by n8n nodes, custom microservices, or autogen scripts—is tracked under unified OpenTelemetry standards.

You can manually build and test this gateway using the Python code provided above.

If you prefer a pre-built, production-hardened infrastructure package with Docker Compose topologies, Grafana dashboards, ClickHouse schema migrations, Redis rate limiting, and n8n verification workflows, you can grab the complete turnkey distribution:

Instant Access on Whop
Direct Download on Gumroad (Use promo code EARLYBIRD for 20% off)

Putting the trace before the loop; Observability-first for an AI agent, and what it bought me

Wed, 23 Sep 2026 08:38:45 +0000

An observability-first approach for building an AI agent, and what it bought me.

A couple of weeks ago I started implementing Kept, a self-hostable post-purchase support agent for e-commerce, with reliability as its core offering. Besides the product itself, my objective in building it is to delve into the depths of agentic system design, and see what it actually means to build an agent with "reliability at its core".

Observability first

I started from a theory my experience validated again and again throughout the years: observability is the bedrock of reliability. Proper logs and metrics beat an ideal architecture, industry-leading frameworks or best coding patterns.

That was the case well before the agentic era, and I've always paid particular attention to this layer. But with the advent of non-deterministic LLMs sitting at the core of products, I decided to take that to the next level: designing and implementing the full tracing layer before writing a single line of code for the main agent loop/product. Turned out to be an insightful experience.

Up next I'll walk through how I designed and implemented the tracing layer, then present 3 benefits and 2 drawbacks that I consider worth highlighting from the observability-first approach, and end with some thoughts on how to apply the approach if you don't have the luxury of starting with a fresh codebase.

Tracing layer design and implementation

I started with an AI-assisted research phase. Here's a list of resources I found particularly useful:

0. The guiding design decision

The research produced the first important insight. My original idea was to have Langfuse as the source of truth that saves all spans and traces and serves them wherever they're needed. But then I realized I'd have two consumers with very different needs. First we have the human, who needs to analyze, search and filter the traces in a well-structured UI. Then we have the machine (mostly evals) that needs fast access to a reliable record of what happened. Langfuse was the right choice for the human side, but having evals rely on it to get their traces meant additional API calls that added cost and latency, plus complex concurrency scenarios that needed to be accounted for.

So I defined the first design decision in ADR-0001, which states that Langfuse is a sink for humans, not a source for machines. The practical consequence was an in-process model of the trace as the source of truth (which evals could use to run their checks), and Langfuse as a projection of it. This meant no Langfuse SDK and no OTel SDK in the core package, with a boundary-check script (check-eval-boundary.mjs) that enforced the ADR.

The rest of the design/implementation process looked like this:

1. Deciding the shape of the span tree

I started sketching the shape of the span tree, using a self-test that drove every sketch round: can the types express "a model call inside turn 2 followed by a tool span whose result state was unknown"?

After multiple rounds of sketches, I went from "every model call is a span" to three span kinds with a parent link: turn, model call, tool execution. The turn span would be the parent, and model calls would sit as siblings to tool executions, with causality carried by a callId. This move also enabled a three-layer taxonomy:

Layer	Where it lives	Fields
What the customer saw	The turn span (types.ts#L84-L87)	`customerInput` at start, `outcome` at end. The outcome is a union: reply with message text, failed with a reason, conversation full.
What the model saw	The model-call span (types.ts#L27-L39)	`inputMessages` at start, `outputMessages` at end. The assembled prompt, the model's reply including any tool-call requests, and tool responses as presented back to it.
What actually happened	The tool-execution span (types.ts#L48-L54)	`toolName`, `callId`, `args` at start. `resultState` and `result` at end. The args really executed and the state really returned.

2. Choosing the stamps by consumer

The next task was deciding what gets stamped on every trace and span (types.ts#L129-L144). Guided by the OTel GenAI semantic conventions, I picked the stamps by asking who reads each one later. A few examples:

promptHash for evals, hashing the template rather than the rendered prompt so the hash means "prompt version";
backendKind to localize a failure to adapter or core;
traceId kept distinct from sessionId, so evals can replay a conversation without merging the traces into one garbled tree;
a faultToggles array in preparation for a later CI matrix: every trace declares which faults were injected when it ran, so a red in the matrix can be attributed to the toggle that caused it.

3. Lifecycle and end states

A span starts as in_progress while its corresponding process is running, and when the process finishes, it can end either in a completed or error state. If the span is still in_progress when a trace is ended (trace.ts#L82-L91), it's marked as undetermined (span.ts#L92-L105), so a crash mid-span surfaces as a span rather than a missing one. undetermined is deliberately not error, because the step never reported failure, and not absent, because absence is indistinguishable from never started.

Within the model-call span, there can be multiple tool_call_response messages, each with its own status ('ok' | 'failed' | 'unknown'), which is kept distinct from the span status. This way, the span status describes the step, while the tool result state describes the tool's answer, so a tool that cleanly reports failure is a completed span.

A few more notes on the span lifecycle:

for each span, first completion wins. A completed span can't be later changed to undetermined;
trace end is idempotent and cached;
starting a span after end (trace.ts#L34-L43) is an error, thrown in dev and dropped in prod (trace.test.ts#L135-L144);
using the wall clock to mark the span start and a monotonic mark for duration.

4. Interfaces

Up next was writing the types according to the conceptual design. Here's where I defined the shapes of the ModelCallPayload, ToolExecutionPayload and TurnPayload as well as the actual TracePayload. And I also defined the Message type and the discriminated union of MessageParts it can contain ('text', 'tool_call' or 'tool_call_response'); these types ended up hoisted outside the tracing module since they represent a core part of the domain. More on that in the next article.

You can view the types and their annotations in these files: types.ts, messages.ts.

5. Classes

This step was mostly about the design surviving the compiler. I implemented the spans as a generic, abstract SpanBase class with its end(), error() and abandonIfInProgress() methods corresponding to the end states I mentioned in the lifecycle design, plus some helpers. Then came the Trace class that was responsible for starting the spans, keeping them in a spans array and sweeping the unfinished ones in trace.end().

A noteworthy implementation detail was deriving Start... and End... payloads that corresponded to a span's constructor payload and its end() method, respectively. This was particularly useful to make the TypeScript compiler a friend and not a foe: the partition rule (types.ts#L12-L15), the model-call partition (types.ts#L41-L46), the generic base whose constraint is the invariant (span.ts#L36-L41), and the subclass where it is enforced (span.ts#L108-L112).

6. Trace exporter as four pieces

By this point, I had the first part of ADR-0001 done: the in-process model of the trace, ready to be used by evals and testing harnesses. What was left was the exporter that packaged the in-process/in-memory model and sent it over to Langfuse. I implemented it in four parts:

a map holding the OTel GenAI attributes (otel-attributes.ts) I picked for the traces and spans (for example, sessionId as 'gen_ai.conversation.id')
an OTLP mapper (otlp.ts#L245-L270) that maps the captured attributes from their internal implementation (modelCallSpan.model) to the corresponding OTel GenAI semantic-convention name ('gen_ai.request.model'), and wraps them into an OTLP envelope to be sent over the network
a generic interface for the exporter (exporter.ts#L3-L21) that contained:
- an export(input: CompletedTrace) function that takes a completed trace, uses the mapper to turn it into an OTLP envelope and fires the transport call, saving the promise for flush()
- a flush() function that awaits all pending exports
the Langfuse implementation of the generic interface (langfuse.ts#L50-L101). The export layer is the only place where we use Langfuse-specific lexicon, spread across the attribute map, the OTLP mapper and the transport itself. Since I already had the attributes following the OTel GenAI convention, I decided to send them to Langfuse via a simple POST request, skipping any extra dependencies

7. Verifying with a fabricated conversation

Since no agent existed at this point, I had to implement a script (trace-demo.ts) that created a fake WISMO ("where is my order") turn so I could see it in the Langfuse dashboard. I also left a span deliberately open (trace-demo.ts#L108-L117), so I could watch the trace sweep in action.

At this point I discovered a bug that actually made me happy: Langfuse was rendering "empty object" as a tool call argument, while args was populated in the in-process trace. While the fix (otlp.ts#L101-L125) was pretty simple (the attribute mapping was incorrect), this bug proved the utility of ADR-0001 defined at step 0. Had evals been reading Langfuse, an assertion like "the tool was called with order 1042" would have gone red, looking exactly like a policy bug, when the only defect was a key name in the mapper. Because evals were designed to read the in-process object, that class of failure cannot produce a false red.

With the clear split between the sources, I could implement tests that touched the core functionality of the agent, and handle transport/presentational errors separately (those still need to be tested, of course).

After a few more fixes and design updates, the tracing layer was done and I was ready to implement the main agent loop. More on that in the next article.

Tradeoffs

As I said in the beginning, designing the full observability layer before writing a single line of product code was a novel approach for me. The process ended up taking more time than I expected, but it was quite insightful. Here are a few benefits and drawbacks of the approach that I find valuable.

Benefit #1: you ask the production questions on day one

The first benefit of this approach was having to visualize the entire user-system-model-tools interaction from the perspective of the dashboard I'll stare at when debugging the ugliest bugs.

I had to ask myself questions that would normally come up only when the product misbehaved in production:

what logs would be helpful when trying to understand what the model did?
how would I search these logs?
what attributes would I filter by?
how would I know what state a conversation is in, and how it got there?

Besides the basic questions, edge cases came up, too. And thinking about them from the perspective of how I'll debug them was particularly useful:

a tool span never completes, but the trace reaches its end. What does the trace show?
- that question is where the undetermined state in step 3 came from: in production, the unfinished span stays in the tree with a state that says "nobody knows". In development, it throws so I can fix the lifecycle bug.
the eval runner replays the same conversation three times. What do I see?
- one garbled tree, if the trace id doubles as the conversation id, because any OTel backend merges every span that shares a trace id. That is the reason step 2 keeps traceId distinct from the session id: the recorder mints a fresh trace id per run, while the session id stays a correlation key owned by the outside world, so three replays render as three traces inside one session.
a policy check changes the args between what the model requested and what got executed. Which one does the trace record?
- both, in different places: the model's request stays in the model-call span's outputMessages, while the args that actually ran go on the tool-execution span. That's exactly the purpose of the three-layer split from the step 1 table. The layers are allowed to disagree by design.

Benefit #2: no existing constraints means best practices become applicable

As I described in the beginning, I started the design phase with research on best practices and novel design/implementation patterns. Pretty standard approach. But this time it felt different.

Previously, when I had to implement an observability layer, I always had to start from an existing product or codebase. That became the toughest set of constraints that ended up dictating the design of the entire layer.

Practical example from Kept: a vendor-neutral trace model, with the vendor confined to the exporter. Try declaring your own plain-data trace the source of truth when you have fifteen services already instrumented with the vendor SDK's decorators; half of them are in Python; and there's a dashboard where three teams filter by attribute names nobody wrote down. You can already feel the headaches building up. But then try it in a fresh codebase: it is one ADR (ADR-0001), one types file (types.ts), a 45-line attribute dictionary (otel-attributes.ts), 111 lines for the exporter (langfuse.ts), and a boundary script (check-eval-boundary.mjs) that fails the build if anyone imports the SDK into core. Easy as pnpm check.

Researching best practices is not very useful when the way your product/codebase is implemented makes it too expensive to adopt them. I realized I didn't have that problem. Having no existing constraints makes the best pattern cheap to adopt.

Benefit #3: you get a first sketch of your architecture for free

So I answered the questions and edge cases from the observability-first angle. I researched best practices and I implemented them in my greenfield tracing layer. I hooked up the self-hosted Langfuse instance and wrote some throwaway scripts to populate the dashboard. Fixed the bugs that came up, too.

All looked good, so I turned my attention to the agent loop. And here's where I got a very pleasant surprise. The loop design was already halfway done:

the span tree was the loop's control flow. One turn span per invocation, one model-call span per round, one tool span per execution, and the turn's end payload as the return value.
the end-turn payload makes outcome required, so every exit path had to have one. Enumerating the exits turned the single reply outcome into reply | failed(reason) | conversation_full, and the failure reasons became the loop's terminal states: max rounds, refusal, max tokens, unknown stop reason, empty reply. The state machine was the trace type, filled in.
the tool span's end payload became the tool contract. Closing a tool span takes resultState and result, so the ToolResult contract adopted those exact names, plus a separate response for the text the model sees.

We all know how hard it is to start with a blank canvas. For me, the first line takes about as much mental effort as the rest of the diagram combined. Where do I start? How do I structure it? How do I avoid a foundation that locks me into the wrong pattern?

The observability-first approach saved me from that trouble and gave me a clear starting point. And by the nature of the exercise, I started with edge cases/bugs first, as opposed to the happy-path approach I'd normally take, which increased my confidence in the foundation even more.

Drawback #1: you'll have to revise the tracing layer later

While asking debugging-oriented questions at the beginning and trying to picture edge cases is an insightful exercise, you won't cover every angle from the start. Naturally, some details will show up only as you're implementing the agent and they will require modifying the tracing layer.

You can argue the tracing layer is more stable when you place it over a codebase that's already live and working, and there's truth to that: you know what the system actually does, instead of trying to picture it upfront. Some of that stability can also come from constraints that make changes expensive. Still, the retrofit can cost less redesign time, and that is a real tradeoff of this approach.

Drawback #2: throwaway scripts

Since you don't have a product in place, you'll need to put some extra work in scripts that simulate real traffic to see what your dashboard will actually look like; and those scripts might not reflect reality as it will present itself in production. You'll end up with code you have to throw away, and if you're not careful, the picture painted by the scripts can be misleading in important ways.

What if I don't have the luxury of working with a fresh codebase?

Initially I wanted to list the "fresh codebase" precondition as a tradeoff, which it is. The approach obviously works best when there's no code and no constraints in place. But I think the insights can be applied to an existing codebase, too. Here are some ideas:

Go trace-first on every new AI feature. Before writing the feature, write its trace design: which steps become spans, which attributes you will need when it misbehaves, what a crash halfway through should look like. Then write the code to satisfy the trace. The feature boundary is a small greenfield, granted, it needs to be compatible with the system currently in place; and if you find ways to isolate it, all the better!
Go trace-first after an incident. Same idea from a different angle: instead of starting from a roadmap, prioritize by real pain. Once the immediate problem is mitigated, design the trace that would have made the diagnosis take five minutes, ship that trace, and use it to understand the failure and verify the permanent fix.
Design the ideal tracing layer as if the codebase didn't exist. Apply the unconstrained research and design step as an exercise. Sketch what one request should look like in the dashboard, ignoring what the code can currently emit, then diff that against what you actually see. You end up with a backlog pre-ordered by debugging value rather than by what is easy. As a bonus, you skip drawback #2 (inaccurate throwaway scripts), since you already have production data to work with.

The obvious caveat here is that Benefit #2 loses its relevance, unless you find a way to fully isolate a new feature from the existing implementation/system.

Conclusion

Observability-first is not a novel idea; observability-driven development has been argued for since the late 2010s. In practice, though, tracing is usually deferred as premature in the rush to ship an MVP and learn from customers. Those practices have earned their place. But with non-deterministic language models sitting at the core of a growing class of products, I'd argue observability deserves to move from afterthought to first step.

How would you iterate on your AI product if you knew what the model did at every step? How would you trust your evals if, instead of judging an output, they looked at the whole picture? How would you respond to incidents? How would you feel about telling your stakeholders "you can trust this product"?

Would it be worth sacrificing fast early progress in favor of some extra time to think about observability?

I pentested my own AI hub and shipped the method, not the map.

Tue, 25 Aug 2026 08:07:08 +0000

I ran a penetration test on my own infrastructure last week. No Burp Suite, no exploit fired at production, no CVE popped. The whole engagement came down to one habit: refusing to believe a control was working until I had watched it work.

The target is a small observability hub I built for my own AI-assisted coding. Six services in one compose file: a tunnel, an OpenTelemetry Collector taking metrics and logs from Claude Code, Prometheus, Grafana, Loki, and a status API. The public surface is three aggregate numbers. Everything else stays private. That boundary, three numbers out and nothing else, was the whole thing I was testing.

The word "pentest" carries a picture that does not match, so: no attack traffic at the live system. The platform bills by usage and there is a WAF in front, so a flood of probes would have cost money and poisoned its own results. What I did was a read-only audit of the code and config, plus a dynamic run against the whole stack brought up locally in Docker.

I expected the findings to cluster around the parts nobody had looked at. They did the opposite. Nearly every serious defect sat inside a control written days or hours earlier, usually by me, usually with a comment beside it naming what it protected against. Old code has been observed: it has run against real traffic and somebody has been surprised by it. A defence written yesterday has only been reasoned about, which feels like the same thing and is not.

"Independent" is a measurement, not a comment

The privacy boundary is an allow-list rather than a deny-list, and that part was right. Claude Code was measured sending five identity attributes, user.email among them carrying a real address, and no flag turns them off. A delete_key for each works until the client adds a sixth, and this telemetry is beta: its attribute set is not a contract.

- context: resource
  statements:
    - keep_keys(resource.attributes, ["service.name"])
    - set(resource.attributes["service.name"], "claude-code")

The second line is not redundant: keep_keys filters keys, not values, and service.name is the one attribute that becomes an index label in Loki. On 2026-08-20 a sender holding the ingest token wrote it as claude-code-…victim@example.com, and the address arrived as an index label.

What was wrong was a sentence in my design note calling the two barriers on that boundary independent. One is in the Collector, one in Loki, which re-filters whatever reaches it. keep_keys(log.attributes, …) governs record attributes, and Loki's otlp_config has three sections, all three of attributes. Scope attributes crossed both untouched: remove Loki's list to test the isolation and a planted scope.secret was suddenly queryable, while identity and content stayed out. The repair reads like a no-op and is the whole fix:

- context: scope
  statements:
    - keep_keys(scope.attributes, [])

What should have warned me is that the same hole existed twice. Two days later the metrics path turned out to be leaking scope attributes as otel_scope_* labels past any allow-list, while the comment beside that exporter declared the boundary closed.

A proof that executes is not a proof that exercises

A line in the log path was meant to zero the trace ID on every record before storage. Written correctly, in the right place, covered by a proof that ran it.

# looks right, fails on every record
- set(log.trace_id.string, "")

ParseTraceID wants 32 hex characters and the empty string is not one, so the statement failed on every record, the Collector logged failed to execute statement and carried on, and the field reached Loki untouched. Measured on 2026-08-21 against the first real traffic: two warnings per record, around 180 per session, and a barrier that was declared and absent. The proof could not see it, because the payload never carried a trace ID. Green, and blind.

Its twin is worse, because there the failure was conditional. OTTL documents that set does nothing at all if the value resolves to nil, so a line collapsing the log body to the event name did nothing on any record without an event.name, and a body containing a prompt and an address arrived in Loki verbatim. Neither proof could have caught it: the client always sends event.name, and the synthetic payload had to include it to satisfy a different assertion. The defence was a no-op in exactly the case it existed for. Both close the same way, by making the payload carry the thing.

The worst one was inside the proofs

Those shell proofs are the instrument this project uses to not have silent failures. During the audit I found a silent failure inside the instrument, six hours old and mine.

Two of them pinned the Collector image literally, 0.158.0, under a comment claiming it was the same digest as production. A dependency PR had moved compose and the Railway Dockerfile to 0.159.0, Dependabot does not read shell, and the proofs went on pulling the old image and passing. So the sentence I had written to verify that upgrade, "contract proof green on the new image", was false. The pin is no longer copied: it is read out of docker-compose.yml, and each proof prints the image it is running on.

Then I wrote a CI gate so it could not happen again, and an adversarial review found the gate born broken. It counted how many proofs derive their image by searching the whole file for the string docker-compose.yml, comments included, so the comment describing the derivation survived the derivation: delete the real line and the count stayed at three and the gate stayed green. Fifteen lines below a comment forbidding exactly that pattern, in a file where the same mistake had already been made three times.

The shape has siblings. A blocking image scan went green because uninstalling pip is not removing it: ensurepip/_bundled/ keeps a second copy as a wheel, and the scanner does not read inside an archive. Each of those looks like a pass. None is evidence of the property you wanted.

The dynamic run, and what it does not prove

I brought the metrics path up locally with fake secrets and pushed one metric carrying identity with a valid token: an email in user.email, an id in organization.id, and a hostile value inside service.name itself.

# auth first, before anything else gets a vote
no auth      -> 401
wrong token  -> 401
valid token  -> 200   # accepted, now let's see what survives it

Then read the exporter. Here is the single series it exposes, in full:

claude_code_token_usage{job="claude-code",model="claude-opus-5",session_id="sess-canary",type="input"} 4242

The email is gone, the organization id is gone, the hostile service.name was pinned instead of becoming a label. What stayed are the three keys I allowed.

And 4242 went all the way to the public numbers, which is the honest half. An allow-list of names does not constrain values: whoever holds the ingest token can write claude_code.token.usage with any number in it, and the public queries read that counter with max_over_time(…[25h]), so an injected spike stays stuck for twenty-five hours. Measured on a test stack: 1e12 tokens. It does not close here, because the token identifies the trusted producer and these numbers have no second source. What does close is the number arriving from somebody else: the three public queries now carry {job="otel-collector"}.

The payload is also synthetic, so it proves the allow-list discards what I hand it, not that the client only sends that. On this same project a synthetic payload has already confirmed a query and then lied to me.

What I did not publish, and why

There is a version of this post that lists every residual weakness in the running system by name, with the exact route and the exact window. I wrote that report. It stays in the drawer.

The obvious objection is that the repository is public, so what am I withholding. The answer is the aggregation. Every defect above is closed in the open, with the measurement that found it sitting in the commit that fixed it. A list of what is still open, in one place, with the routes and the timings next to each other, is a different object. It is not a disclosure, it is directions.

Worth checking on yours

If you send telemetry from an AI coding client, read one raw record before you read your config. Identity ships by default in this class of product, and every deny-list you write today is a list of the fields that existed this morning.

If you lean on two barriers, the useful question is not whether both are configured. It is which set of data only one of them is actually seeing. Break each on purpose and query the other back.

And if you have proofs, ask what your last three green runs actually ran against. Mine were pulling an image production had already left behind, and they told me so in the friendliest way available: by passing.

Why your LLM and agent tracing should be OpenTelemetry-native

Thu, 25 Jun 2026 09:53:17 +0000

A trace is one of the more useful things you can capture from an LLM app, and many teams store it in a format they cannot move later. If your AI tracing uses its own private format, the spans sit in a silo, a backend change means re-instrumenting, and the AI traces never join the HTTP and database traces under one request. Making tracing OpenTelemetry-native from the start avoids that.

It helps to start with what traces give you that logs do not. The failures that matter most in an LLM app often do not throw an error. An agent returns a confident but wrong answer, nothing errors, and the log says the request finished in 4.2 seconds with a 200.

A trace shows the same request as a tree: an LLM call that planned three tool calls, a tool call whose return was an empty array, and a second LLM call that took that empty array and answered anyway. The trace shows the problem because it records the input and output of every step, not just that the step ran.

A log line for a tool call is a timestamp and a status. A tool span is the tool name, the arguments passed, the return payload, and the latency, all queryable, so you can filter to every empty tool return across a week of traffic.

That only works if the tracing layer models what an agent actually does. A generic OpenTelemetry install knows about HTTP requests and database queries, but it has no concept of an LLM call, a retrieval step, a tool invocation, or an agent that loops. A tracing layer for AI apps needs five span kinds:

LLM spans: model, provider, input and output, token counts, temperature.
Retriever spans: the query, the retrieved chunks, the embedding model, top-k. A wrong RAG answer usually comes from retrieval, not the model.
Tool spans: the tool name, the JSON arguments, the return payload. Most silent agent failures show up here.
Agent spans: the parent wrapping one agent's turn, so a multi-agent system shows which agent did what.
Chain spans: one per multi-step sequence, giving the per-step view a single root span buries.

When an answer is wrong, you read the tree from the bottom: was the final LLM span ungrounded, did a tool return something empty, did the retriever pull the wrong chunks. The span kinds turn "the agent is broken" into a specific failing node.

The thing worth getting right early is the trace format itself. A tracing tool tied to a single AI-specific dialect works until the rest of your stack runs on OpenTelemetry and the AI spans cannot join it. OpenTelemetry-native tracing keeps the spans portable, so a backend change moves your data with you instead of leaving it behind. It puts your LLM and tool spans under the same trace ID as the HTTP and database spans your services already emit, so a slow request is one tree instead of two dashboards. And it exports over OTLP, the wire format your collector already handles, with no separate ingest path.

The library I use for this is traceAI: Apache 2.0, built on the OpenTelemetry SDK for Python and TypeScript, with the five span kinds above and 30+ provider and framework integrations, so for most stacks the instrumentation is an install and one register call.

When you compare tracing layers, four questions cover it:

Does it cover the span kinds your app produces?

A chat wrapper needs LLM spans. A RAG app needs retriever spans too. An agent needs tool and agent spans. A multi-step pipeline needs chain spans. A tracing layer that only models LLM calls leaves you blind on the steps that actually fail. traceAI covers all five. Check this first, because no amount of backend polish fixes a missing span kind.

Does it integrate with your frameworks?

Count the providers and frameworks in your stack, then check how many the library ships an integration for. Every gap is instrumentation you write and maintain by hand. TraceAI's 30+ integrations cover the common Python and TypeScript stacks; if yours is exotic, confirm the hand-emit path works for you.

How deep is the backend?

A trace tree you can look at is the floor. The useful question is what runs on top: can the backend score spans for correctness, cluster failures into named issues, attach guardrails inline, and feed an optimization loop. Tracing tells you what happened. The backend is where you find out whether it was right and what to do about it.

What is the license, and is the instrumentation independent of the platform?

An Apache 2.0 instrumentation layer built on open OpenTelemetry means you can self-host, modify, and avoid a per-seat fee on the tracing itself. traceAI is Apache 2.0, and the instrumentation runs independently of the FAGI platform; point the exporter at a self-hosted OTel backend and no span leaves your infrastructure.

Once the spans land, the trace becomes the input to the rest of the loop. An evaluation layer scores each span for things like groundedness, context relevance, and task completion, and the scores attach back onto the spans so a failing one is filterable. The low-scoring spans get clustered into named failure modes, each with a likely root cause and a fix to try.

From there an optimizer acts on those failures: agent-opt ships six prompt optimizers that run against your eval results, and every run is human-initiated and gated by an evaluator rather than an automatic rewrite. The order is simple: traces produce the spans, evaluation scores them, clustering groups the failures, and optimization addresses them.

The trace format is the part that is hard to change later, so it is the one to settle first. The instrumentation can stay open and OpenTelemetry-native regardless of which backend you end up on, and the backend is a separate choice based on what it does with the trace once it lands.

OpenTelemetry-native, all-in-one, simple

Mon, 15 Jun 2026 23:34:33 +0000

The standard observability stack:
Grafana + Loki + Tempo + Prometheus. Four services to deploy, four configs to learn, dashboards to set up before you see a single trace.

What I actually needed:
One place to see traces, logs, and metrics from my services.

What I built:
One container. docker compose up. Done.

I work with .NET microservices and at some point I went through the full Grafana setup. And look, it works fine, it's a solid stack. But for what I needed it was way too much. I spent more time configuring things than actually looking at my data. The .NET Aspire dashboard is on the other end: you get a nice UI, but nothing is persisted. Close the browser and everything's gone. No customization either.

So I built something in between.

What it is

OtlpDashboard is a single container that receives OpenTelemetry data and stores it. It speaks standard OTLP on port 4317 (gRPC) and 4318 (HTTP/Protobuf). If your app already exports telemetry, you just point it here. No collector needed (works behind one too if you already have it).

Storage is SQLite by default, so zero config, just a file. You can switch to PostgreSQL or SQL Server if you want.

The frontend is a Vue 3 / Nuxt 4 SPA served on the same port. You get a grid of widgets you can customize: stat cards, line charts, sparklines, gauges, heatmaps, log streams, trace viewers, and a few more. Auth is on by default.

How AI helped me ship this in under a month

I want to be upfront about this. I used Claude Code a lot, especially for the frontend. I'm a backend dev. I know .NET, EF Core, gRPC. But Vue and Nuxt? Not really my thing. Without Claude Code I would have spent days stuck on CSS and component wiring. Instead I could focus on what I actually care about: the OTLP ingestion, the storage layer, how widgets work.

All the architecture and decisions are mine. The AI did the typing.

And honestly, I think that's the best use of these tools right now if you're a solo dev. They don't replace you. They cover the parts where you're slow, so you can ship something that would normally need a small team.

Some stuff I like about how it turned out

You can install widget packs from Git repos. There's a pack.json that bundles widgets and dashboards together, so if you have a team you can share a curated setup.

There's also a spec engine for building custom widgets with Mustache templates. It's sandboxed with DOMPurify, no JS eval. I wanted people to be able to build their own cards without opening security holes.

Retention is configurable per signal. Set 7 days for traces, 30 for metrics, whatever you want. Old data gets cleaned up on its own.

And there's an MCP server at /mcp if you use Claude Code or similar tools. You can query your telemetry from the LLM directly. I didn't expect to use this much but it's actually pretty handy when debugging.

Try it

Quickest way:

docker compose up --build

This starts everything including a sample workload that generates real data, so you're not staring at an empty dashboard.

Or just check the live demo. It's a static build with mock data, login accepts any password: Live demo

The repo is here: GitHub

It's GPLv3. If you try it out let me know what you think, I'm curious what's missing or broken.

Heimdall MCP

Mon, 11 May 2026 17:41:11 +0000

If you've been building with MCP servers lately, you've probably hit this wall: something goes wrong (or just feels slow) and you have zero visibility into what actually happened.

Which tool did Claude call? What input did it send? Did the server error silently? How long did it take?

That's the gap Heimdall fills.

What is Heimdall?

Heimdall is a transparent proxy for MCP servers. It sits between your MCP client (Claude Desktop, OpenCode, Cursor, or any other) and your MCP server — local or remote — and records every interaction as an OpenTelemetry span.

No modifications to your server. No SDK to integrate. No infra to run. Just wrap your existing server and start seeing what's happening inside.

The name comes from Norse mythology: Heimdall is the guardian of the Bifrost bridge, with the ability to see and hear everything that crosses between worlds. That's exactly the role this proxy plays.

The problem in one screenshot

You're running a Claude agent that orchestrates 5+ MCP tools. One of them is consistently slow. Another occasionally returns empty results. You have no way to know which one, when, or why — unless you dig into logs manually.

With Heimdall, every tool call becomes a structured span:

{
  "name": "mcp.tool.call",
  "attributes": {
    "gen_ai.tool.name": "search_documents",
    "mcp.duration_ms": 843,
    "mcp.status": "ok"
  },
  "events": [
    { "name": "request",  "body": { "query": "quarterly report" } },
    { "name": "response", "body": { "results": [...] } }
  ]
}

Stored. Queryable. Yours.

Zero config wrapping via mcp.json

The easiest way to use Heimdall requires zero access to the server's source code. Just install it globally and update your mcp.json:

npm install -g @cardor/heimdall-mcp

Before — your current config:

{
  "mcpServers": {
    "my-server": {
      "command": "node",
      "args": ["my-server.js"]
    }
  }
}

After — wrapped with Heimdall:

{
  "mcpServers": {
    "my-server": {
      "command": "heimdall",
      "args": [
        "--store", "sqlite://~/.heimdall/traces.db",
        "--",
        "node", "my-server.js"
      ]
    }
  }
}

That's it. Restart your client and every tool call starts being recorded. The -- separator tells Heimdall where its args end and the real server command begins.

For remote servers (HTTP or SSE), it's just as simple:

{
  "mcpServers": {
    "remote-server": {
      "command": "heimdall",
      "args": [
        "--store", "postgres://user:pass@localhost/mydb",
        "--target", "http://my-remote-mcp.com/sse"
      ]
    }
  }
}

Heimdall exposes stdio to your client, talks HTTP/SSE to the real server, and intercepts everything in between.

What gets recorded

Heimdall captures all standard MCP events:

Event	What's recorded
`tools/call`	Tool name, input args, response, duration, status
`tools/list`	Available tools and their schemas
`resources/read`	URI, MIME type, response size
`prompts/get`	Prompt name, arguments, rendered output
`initialize`	Client/server versions, negotiated capabilities
`shutdown`	Total session duration

Every span follows the OpenTelemetry gen_ai.* semantic conventions, so if you later want to send traces to Jaeger, Honeycomb, or Grafana Tempo, the data is already well-formed.

Storage options

Pick the backend that fits your setup:

SQLite — local file, zero infra

--store sqlite://./traces.db
# or with absolute path
--store sqlite:///Users/you/.heimdall/traces.db

Best for: local development, single-machine setups, quick debugging sessions.

Uses @libsql/client under the hood — pure WASM, no native bindings, no node-gyp.

Postgres — your existing database

--store postgres://user:password@localhost:5432/mydb

Best for: production environments, teams sharing observability data, long-term storage with SQL queries across multiple servers.

MySQL

--store mysql://user:password@localhost:3306/mydb

Same use case as Postgres. Pick whichever you already run.

All three use drizzle-orm with a shared schema, so the query experience is consistent regardless of which backend you choose.

Using it as a TypeScript library

If you have access to your server's code and want tighter integration, Heimdall ships as a fully-typed TypeScript library with a fluent builder API:

import { McpProxy } from 'heimdall-mcp'

const proxy = await McpProxy
  .create()
  .inbound({ transport: 'stdio' })
  .outbound({ transport: 'http', url: 'http://localhost:3001' })
  .store('sqlite://./traces.db')
  .build()

await proxy.start()

You can also plug in custom interceptors — for example, to redact sensitive fields before they're persisted, or to add your own business logic on top of the tracing:

import { McpProxy, type Interceptor } from 'heimdall-mcp'

// Custom interceptor: redact API keys from tool inputs before storing
const redactSecrets: Interceptor = {
  name: 'redact-secrets',
  async intercept(request, ctx, next) {
    const sanitized = redactSensitiveFields(request)
    return next(sanitized)
  }
}

const proxy = await McpProxy
  .create()
  .inbound({ transport: 'stdio' })
  .outbound({ transport: 'http', url: 'http://localhost:3001' })
  .store('postgres://user:pass@host/db')
  .intercept(redactSecrets)
  .build()

Design goals

A few decisions worth calling out:

No native dependencies. SQLite runs via WASM (@libsql/client), Postgres via the pure-JS postgres package, MySQL via mysql2. You can install and run Heimdall in any Node 22+ environment without compilation steps.

Transport mixing. Your client connects via stdio. Your server can be stdio, HTTP, or SSE. Heimdall bridges them transparently — useful for wrapping local CLI servers behind an HTTP interface without touching their code.

Get started

npm install -g @cardor/heimdall-mcp

GitHub: github.com/enmanuelmag/heimdall-mcp
Website: https://stack.cardor.dev/heimdall

If you try it out, I'd love to hear what you think — especially feedback on the interceptor API design and the OTel schema. Open an issue or find me on X \[@enmanuelmag\].

OpenTelemetry just standardized LLM tracing. Here's what it actually looks like in code.

Tue, 31 Mar 2026 10:24:21 +0000

Every LLM tool invents its own tracing format. Langfuse has one. Helicone has one. Arize has one. If you built your own — congratulations, you have one too.

OpenTelemetry just published a standard for all of them.

It defines how to name spans, what attributes a tool call should have, how to log prompts without leaking PII, and which span kind to use for an agent. It's called GenAI Semantic Conventions. It's experimental. And almost nobody has written about what it actually looks like when you implement it.

I know because I searched. "OTel GenAI semantic conventions" gives you spec pages. Zero practical articles. "How to trace LLM agent with OpenTelemetry" gives you StackOverflow questions with no answers.

We implemented it. Four PRs, a gap analysis, real before/after code. We also discovered, mid-implementation, that our traces never exported at all — but that's a different story.

Here's what the spec actually says, where we got it wrong, and what you should do today.

The wild west of LLM tracing

Right now, if you trace LLM calls, you're probably doing something like this:

span.setAttribute("llm.provider", "openai");
span.setAttribute("llm.model", "gpt-4o");
span.setAttribute("llm.tokens.input", 150);
span.setAttribute("llm.cost", 0.003);

That's what we did in toad-eye v1. Made sense to us. Worked fine in our dashboards.

Problem: nobody else's dashboards understand these attributes. Switch from Jaeger to Arize Phoenix — reconfigure everything. Export traces to Datadog — they see raw spans with no LLM context. Your tracing is a walled garden. You built vendor lock-in into your own code.

This is exactly what OpenTelemetry was created to solve. And now it has a spec for GenAI.

Three types of GenAI spans

The spec defines three operations. Every LLM-related span gets one:

chat gpt-4o                    ← model call
invoke_agent orchestrator      ← agent invocation  
execute_tool web_search        ← tool execution

The span name format is {operation} {name}. Not your custom format. Not gen_ai.openai.gpt-4o (that's what we had — no backend recognizes it).

Here's what we changed:

Span naming migration: the old format was invisible to every GenAI-aware backend.

Agent attributes — we had the paths wrong

If you're building agents (ReAct, tool-use, multi-step), the spec defines identity and tool attributes:

// What OTel says:
span.setAttribute("gen_ai.agent.name", "weather-bot");
span.setAttribute("gen_ai.agent.id", "agent-001");
span.setAttribute("gen_ai.tool.name", "search");
span.setAttribute("gen_ai.tool.type", "function");

// What we had:
span.setAttribute("gen_ai.agent.tool.name", "search");  // wrong path
// gen_ai.agent.name — didn't exist at all

The gen_ai.agent.tool.name path looks reasonable. It even reads well. But the spec puts tool attributes at gen_ai.tool.* — flat, not nested under agent. Our format, again, invisible to any backend that follows the standard.

Content recording — the spec agrees with us (feels good)

This was the one thing we got right from day one, and it's worth calling out because most teams get it wrong.

The spec says: don't record prompts and completions by default. Instrumentations SHOULD NOT capture content unless explicitly enabled.

Three official patterns:

Default: don't record. No prompt, no completion in spans. Privacy first.
Opt-in via span attributes. gen_ai.input.messages and gen_ai.output.messages as JSON strings.
External storage. Store content elsewhere, put a reference on the span.

We had recordContent: false as default since v1. When the spec confirmed this approach, it was one of those rare moments where your gut feeling gets validated by a committee of very smart people.

If you're logging prompts in spans by default — you might want to reconsider before your security team does it for you.

The honest gap analysis

Here's the full picture. No spin, no cherry-picking.

What we got right from day 1

Our attribute	OTel spec	Verdict
`gen_ai.provider.name`	`gen_ai.provider.name`	✅ Exact match
`gen_ai.request.model`	`gen_ai.request.model`	✅ Exact match
`gen_ai.usage.input_tokens`	`gen_ai.usage.input_tokens`	✅ Exact match
`error.type`	`error.type`	✅ Exact match

What we got wrong

What	Our version	OTel spec	Status
Span name	`gen_ai.openai.gpt-4o`	`chat gpt-4o`	Fixed
Tool name attribute	`gen_ai.agent.tool.name`	`gen_ai.tool.name`	Fixed
Custom attributes	`gen_ai.agent.step.*`	Reserved namespace	Moved to `gen_ai.toad_eye.*`
Agent identity	Didn't exist	`gen_ai.agent.name`	Added

What we built beyond the spec

Feature	Namespace	Why it's not in OTel
Cost per request	`gen_ai.toad_eye.cost`	Pricing is vendor-specific
Budget guards	`gen_ai.toad_eye.budget.*`	Runtime enforcement ≠ observability
Shadow guardrails	`gen_ai.toad_eye.guard.*`	Validation is app-level
Semantic drift	`gen_ai.toad_eye.semantic_drift`	Quality metric, not trace standard
ReAct step tracking	`gen_ai.toad_eye.agent.step.*`	ReAct is one pattern; spec is pattern-agnostic

The key insight: OTel spec covers WHAT happened. We cover WHY and HOW MUCH. Not competing — complementary. Your custom metrics go under your namespace. The spec's attributes go where backends expect them.

The migration: dual-emit, don't break users

We didn't do a clean break. v2.4 emits both old and new attribute names:

// New (OTel spec-compliant)
span.setAttribute("gen_ai.tool.name", toolName);

// Old (deprecated, still emitted for backward compat)
span.setAttribute("gen_ai.agent.tool.name", toolName);

Dual-emit approach: old attributes get @deprecated, new ones follow the spec. Both emitted until v3.

An environment variable controls when to stop emitting deprecated attributes:

OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental

This was four PRs (#170, #171, #172, #173). v3 will remove the deprecated aliases entirely.

The irony

While implementing all of this, we did a round of manual testing.

Turns out our traces never exported. At all. Ever. The OTel NodeSDK silently disables trace export when you pass spanProcessors: []. We had 252 passing tests. All of them mocked the SDK.

So we standardized our attributes perfectly — for traces that nobody could see.

We fixed both. Published six patch versions in one day. The full story is in article #2.

Which backends actually support this

This is the reason to care. Emit the right attributes today → six backends visualize your traces tomorrow:

Backend	Recognizes GenAI spans	Agent visualization	Cost
Jaeger	Basic (nested spans)	Hierarchy view	Free
Arize Phoenix	Full GenAI UI	Agent workflow	Free tier
SigNoz	GenAI dashboards	Nested spans	Free / Cloud
Datadog	LLM Observability	Agent tracing	Paid
Langfuse	Full GenAI UI	Session view	Free tier
Grafana + Tempo	Query by attributes	Custom dashboards	Free

No vendor lock-in. One set of attributes. Six places to visualize them.

What you should do today

If you're tracing LLM calls — even with custom code — aligning with the spec now saves you pain later. The conventions are experimental, but the direction is locked in.

Quick checklist:

Set gen_ai.operation.name on every LLM span: chat, invoke_agent, or execute_tool
Format span names as {operation} {model_or_agent_name}
Use official attributes: gen_ai.agent.name, gen_ai.tool.name, gen_ai.tool.type
Put YOUR custom attributes under YOUR namespace — not gen_ai.*
Don't record prompt/completion by default — make it opt-in
Test your traces in at least 2 backends (Jaeger + one GenAI-specific like Phoenix)

Full spec: OpenTelemetry GenAI Semantic Conventions
Agent spans: GenAI Agent Spans

Previous articles:

toad-eye — open-source LLM observability, OTel-native: GitHub · npm

️

Distributed Tracing in Spring Boot: A Practical Guide to OpenTelemetry and Jaeger

Sat, 31 Jan 2026 18:28:46 +0000

TL;DR

Distributed tracing helps you understand how requests flow through microservices by tracking every hop with minimal overhead. This guide covers OpenTelemetry integration in Spring Boot 4 using the native starter, explains core concepts like spans and context propagation, and demonstrates Jaeger-based tracing with best practices for production. Whether you're debugging latency issues or optimizing service dependencies, distributed tracing provides the visibility modern architectures demand.

GitHub Repository: learning-distributed-tracing

The Problem: Debugging in the Dark

In a monolithic application, debugging a slow request is straightforward. Add some logging, attach a profiler, and you can see exactly where time is spent. But microservices change everything. A single user request might touch ten or more services, each with its own logs. Failures often happen between services, not inside them. When something breaks or slows down, where do you even start?

Traditional logging falls short here. Sure, you can correlate logs by request ID, but manually piecing together the journey across services, databases, and queues is tedious and error-prone. You need something that automatically tracks the entire execution path, measures timing at each step, and shows you the complete picture. That's distributed tracing.

Understanding Observability: Metrics, Logs, and Traces

Modern observability rests on three pillars. Metrics are numerical measurements like CPU usage or request count—great for alerting but lacking context for debugging. Logs are discrete events that tell you what happened at a specific moment but struggle with correlation across distributed systems. Traces capture the complete journey of a request through your system, showing execution flow and timing.

These pillars complement each other. Metrics tell you there's a problem, logs provide event details, and traces show you the execution path. Together, they form a complete observability strategy.

It's worth distinguishing observability from monitoring. Monitoring answers "Is the system healthy?" through dashboards and alerts. Observability answers "Why is the system behaving this way?" by designing systems to answer questions you didn't anticipate. Distributed tracing is a core enabler of observability, not a replacement for monitoring.

The Fundamentals of Distributed Tracing

Telemetry refers to automated data collection from remote sources—your application constantly reporting its health and activity. Spans are the building blocks of traces, representing units of work with start time, duration, and metadata. When Service A calls Service B, both create spans that form a parent-child relationship showing the call hierarchy.

Traces are collections of spans representing a single transaction. A trace ID ties all related spans together across service boundaries. Context Propagation maintains trace continuity—when Service A calls Service B, it passes the trace context in HTTP headers, allowing Service B to create child spans under the same trace.

OpenTelemetry: The Industry Standard

Before OpenTelemetry, every observability vendor had proprietary SDKs and formats. If you wanted to switch from Jaeger to Zipkin, you'd re-instrument your entire codebase. This vendor lock-in meant architectural decisions became permanent commitments.

OpenTelemetry is a vendor-neutral framework providing APIs, SDKs, and tools for telemetry data. Formed by merging OpenTracing and OpenCensus, it provides a single instrumentation API that works with any backend. The value proposition is simple: instrument once, send data anywhere.

The architecture includes the API and SDK for creating telemetry, Auto-instrumentation for frameworks like Spring and JDBC, and the Collector—an optional but recommended component that receives, processes, and exports telemetry.

While this article focuses on distributed tracing, it's worth noting that OpenTelemetry standardizes all three pillars of observability—metrics, logs, and traces. The same SDK and protocol handle all three, giving you a unified approach to instrumentation across your entire observability stack.

OTLP (OpenTelemetry Protocol) is the wire format for transmitting telemetry data. Supporting both gRPC and HTTP transports, OTLP defines how traces, metrics, and logs are serialized and sent to collectors or backends. The protocol handles backpressure, retries, and batching for reliable delivery. Most modern observability tools now support OTLP natively, making it the de facto standard.

Spring Boot 4 and OpenTelemetry Integration

Spring Boot 4 brings first-class support for OpenTelemetry through the spring-boot-starter-opentelemetry dependency. This starter provides automatic configuration and instrumentation for common scenarios like HTTP requests, database calls, and messaging.

Previous versions of Spring Boot required manual setup using the OpenTelemetry Java agent or custom configuration. Spring Boot 2 and 3 users could leverage the Java agent for bytecode instrumentation, which worked but added operational complexity. The agent approach meant deploying a JAR alongside your application and configuring it via environment variables or system properties.

With Spring Boot 4, the starter eliminates much of this complexity. Add the dependency, configure a few properties, and you're done. Under the hood, it uses Spring's auto-configuration to set up the OpenTelemetry SDK, register instrumentation libraries, and configure exporters based on your application properties.

The starter automatically instruments:

HTTP requests and responses via Spring MVC and WebFlux
RestTemplate, RestClient, and WebClient calls
JDBC database operations
Logs (automatically includes trace and span IDs)

For additional instrumentation like Kafka messaging, you can use the @WithSpan annotation for manual instrumentation, or use the OpenTelemetry Java Agent which provides automatic instrumentation for 150+ libraries.

Spring Boot Actuator's Role: While Actuator isn't required for tracing, it plays a complementary role in Spring Boot 4's observability story. Actuator's ObservationRegistry is what actually observes requests and framework operations. The OpenTelemetry starter bridges these observations into OTel-compliant traces. Think of Actuator as operational introspection (health, metrics) and OpenTelemetry as behavioral introspection (request flows).

You can still use the Java agent if you need instrumentation for libraries outside Spring's ecosystem, but for typical Spring Boot applications, the starter is sufficient and more maintainable. Framework-level instrumentation gives you baseline visibility automatically, while custom spans should be added only where domain insight is needed. This balance is critical—over-instrumentation creates noise, while under-instrumentation hides intent.

Jaeger: Your Trace Backend

Jaeger is an open-source distributed tracing platform originally developed by Uber, providing storage, querying, and visualization for traces. While OpenTelemetry handles generation and collection, Jaeger handles the backend.

Jaeger's architecture includes agents, collectors, a query service, and a web UI. For development, the all-in-one Docker image combines all components. A common misconception is that Jaeger requires Kubernetes—it doesn't. Jaeger runs on Docker, VMs, or bare metal. The all-in-one image works for local development, while production typically uses separate components with external storage like Cassandra or Elasticsearch.

Jaeger supports multiple ingestion formats, including OTLP. With OpenTelemetry's standardization, OTLP is now recommended, meaning your Spring Boot application sends traces in OTLP format directly to Jaeger without needing Jaeger-specific libraries.

Tracing Beyond Services: Databases and Message Queues

One of the most powerful aspects of distributed tracing is visibility into external dependencies. When your application makes a database call or publishes to Kafka, those operations appear as spans in your trace.

Database tracing works through JDBC instrumentation. When your Spring Boot application executes a SQL query, the OpenTelemetry instrumentation automatically creates a span containing the query, execution time, and database connection details. This visibility is crucial for identifying slow queries or N+1 problems—those situations where you're executing one query to fetch entities, then N additional queries to fetch related data for each entity. Database spans make these anti-patterns immediately visible in your trace timeline. However, be mindful of sensitive data. Database spans can include SQL statements with parameter values, which might contain PII. OpenTelemetry provides span processors to redact or mask sensitive information before export.

Message queue tracing extends traces across asynchronous boundaries. When Service A publishes a message to Kafka, it injects the trace context into message headers. When Service B consumes that message, it extracts the context and continues the trace. This creates a parent-child relationship between the producer and consumer spans, even though they execute at different times. The result is end-to-end visibility into asynchronous workflows, making it much easier to debug message processing issues or track down where data transformations went wrong.

Performance Impact and Production Considerations

Distributed tracing adds overhead from creating spans, serializing data, and network transmission. The impact varies by component:

CPU: Span creation and serialization typically add microseconds per operation. The OpenTelemetry SDK uses efficient batching to minimize per-span overhead.

Memory: The SDK buffers spans before export. Configure batch size and timeout based on traffic patterns and memory constraints to prevent excessive buffering.

Network IO: Sending traces to a local collector over localhost has minimal impact. Remote backends introduce latency and bandwidth usage. Using a collector to batch and compress traces reduces network overhead significantly. Importantly, the collector absorbs most of the performance cost, acting as a buffer between your applications and backends.

In practice, overhead is typically under 5 percent for CPU and memory. The key is intelligent sampling—trace 1-5 percent of traffic in production rather than every request (development should trace 100 percent for debugging). OpenTelemetry supports probability-based sampling for production and rate-limiting to cap traces per second.

Best Practices for Distributed Tracing

Use meaningful span names: "validatePaymentRequest" beats "process" every time. Good naming makes traces self-documenting.

Add relevant attributes: Follow OpenTelemetry semantic conventions for HTTP, databases, and queues. Add custom attributes for business context like user ID or tenant ID.

Don't over-instrument: Creating spans for every method produces noise. Focus on external calls, database queries, and significant business logic.

Implement proper error handling: Mark spans as failed and record exception details when errors occur. This helps identify which service and operation caused failures.

Sample intelligently: Trace everything in development (probability 1.0), but use 1-5 percent sampling in production (probability 0.01-0.05). This gives you statistically significant insights without overloading infrastructure. Consider adaptive sampling that increases rates for slow requests or errors.

Watch for orphaned spans: When requests hand off work to async thread pools, ensure context propagation is maintained. If a new thread loses the trace context, your trace will break, resulting in disconnected "orphaned spans" that can't be correlated. Spring Boot 4 usually handles this automatically, but verify your custom executors are properly instrumented.

Use the Collector: It provides buffering, enrichment, routing, and reliability that SDK exporters alone cannot.

Monitor your telemetry pipeline: Track export success rates and latency. If your pipeline breaks, you're debugging blind.

Querying and Analyzing Traces

Jaeger's UI provides powerful analysis tools. Search for traces by service, operation, tags, duration, and time range. The trace timeline shows the complete request flow with parent-child relationships visually nested. For advanced use cases, Jaeger Query Language (JQL) enables programmatic querying and integration with automated alerting systems. The trace comparison feature helps identify performance regressions by highlighting timing differences between trace versions.

Conclusion

Distributed tracing transforms how you understand and debug microservices. By automatically capturing request flows and timing information, it eliminates the guesswork from performance analysis and incident response. OpenTelemetry provides the standardized instrumentation, OTLP handles reliable transmission, and backends like Jaeger give you the visualization and querying tools to make sense of the data.

Spring Boot 4's native OpenTelemetry support makes adoption straightforward. Add the starter, configure your exporter, and you're tracing HTTP requests, database queries, and message queues with minimal code. The result is a system where every request tells its own story, complete with timing, dependencies, and errors.

Start small. Enable tracing in one service, verify the data reaches Jaeger, and gradually expand to your entire application. The visibility you gain will pay dividends the first time you debug a cross-service issue or optimize a slow endpoint. Distributed tracing isn't just a monitoring tool; it's a fundamental shift in how you understand distributed systems.

For hands-on examples and complete configuration, check out the learning-distributed-tracing repository.

Learning Links:

https://spring.io/blog/2025/11/18/opentelemetry-with-spring-boot
https://opentelemetry.io/docs/zero-code/java/spring-boot-starter/
https://foojay.io/today/spring-boot-4-opentelemetry-explained/
https://last9.io/blog/opentelemetry-for-spring/
https://signoz.io/blog/opentelemetry-spring-boot/
https://vorozco.com/blog/2024/2024-11-18-A-practical-guide-spring-boot-open-telemetry.html
https://medium.com/cloud-native-daily/how-to-send-traces-from-spring-boot-to-jaeger-229c19f544db
https://medium.com/xebia-engineering/jaeger-integration-with-spring-boot-application-3c6ec4a96a6f
https://blog.vinsguru.com/distributed-tracing-in-microservices-with-jaeger/
https://last9.io/blog/distributed-tracing-with-spring-boot/
https://signoz.io/blog/jaeger-vs-zipkin/

OpenTelemetry configuration gotchas

Thu, 14 Aug 2025 11:19:18 +0000

Last week, I described several approaches to OpenTelemetry on the JVM, their requirements, and their different results. This week, I want to highlight several gotchas found across stacks in the zero-code instrumentation.

The promise of OpenTelemetry

Since its inception, OpenTelemetry has unified the 3 pillars of observability. In the distributed tracing space, it replaced proprietary protocols Zipkin and Jaeger. IMHO, it achieved such success for several reasons:

First, a huge industry pressure to work across proprietary tools
Zero-code instrumentation, allowing developers to be unconcerned by OpenTelemetry
Easy and unified configuration mechanism via environment variables.

The latter is a boon for the Ops team, as they don't have to know the underlying framework (or stack!) details. They only need to check the Environment Variable Specification and they are done.

Here's a simple snippet to illustrate my point:

environment:
  OTEL_EXPORTER_OTLP_ENDPOINT: http://collector:4317             #1
  OTEL_SERVICE_NAME: foobar                                      #2

Configure the endpoint to send data to
Set the component's name

From the above, you can't guess what the underlying stack is.

On the OpenTelemetry side, different developers contribute to different language libraries. Developers of the Java agent are different from the those of the Python library. Moreover, each stack has specific approaches and limitations.

It naturally creates differences in the different implementations.

Gotchas

Here are a couple of gotchas I found out, but the list is not exhaustive.

Path or no path?

Let's start easy with a gotcha that exists across stacks.

OpenTelemetry offers some configuration parameters for endpoints.

A generic one, to use for an OpenTelemetry Collector that accepts all signals: OTEL_EXPORTER_OTLP_ENDPOINT
Parameters specialized for a single signal, e.g., OTEL_EXPORTER_OTLP_TRACES_ENDPOINT

In the latter case, one can set the root path to the endpoint, e.g, http://collector:4317. The SDK will automatically append the default path, depending on the signal type. For example, it appends /v1/traces for the OTEL_EXPORTER_OTLP_TRACES_ENDPOINT configuration parameter.

Alternatively, one can set the full path to the endpoint, e.g, http://collector:4317/v1/traces.

The gotcha will catch you when the OpenTelemetry evolves to v2 if you didn't use paths. Because the library will automatically append /v1/<xxx>, you'll have to make sure the backend offers both /v1 and /v2 endpoints.

Python logging

To open the list, here's the first gotcha: by default, the Python library doesn't send logging data. It must be enabled explicitly!

environment:
  OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED: true         #1
  OTEL_EXPORTER_OTLP_ENDPOINT: http://collector:4317
  OTEL_SERVICE_NAME: foobar

Enable logs

Developers also must be involved:

>Unlike Traces and Metrics, there is no equivalent Logs API. There is only an SDK. For Python, you use the Python logger library, and then the OTel SDK attaches an OTLP handler to the root logger, turning the Python logger into an OTLP logger. One way to accomplish this is documented in the logs example in the OpenTelemetry Python repository.

>-- Logs Auto-Instrumentation Example

Micrometer Tracing

Before OpenTelemetry, Jaeger and Zipkin reigned supreme in the distributed tracing area. In the great Spring tradition, the project created Spring Cloud Sleuth to offer a facade over Zipkin. Over time, it evolved to be compatible with OpenTracing, one of OpenTelemetry's parents, along with OpenCensus.

I cannot be sure whether OpenTelemetry was the cause, but Spring Cloud Sleuth evolved into the (badly-named) Micrometer Tracing library. Micrometer Tracing targets Zipkin first, but also supports OpenTelemetry. I find adding the requested libraries a bit complex, but it's manageable.

However, the configuration doesn't conform to the OpenTelemetry environment variables. It brings its variables:

environment:
  MANAGEMENT_OTLP_TRACING_ENDPOINT: http://jaeger:4318/v1/traces #1-2
  OTEL_SERVICE_NAME: foobar                                      #3

Non-OpenTelemetry environment variable name
MUST set the full path
Conform to the OpenTelemetry spec since Spring Boot 3.5. Before it, it used spring.application.name

Quarkus

Compare with Quarkus, which prefixes regular OpenTelemetry environment variables with QUARKUS_:

environment:
  QUARKUS_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://jaeger:4317
  QUARKUS_OTEL_SERVICE_NAME: foobar

It's consistent. And yet, the Quarkus instrumentation has another gotcha:

>Only the tracing signal is enabled by default. To enable metrics and logs, add the following configuration to your application.properties file:

>-- Quarkus instrumentation

Meanwhile, if you're an expert at operating OpenTelemetry, these defaults will surprise you. In OpenTelemetry, they are governed by other configuration parameters:

environment:
  OTEL_METRICS_EXPORTER: none
  OTEL_LOGS_EXPORTER: none

Summary

OpenTelemetry has become a de facto standard in a few years. However, the promise of ubiquitous configuration, if there was ever such a thing, doesn't hold. Operators of OpenTelemetry can't treat services as black boxes. They must consider the underlying stack and framework, and learn how to configure them accordingly.

On the JVM, I'd recommend to stick to the Java Agent as much as possible, but if you must across stacks, there's no magical solution.

To go further:

Originally published at A Java Geek on August 10^th, 2025

OpenTelemetry Tracing on the JVM

Thu, 07 Aug 2025 17:24:20 +0000

You may know I'm a big fan of OpenTelemetry. I recently finished developing a master class for the YOW! conference at the end of the year. During development, I noticed massive differences in configuration and results across programming languages. Even worse, differences exist across frameworks inside the same programming language.

In this post, I want to compare the different zero-code OpenTelemetry approaches on the JVM, covering the most widespread:

Spring Boot with Micrometer Tracing
Spring Boot with the OpenTelemetry Agent
OpenTelemetry Spring Boot Starter
Quarkus
Quarkus with the OpenTelemetry Agent

Commonalities

I keep the architecture pretty simple:

I'm using Reactive data access on both the remote service and the database to spice up things a bit, more specifically, Kotlin coroutines. Here's the general structure:

val products = coroutineScope {
    val ping = async {
        // Call the ping service
    }
    val products = async {
        // Query the database
    }
    println("Received ping response: ${ping.await()}")
    products.await()
}

Here are the features for each stack:

	Quarkus	Spring Boot
Web	Mutiny	WebFlux
HTTP client	REST client	`WebClient`
Database access pattern	Record	Repository
Database access	Hibernate Reactive with Panache	R2DBC

Running the OpenTelemetry Agent

The OpenTelemetry Java Agent is the first approach I used regarding OpenTelemetry.

The only necessary configuration is to set the agent when running the JVM:

java -javaagent:opentelemetry-javaagent.jar -jar otel-boot-agent.jar

The agent supports lots of frameworks and libraries, including Spring Boot, Quarkus, Ktor, Spark, and many others. When the application flow finds a supported framework/library, it logs a span.

The Agent upholds the standard OpenTelemetry environment variables.

services:
  otel-boot-agent:
    build: otel-boot-agent
    environment:
      OTEL_SERVICE_NAME: OTEL Boot Agent                             #1
      OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4318                #2
      OTEL_METRICS_EXPORTER: none                                    #3
      OTEL_LOGS_EXPORTER: none                                       #3

OpenTelemetry service name
OpenTelemetry endpoint; Spring Boot uses HTTP
Neither metrics nor logs

Here's the Jaeger trace when calling the endpoint on the Spring Boot application:

And here's the one on Quarkus:

Spring Boot features an additional span that displays the repository call. There's no such thing available on Quarkus, as I'm using the Record pattern to access the data.

The Agent outputs the SQL query in both frameworks, i.e., SELECT product.* FROM product. The Java Agent works out-of-the-box.

Micrometer Tracing on Spring Boot

Spring Boot provides dedicated OpenTelemetry support via Micrometer Tracing.

>Micrometer Tracing provides a simple facade for the most popular tracer libraries, letting you instrument your JVM-based application code without vendor lock-in. It is designed to add little to no overhead to your tracing collection activity while maximizing the portability of your tracing effort.

Besides the Micrometer Tracing dependency itself, you need additional ones:

The Spring Boot Actuator
A bridge–OpenTelemetry
An exporter–OpenTelemetry as well

Configuration doesn't follow the OpenTelemetry standard:

services:
  otel-boot-micrometer:
    environment:
      SPRING_APPLICATION_NAME: OTEL Micrometer                       #1-2
      MANAGEMENT_OTLP_TRACING_ENDPOINT: http://jaeger:4318/v1/traces #1-3

Different values from the OpenTelemetry specification
The Spring application name serves as the OpenTelemetry service name
Full path to the API endpoint

The above setup doesn't register the database call. To fix it, we need an additional dependency: R2DBC Proxy.

The new trace contains the database span:

You might notice another issue: calls to the service and the database are sequential, where they should be parallel. It stems from Spring Boot not handling context propagation properly to the coroutine scope. It's an underlying work from the Spring team. Subscribe to the GitHub issue if you're interested.

OpenTelemetry Spring Boot Starter

The OpenTelemetry project provides a Spring Boot starter. You need only a single dependency, and like other starters, Spring Boot magic takes care of configuration:

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-spring-boot-starter</artifactId>
</dependency>

The result is very similar to the previous one, including the not-parallel-but-serial issue.

Quarkus

We saw the results of using the OpenTelemetry Agent in the first section. It's quite straightforward to use OpenTelemetry without the Agent; you need a single dependency:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-opentelemetry</artifactId>
</dependency>

Quarkus prefixes regular OpenTelemetry environment variable names with QUARKUS_:

services:
  otel-quarkus:
    environment:
      QUARKUS_OTEL_SERVICE_NAME: OTEL Quarkus                        #1
      QUARKUS_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://jaeger:4317 #2

OpenTelemetry service name
OpenTelemetry endpoint; Quarkus uses gRPC

Results are as expected:

Discussion

OpenTelemetry approaches vary widely in both configuration and results. Unless you're prevented from using Java agents for technical or organizational reasons, I recommend using the OpenTelemetry Agent first. It handles everything you can throw at it out of the box, including the most common libraries. Barring that, you need deep knowledge of the stack you're using, lest results don't represent what happens in reality.

The complete source code for this post can be found on GitHub.

To go further:

Originally published at A Java Geek on August 3^rd, 2025

Even more OpenTelemetry - Kubernetes special

Thu, 10 Apr 2025 07:29:21 +0000

I have presented my OpenTelemetry demo many times, and I still do. Each time, the audience is different. To make no two presentations the same, I always ask attendees what stack they are more interested in. I also regularly add new features for the same reason.

I was a victim of the IT crisis last summer, and my company fired me, so I no longer work on Apache APISIX. They say that the Chinese ideogram for crisis contains the ideogram for opportunity. I used this opportunity to join LoftLabs. LoftLabs offers a cluster virtualization solution called vCluster.

Hence, this batch of changes to the demo has a lot of Kubernetes-related features.

From Docker Compose to Helm

Docker Compose is great for demos: docker compose up, and you're good to go, but I know no organization that uses it in production. Deploying workloads to Kubernetes is much more involved than that. I've used Kubernetes for demos in the past; typing kubectl apply -f is dull fast. In addition to GitOps, which isn't feasible for demos, the two main competitors are Helm and Kustomize. I chose the former for its ability to add dependencies.

Helm subcharts for the win!

I started to migrate by creating my own Helm templates for infrastructure. While it worked, it was hugely inefficient. In the end, I used the following Helm Charts as sub-charts in my infra chart:

At the start, I kept a custom Jaeger manifest because the official Jaeger Helm Chart doesn't provide a way to expose the UI outside the cluster. Then, I realized that I was using Traefik and that I could use it. It's precisely the architecture I used on Docker Compose with Apache APISIX.

I did keep a custom-made manifest for Mosquitto, but that's because the official one doesn't add benefit IMHO: you need a config file.

Kubernetes topology

Besides OpenTelemetry, I want to highlight the capabilities of vCluster. For this reason, I separate the different Docker Compose services into two groups: an infrastructure group (Traefik, Jaeger, Valkey, Mosquitto, and an OpenTelemetry Collector) and an app group (PostgreSQL, Mosquitto, the OTEL Operator, an OTEL collector, and all apps). I deploy infrastructure-related pods on the host cluster and the app pods in a dedicated virtual cluster's default namespace.

helm upgrade --install vcluster vcluster/vcluster --namespace otel --values helm/vcluster.yaml #1
helm upgrade --install otel-infra infra --values helm/infra/values.yaml --namespace otel       #2
vcluster connect vcluster                                                                      #3
helm upgrade --install otel-apps apps --values helm/apps/values.yaml                           #4

Create the virtual cluster in the otel namespace
Deploy infrastructure pods in the otel namespace on the host cluster
Connect to the virtual cluster. I'm now in a completely isolated Kubernetes cluster. Only we know it's a virtual one.
Schedule infrastructure pods in the default namespace of the vcluster virtual cluster

Traefik as an Ingress Controller

When I stopped working on APISIX, I decided to see how API Gateways worked and migrated to KrakenD on Docker Compose. For basic usage, it's simple enough.

I wanted to try another one on Kubernetes. I chose Traefik because my searches mentioned it was the easiest to use in Kubernetes. As I mentioned, Traefik provides a Helm Chart, which makes it easy to install. Additionally, it integrates with OpenTelemetry.

traefik:
  fullnameOverride: traefik
  deployment:
    replicas: 1
  gateway:
    enabled: false
  rbac:
    enabled: true
  ports:
    web:
      nodePort: 30080
  tracing:
    otlp:
      enabled: true                    #1
      grpc:
        enabled: true                  #2
        endpoint: collector:4317       #3
        insecure: true                 #4

Enable OpenTelemetry
Prefer gRPC over HTTP
Set the collector endpoint
Security is not the point of the demo

Exposing services and the Ingress class

The idea of scheduling infrastructure pods on the host cluster and apps pods in the virtual cluster is excellent from an architectural point of view, but how do the latter access the former? You probably use Service to access pods inside the same cluster. Services provide a stable IP over pods that come and go. Across regular clusters, you'll need an Ingress Controller or a Gateway API implementation. vCluster offers a straightforward alternative.

In the demo, I must:

Access services on the host cluster from pods scheduled on the virtual cluster
Configure Ingresses on the virtual cluster with an Ingress class whose Ingress Controller is deployed on the host cluster

As seen above, we can configure both when we create the virtual cluster.

networking:
  replicateServices:
    fromHost:
      - from: otel/valkey-primary      #1
        to: default/cache
      - from: otel/jaeger              #1
        to: default/tracing
      - from: otel/traefik             #1
        to: default/ingress
      - from: otel/mosquitto           #1
        to: default/messages
sync:
  toHost:
    ingresses:
      enabled: true                    #2

Synchronize a Service from the host cluster to the virtual cluster. The pattern is <namespace>.<name>
Ingresses deployed on the virtual cluster are collected on the Ingress Controller configured on the host cluster

A touch of the OpenTelemetry collector

I chose not to use an OpenTelemetry Collector in the Docker Compose setup because it didn't add anything. With the new two-cluster architecture, it was beneficial to introduce one per cluster.

At first, I configured each Collector to add a custom tag regarding the cluster type, host or virtual. Then, I discovered that the Collector Helm Chart provided a Kubernetes preset.

opentelemetry-collector:
  image:
    repository: "otel/opentelemetry-collector-k8s"
  fullnameOverride: collector
  enabled: true
  mode: deployment
  presets:
    kubernetesAttributes:
      enabled: true

It automatically adds Kubernetes-related data. Here's the result in one of the app pods:

Key	Value
`k8s.deployment.name`	`warehouse-uk`
`k8s.namespace.name`	`default`
`k8s.node.name`	`orbstack`
`k8s.pod.ip`	`192.168.194.44`
`k8s.pod.name`	`warehouse-uk-5965658cfd-sbkgm`
`k8s.pod.start_time`	`2024-11-29T16:36:30Z`
`k8s.pod.uid`	`b6ba838e-621c-46b1-ac32-c9225c80e142`

Initializing PostgreSQL with data

The "official" PostgreSQL Docker image allows initializing the database via SQL scripts. You only need to mount the scripts folder to the container's /docker-entrypoint-initdb.d folder. When the container starts, it will run these scripts in alphabetical order. On Kubernetes, I started to write all the scripts' content in a static ConfigMap, but I didn't find this approach maintainable.

Given I'm using a Helm Chart, I thought I could find a better way, and I did. I put all the scripts in a dedicated files/sql folder in the Helm Chart. Then, I wrote the following template to concatenate them in a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-init-scripts
  namespace: {{ .Release.Namespace }}
data:
{{- $files := .Files.Glob "files/sql/*.sql" }}    #1
{{- range $path, $content := $files }}            #2
  {{ base $path }}: >                             #3
{{ $content | toString | indent 4 }}              #4
{{- end }}

List all files under the files/sql/ folder
Iterate through all files
Write the file name
Write the file's content

At this point, we can use the ConfigMap in the global Helm file under the PostgreSQL section:

postgresql:
  primary:
    persistence:
      enabled: false
    initdb:
      scriptsConfigMap: postgres-init-scripts

Kubernetes instrumentation of pods

I recently learned that Kubernetes could instrument pods running certain technologies with OpenTelemetry. I wanted to try it out, so I added a new OpenTelemetry-free Python-based image to the overall architecture. It's a two-step process:

Install the OpenTelemetry Operator
Apply an Instrumentor:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: demo-instrumentation
spec:
exporter:
```
 endpoint: http://collector:4318               #1
```
propagators: #2
```
 - tracecontext
 - baggage
```
sampler:
```
 type: always_on                               #3
```
Configure the OpenTelemetry collector endpoint
Set the propagators
I want all the traces for my demo; in production environments, you'd probably want to sample them

At this point, when we schedule pods, Kubernetes will pass them through a new mutating webhook. If the pod has annotations that the instrumentation recognizes, the latter will add a sidecar that instruments the main pod with OpenTelemetry. For the Python app I mentioned above, it looks like the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: recommendations
  labels:
    app: recommendations
spec:
  replicas: 1
  selector:
    matchLabels:
      app: recommendations
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-python: "true"     #1
      labels:
        app: recommendations
    spec:
      containers:
        - name: recommendations
          image: recommendations:1.0

The mutating webhook instruments the pod with the Python-specific image

Here's a screenshot of the resulting traces:

What's impressive is that it's all based on Kubernetes. Neither the app developer nor the container build had to do anything related to OpenTelemetry.

A new Quarkus component

This last section is unrelated to Kubernetes. I'm already using Spring in two components for the demo, one instrumented with the OpenTelemetry Java agent and the other with Micrometer Tracing and compiled with GraalVM native. I wanted to demo the OpenTelemetry integration capabilities of Quarkus.

You only need two dependencies for auto-instrumentation:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-opentelemetry</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-jdbc</artifactId>
</dependency>

You can customize the traces with annotations:

 @Path("/stocks")
 @Produces(MediaType.APPLICATION_JSON)
 public class StockLevelResource {
 
     private final StockLevelRepository repository;
 
     @Inject
     public StockLevelResource(StockLevelRepository repository) {
         this.repository = repository;
     }
 
     @GET
     @Path("/{id}")
     @WithSpan                                                      //1
     public List<StockLevel> stockLevels(@PathParam("id") @SpanAttribute("id") Long id) { //2
         return repository.findByProductId(id);
     }
 }

Add a new span whose value is the method's name
Add a new id attribute whose value is the id's value at runtime

Summary

I added a couple of new features to my existing OpenTelemetry demo:

Migrating from Docker Compose to Kubernetes
Installing via Helm Charts
Using vCluster to isolate the app pods from the infrastructure pods
Using Traefik as an Ingress Controller
Installing an OpenTelemetry collector instance in each cluster
Instrumenting an OpenTelemetry-free app with the Kubernetes Instrumentation
Adding a new Quarkus application

I'm always happy to add new components that feature a new stack. At the moment, I'm thinking .Net is missing from the landscape. Suggestions and PRs are welcome!

To go further:

Originally published at A Java Geek on April 6^th, 2025