Coder Legion - Recent posts tagged agent-evaluation

How to evaluate agents with one trace across MCP, A2A and Realtime

Wed, 14 Oct 2026 16:22:41 +0000

A support agent passes every eval in staging. In production the same agent is reached three ways: from an IDE over MCP, from the web app over the OpenAI Responses API, and from a triage agent over A2A. The dashboard stays green for weeks. Then a customer escalation shows the triage agent handed over a ticket without the customer's region, and the support agent quoted the refund policy for the wrong country. No evaluator flagged it, because no evaluator ever looked at the hop.

The reason is structural. Eval is a property of the path a request took, and the protocol defines the path. Most eval stacks were written against one surface, usually chat completions, so they score whatever still looks like a chat completion and quietly skip the rest. A protocol-neutral eval stack scores the path instead, whichever protocol carried the request.

Every protocol adds seams to the path

A plain chat completion has one path: prompt in, response out. One rubric can grade it. Each protocol you add forks that path into more decision points, and each decision point needs its own check.

MCP adds a tool catalog. The path becomes prompt, tool descriptor list, tool selection, call arguments, tool result, final answer. Score each tool call against policy, and score the catalog as a whole, because a poisoned descriptor looks fine at call time and only shows up against the full catalog.
A2A turns the path into a tree. Every hop needs three answers: did the calling agent pass enough context, did the called agent stay in scope, and did the returned artifact match the task that was asked for.
The Responses API puts three outputs in one envelope. Structured JSON gets graded against a schema, streamed text gets graded as it arrives, and a tool call gets graded against a per-tool policy. One end-to-end pass mixes three things that share nothing except the wrapper.
Realtime has no final message. Audio streams in both directions, so checks have to run mid-stream, every N tokens or N audio chunks, with a stop or disclaimer action when a rule fires. The end-to-end budget sits below 500ms, which caps how heavy each check can be.
Inline vision needs a second judge. When images ride in the same message as tool blocks, the text portion and the image portion need different rubrics, and the image rubric needs a judge that can open the image.
Session state spans turns. On a framework with explicit session state, the evaluator has to read state across turns instead of grading each turn alone.

Trace fidelity caps eval fidelity

An evaluator can only score what the trace can represent. If your trace has no way to express an agent-to-agent hop or a streaming chunk, no rubric downstream will catch a failure inside one.

So the first fix is in instrumentation, before any rubric work:

Model both sides of a hop. Emit one span kind on the calling agent and another on the called agent, and carry the trace ID across the wire so the whole multi-agent path lands in one trace.
Give each modality its own span kind. Voice, image and computer-use spans should be distinguishable from text spans, so each one reaches an evaluator built for that modality.
Record guardrail and evaluator results as spans. When a guardrail decision and an evaluator score sit in the same trace as the model call, you can see which check ran on which step.

The MCP spec helps here. The 2026-07-28 revision documents trace context propagation through _meta (traceparent, tracestate, baggage), so MCP tool calls can join the same trace as the rest of the agent loop.

Normalize the spans, then write each rubric once

The same agent identity shows up differently on every protocol. If you write a groundedness check against raw MCP payloads and another against raw Responses payloads, you now maintain two checks that drift apart.

Normalize first. Map every span to one semantic convention, such as the OpenTelemetry GenAI conventions, before any evaluator reads it. Then write rubrics against the normalized trace. One groundedness rubric can then grade a RAG answer that arrived over the Responses API and a RAG answer that arrived over MCP, and one task-completion rubric can grade the support agent no matter which client reached it.

This is the same move observability teams made with OpenTelemetry. Teams that locked to one vendor's tracing format rebuilt their instrumentation every time they switched vendors. Agent protocols are repeating that pattern one layer up.

Gate releases per protocol

A single overall score hides which surface regressed. A drop in the MCP tool-audit rubric and a drop in the Realtime guardrail are different failures with different fixes.

Keep a regression dataset per protocol surface. Run the matching rubric set against it on every change.
Set a threshold per protocol. Fail the release when any one surface drops below its own threshold, even if the blended number looks fine.
Key each suite on protocol and version. MCP's 2026-07-28 revision removed the initialize handshake, so every request now carries its protocol version in _meta, and the new deprecation policy gives every deprecated feature a minimum twelve-month window. Tag each regression case with the revision it ran on, and keep the tests for deprecated features until their window closes.
Keep the rubric sets in the repo as code. Reviewers should be able to read exactly what gets scored on each surface.

A five-step readiness audit

Inventory every surface in production. MCP servers, A2A endpoints, streaming, voice, vision and computer-use, with the protocol version and client library for each. Most teams find one they did not plan for, usually an MCP server a developer installed last sprint.
Instrument and pull one real trace per surface. Confirm the hop spans appear on both sides with the propagated trace ID, and that voice and image calls emit their own span kinds. Fix gaps before writing rubrics.
Write the rubric set for each protocol. Per-tool audit on MCP, per-hop audit on A2A, split rubrics on the Responses API, mid-stream checks on Realtime, an image-capable judge for vision.
Run a per-protocol regression suite and gate on per-protocol thresholds.
Cluster production failures by protocol. Failures that look like noise one at a time often form a pattern once you group them by surface.

If you only have half a day, do step 1. The list is usually longer than the team expects, and it tells you which evaluators are scoring the wrong path today.

Pitfalls that keep showing up

Evaluators hard-coded to the chat completions shape. Every new protocol triggers a rewrite.
Scoring each agent in isolation. The end-to-end view looks green while a handoff in the middle dropped context.
Post-call checks only on streaming surfaces. By the time the evaluator sees the output, the risky tokens are already on the wire.
Accepting tool descriptors from any MCP server without a scan. An untrusted server can change what your agent does, and the eval layer never sees it.
Scoring voice and computer-use on text alone. Errors at the ASR layer and the vision layer stay invisible.

The span kinds, the rubric set for each of the six protocols and the full audit are in the longer protocol-neutral eval playbook.

New protocols and revisions will keep arriving. Score the path the request took, and each new surface plugs into a trace model and a rubric library you already maintain.

The four axes generic RAG eval forgets for search agents

Tue, 18 Aug 2026 09:20:10 +0000

A search-augmented agent looks like RAG until the citation list breaks. The first user clicks through, lands on a 404, and the trust budget spends in one impression. Three months of eval work, groundedness and context adherence and factual accuracy all green, did not catch it.

The reason is structural. Generic RAG eval was built for a frozen vector store and a stable ground truth. Search agents have neither. The corpus is the live web, the agent writes the literal query string that drives what comes back, sources span government filings and LinkedIn takes in one result set, and users click the URLs. Skip the four axes unique to this shape (query quality, source freshness, source diversity, citation validity) and you ship an agent that confidently cites stale or fabricated URLs. The rest of RAG eval still applies, but the rest of RAG eval was already passing. There is a longer version with the rubric code if you want it.

Three differences pull this out of the RAG bucket. The corpus is not yours: two runs of the same query twelve hours apart return different URLs, so reproducibility lives in a snapshot, not the results page. The query is the lever: search is sensitive to the literal string the agent writes, and a rewrite that drops a constraint returns a clean-looking set that misses it, which no synthesis prompt recovers from. And the user verifies: a citation is a URL the user clicks, so a fabricated or 404 link is a one-click trust failure that groundedness against the retrieved set never sees.

Score the four axes per query, not aggregated. A single number hides which one collapsed, and the diagnostic value is in the vector.

Axis 1: query quality

The agent turns the user's prompt into a search string, and that string is what the engine literally reads. Three failure modes show up in almost every run.

Constraint drop. "Laptops with battery over 10 hours" compresses to "best laptops 2026 for ML," and the results average five-hour batteries. Groundedness is fine because the brief grounded itself in the retrieved set. The constraint is gone.
Entity loss. A question about a specific company or regulation abstracts to the category, and the brief answers a different question well.
Over-broad rewrite. A narrow factual lookup expands into a topic, the search returns a Wikipedia overview, and the brief paraphrases it and misses the fact.

Score it with a judge that takes the original prompt and the rewrite as a pair and rates constraint preservation and entity recall 1 to 5, penalizing unnecessary expansion as much as a drop. Run this over a week of traffic and the first cluster is usually "rewriter dropped the year, version, or region" on roughly one in eight queries. Cheapest fix in the loop, and the one most teams skip.

Axis 2: source freshness

Freshness is a function of the question, not a global setting. "Capital of France" is insensitive. "Current price of X" is real-time. "FDA guidance on AI as a medical device" is quarter-sensitive. The agent has to decide the regime and behave: prefer recent sources, refuse stale ones, and add a recency caveat when the answer depends on current data.

Two checks carry it. Classify the freshness regime per query (insensitive, slow, fast, real-time) and attach it as a span attribute. Then attach a recency value per source, take the median recency of the cited sources, and fail when it exceeds the floor for that regime. The stale-fact bug is the one users notice last and complain about loudest: correct the day the source was indexed, wrong every day after, and invisible to every other metric.

Axis 3: source diversity

A general search returns Wikipedia, vendor blogs, news, government docs, and SEO spam in one set, and the agent cites three to five. The failure is monoculture: every citation from one domain, or a brief that reads like consensus when it summarizes one tab.

Three checks cover it, and the first two are deterministic and run in microseconds:

Unique-domain count per citation set, with a floor by question class. A date lookup can be tight; a synthesis question needs four or more.
Per-domain cap. No domain contributes more than two citations, unless the question is about that domain (an EU AI Act question can cite eur-lex repeatedly).
Primary-vs-secondary ratio. Classify each citation as primary (a filing, paper, or announcement) or secondary (commentary, aggregator). A regulatory question with zero primary citations is a failure however grounded the brief is.

Axis 4: citation validity

The highest-signal cheap check in the whole stack, and the one most teams find only after the first complaint. Three sub-checks compose it.

The URL resolves. HEAD-request every citation; a non-2xx or a redirect to a different domain is a fail. Tens of milliseconds, and it rejects fabricated and stale links before the user sees them.
The passage exists on the page. Fetch the body and confirm the quoted or paraphrased passage is actually there. A paraphrase that drifts past a similarity threshold is a fail. This catches a real URL with an invented quote.
The claim is supported by that passage. Groundedness and chunk attribution, but per claim, not per answer. A brief can be 0.94 grounded as a whole and 0.61 aligned per claim, and users only see the per-claim view.

Wire it as two stages: the deterministic checks (resolves, passage present) as a hard gate, the claim-passage entailment as a judge on the synthesis span. A citation that fails any check is a hard reject: log it, regenerate the answer with that URL excluded, and re-run the gate. Teams that wire this stop seeing the "user clicked a 404" failure almost immediately.

Cache the snapshot so CI does not flake

Two runs of the same prompt hours apart return different URLs, so without a per-test-case cache the gate flakes for reasons unrelated to the agent and the team stops trusting it. Cache three layers: the search call (query and full result JSON), the page fetch (URL, body, timestamp), and the extraction (cleaned text, date, anchors). Replay every run from cache, and re-snapshot deliberately on a schedule that fits the domain (weekly for news, monthly for stable how-to agents), treating each re-snapshot as a new golden-set version with its own baseline.

Trace it so a bad answer is debuggable

The eval scores attach to spans, and the spans are how you debug. Three span kinds carry the load: a retriever span per search call, carrying the literal query the agent sent plus per-source URL, recency, and authority; a tool span for the wire call to the search API, so latency and cost land on the right node; and an LLM span for the synthesis. Splitting the search call from the retriever span keeps the query-quality input and the network cost on separate, queryable nodes.

Run the four axes per query and read the vector. The axis that drops is the bug, and the standard RAG metrics keep running alongside because they are necessary, just not sufficient.