Coder Legion - Recent posts tagged llm-evaluation

The four axes generic RAG eval forgets for search agents

Tue, 18 Aug 2026 09:20:10 +0000

A search-augmented agent looks like RAG until the citation list breaks. The first user clicks through, lands on a 404, and the trust budget spends in one impression. Three months of eval work, groundedness and context adherence and factual accuracy all green, did not catch it.

The reason is structural. Generic RAG eval was built for a frozen vector store and a stable ground truth. Search agents have neither. The corpus is the live web, the agent writes the literal query string that drives what comes back, sources span government filings and LinkedIn takes in one result set, and users click the URLs. Skip the four axes unique to this shape (query quality, source freshness, source diversity, citation validity) and you ship an agent that confidently cites stale or fabricated URLs. The rest of RAG eval still applies, but the rest of RAG eval was already passing. There is a longer version with the rubric code if you want it.

Three differences pull this out of the RAG bucket. The corpus is not yours: two runs of the same query twelve hours apart return different URLs, so reproducibility lives in a snapshot, not the results page. The query is the lever: search is sensitive to the literal string the agent writes, and a rewrite that drops a constraint returns a clean-looking set that misses it, which no synthesis prompt recovers from. And the user verifies: a citation is a URL the user clicks, so a fabricated or 404 link is a one-click trust failure that groundedness against the retrieved set never sees.

Score the four axes per query, not aggregated. A single number hides which one collapsed, and the diagnostic value is in the vector.

Axis 1: query quality

The agent turns the user's prompt into a search string, and that string is what the engine literally reads. Three failure modes show up in almost every run.

Constraint drop. "Laptops with battery over 10 hours" compresses to "best laptops 2026 for ML," and the results average five-hour batteries. Groundedness is fine because the brief grounded itself in the retrieved set. The constraint is gone.
Entity loss. A question about a specific company or regulation abstracts to the category, and the brief answers a different question well.
Over-broad rewrite. A narrow factual lookup expands into a topic, the search returns a Wikipedia overview, and the brief paraphrases it and misses the fact.

Score it with a judge that takes the original prompt and the rewrite as a pair and rates constraint preservation and entity recall 1 to 5, penalizing unnecessary expansion as much as a drop. Run this over a week of traffic and the first cluster is usually "rewriter dropped the year, version, or region" on roughly one in eight queries. Cheapest fix in the loop, and the one most teams skip.

Axis 2: source freshness

Freshness is a function of the question, not a global setting. "Capital of France" is insensitive. "Current price of X" is real-time. "FDA guidance on AI as a medical device" is quarter-sensitive. The agent has to decide the regime and behave: prefer recent sources, refuse stale ones, and add a recency caveat when the answer depends on current data.

Two checks carry it. Classify the freshness regime per query (insensitive, slow, fast, real-time) and attach it as a span attribute. Then attach a recency value per source, take the median recency of the cited sources, and fail when it exceeds the floor for that regime. The stale-fact bug is the one users notice last and complain about loudest: correct the day the source was indexed, wrong every day after, and invisible to every other metric.

Axis 3: source diversity

A general search returns Wikipedia, vendor blogs, news, government docs, and SEO spam in one set, and the agent cites three to five. The failure is monoculture: every citation from one domain, or a brief that reads like consensus when it summarizes one tab.

Three checks cover it, and the first two are deterministic and run in microseconds:

Unique-domain count per citation set, with a floor by question class. A date lookup can be tight; a synthesis question needs four or more.
Per-domain cap. No domain contributes more than two citations, unless the question is about that domain (an EU AI Act question can cite eur-lex repeatedly).
Primary-vs-secondary ratio. Classify each citation as primary (a filing, paper, or announcement) or secondary (commentary, aggregator). A regulatory question with zero primary citations is a failure however grounded the brief is.

Axis 4: citation validity

The highest-signal cheap check in the whole stack, and the one most teams find only after the first complaint. Three sub-checks compose it.

The URL resolves. HEAD-request every citation; a non-2xx or a redirect to a different domain is a fail. Tens of milliseconds, and it rejects fabricated and stale links before the user sees them.
The passage exists on the page. Fetch the body and confirm the quoted or paraphrased passage is actually there. A paraphrase that drifts past a similarity threshold is a fail. This catches a real URL with an invented quote.
The claim is supported by that passage. Groundedness and chunk attribution, but per claim, not per answer. A brief can be 0.94 grounded as a whole and 0.61 aligned per claim, and users only see the per-claim view.

Wire it as two stages: the deterministic checks (resolves, passage present) as a hard gate, the claim-passage entailment as a judge on the synthesis span. A citation that fails any check is a hard reject: log it, regenerate the answer with that URL excluded, and re-run the gate. Teams that wire this stop seeing the "user clicked a 404" failure almost immediately.

Cache the snapshot so CI does not flake

Two runs of the same prompt hours apart return different URLs, so without a per-test-case cache the gate flakes for reasons unrelated to the agent and the team stops trusting it. Cache three layers: the search call (query and full result JSON), the page fetch (URL, body, timestamp), and the extraction (cleaned text, date, anchors). Replay every run from cache, and re-snapshot deliberately on a schedule that fits the domain (weekly for news, monthly for stable how-to agents), treating each re-snapshot as a new golden-set version with its own baseline.

Trace it so a bad answer is debuggable

The eval scores attach to spans, and the spans are how you debug. Three span kinds carry the load: a retriever span per search call, carrying the literal query the agent sent plus per-source URL, recency, and authority; a tool span for the wire call to the search API, so latency and cost land on the right node; and an LLM span for the synthesis. Splitting the search call from the retriever span keeps the query-quality input and the network cost on separate, queryable nodes.

Run the four axes per query and read the vector. The axis that drops is the bug, and the standard RAG metrics keep running alongside because they are necessary, just not sufficient.

Three drifts that quietly age your LLM eval set

Fri, 17 Jul 2026 14:15:00 +0000

I keep seeing the same failure land on teams that do everything right. The CI eval gate has been green for six weeks, faithfulness at 0.87, task completion at 0.91. Then a support thread surfaces a hallucination on a feature shipped in March. The trace looks clean: well-formed query, plausible retrieval, wrong answer. The failure is not in the eval set. Eight percent of this month's traffic is asking about that feature, and the golden set has zero examples of it. The eval was never wrong about what it covered. It was covering the wrong product.

That is eval dataset drift, and I think it is the most under-managed risk in an LLM eval stack. Your golden set was representative the day you built it. Since then the production query mix has shifted, the prompt template has moved, and the retrieval corpus has rotated, and the eval keeps grading a frozen snapshot of last quarter. It ages in three places, and the score never tells you.

Why the eval score never warns you

A golden set is a snapshot. Production is a distribution. On the day you author it, the snapshot is a representative sample of that distribution. Every day after, the distribution moves and the snapshot does not, and the score stays flat because the eval is still grading the snapshot against itself.

New product surface adds intents you never anticipated. Campaigns and regional rollouts shift the persona and language mix. Users find prompt patterns the curators never imagined. The uncomfortable part is that none of it breaks the API or moves the dashboard. You find out from a customer, which is the most expensive way to learn it.

Drift 1: production prompts move away from the golden set

This is the one you can picture before you measure it. Your golden set has 200 prompts across the intents you knew about when you built it. Production users find an intent you did not anticipate, a campaign shifts the persona, a regional rollout brings a language that tokenizes differently. An edge case that was 0.5 percent of traffic on launch day is now 8 percent.

The signal is embedding distance. Compute one centroid for the golden set on the same embedder you use for retrieval, keep a rolling 7-day centroid over production traffic, and track the cosine distance between them. When it crosses two standard deviations of a 30-day baseline, the distributions have separated and the set has stopped covering the long tail.

For sharper attribution, tag traces with an intent label and compute KL divergence against the golden set's intent mix, or cluster the production embeddings nightly and watch for a new cluster with no golden-set neighbor. Fifty traces in a brand-new cluster is a coverage gap you can file before users complain.

Drift 2: the prompt wrapper changes and the dataset does not

This is the one that hides best, and the one I have watched bite the most teams. Your golden set is 200 curated inputs, but what the model actually sees is those inputs wrapped in a system message, a few-shot block, and a tool schema. That wrapper is the prompt template, and it is not part of the dataset.

Here is how it goes wrong. You ship a system-message tweak on Friday to fix a tool-call issue. The fix is good. The eval set still wraps inputs in last quarter's template, so Monday's CI run grades the dataset under a contract production no longer uses, and the score stays plausible because the rubric was generic enough to grade either one.

Detection is simple: hash the active template every night and compare it against the template hash stored with the eval set, then alert on a mismatch. The fix is dataset versioning. Pin every dataset version to the template hash it was written against, and when the template ships, cut a new dataset version and re-baseline the gate. Keep the previous version around for a release cycle so you can compare the two scores and tell model behavior apart from dataset decay.

Drift 3: the retrieval corpus moves under a RAG eval

This is the RAG-specific one, and it moves the score least while it happens, which is exactly why it is dangerous. Your golden set has fixed inputs and expected outputs, but the retrieval corpus is not in the dataset. The index grows, the chunker re-embeds on a new model, sources rotate, all on a cadence the dataset never sees.

Concretely: the retriever you evaluated in March indexed 12,000 documents at chunk size 800. By May the index has 38,000 documents on a fresh embedder, and the same golden-set query lands on different top-k chunks. The generator grounds faithfully in whatever it was handed, so groundedness still scores 0.94, and the user gets a confident answer from a document that does not apply.

The signal is overlap. Snapshot the retrieved chunk IDs for every golden-set query when you author the dataset. On each refresh, re-retrieve and compare the new top-k against that baseline, and alert when overlap drops past roughly 40 percent. Score the retrieval rubrics (context relevance, chunk attribution) separately from the generation rubrics (groundedness, factual accuracy) so you can bisect: context relevance dropping with groundedness steady means the retriever moved; the reverse means the generator did.

The monthly refresh, in four steps

Detection tells you the set has aged. The refresh is what you do about it, and I tell teams to run it as a standing cadence, not a fire drill: monthly by default, weekly under fast change, and immediately after a model swap or a corpus re-index.

Pull 200 to 500 recent production traces, stratified by intent, persona, and language, from the last two to four weeks so you do not over-fit to a single campaign.
Label them against the rubric already in CI, hand-corrected, with SME review where the domain is complex.
Version the set as a new named dataset (golden-v4-2026-06), pinning the template hash, embedder version, and retrieval snapshot to its metadata, and keep the previous version live for a cycle.
Re-baseline the CI gate against the new set. The score delta between the old and new versions on the same model tells you how much of the last "regression" was dataset decay. Most teams find 30 to 50 percent of the movement between refreshes was the dataset, not the model.

The anti-patterns I see most

Treating the golden set as ground truth instead of a versioned artifact, so it never moves and slowly becomes a snapshot of last quarter's product.
No input-distribution monitoring, when the embedding compute is already running for retrieval and you can reuse it for the drift signal.
Versioning prompt templates and eval sets in separate places, so a template ship lands while the dataset still grades the old contract.
No retrieval-corpus snapshot, so RAG-side decay stays invisible until users complain.

The teams shipping reliable LLM products are not the ones with the biggest golden sets. They are the ones whose sets refresh on a cadence. Input drift moves the distribution, template drift moves the contract, corpus drift moves the grounding. Build the three baselines on day zero (embedding centroid, template hash, top-k snapshot), run the nightly comparison, and version every refresh. The drift will still arrive. That discipline is what decides whether you see it the sprint it shows up or the quarter after, and whether you hear about it from your dashboard or from a customer.