Coder Legion - Recent posts tagged evals

Jev is not as cheap as you think. Unless it is.

Sat, 17 Oct 2026 19:19:48 +0000

Recently, TypeSafe launched Jev, the first of what people are calling System One or decision models: your app sends state plus a query, and Jev sends back typed decisions with probabilities instead of prose. All with a sub-second response time, $0.042 per million input tokens, and output tokens totally free. A viral concept indeed: other open-source alternatives sprang up overnight, with even one from OpenAI within weeks. The play: use Jev for lightning-fast decisions rather than a lumbering general LLM.

Could this work for decisions in DumbQuestion.ai?

My satirical AI has two places that evaluate user input before any answer model runs. Search intent: does this question need a web search? Self-awareness: is this person poking at whether the AI knows it's alive? Both run in-process today, a regex for one and an embedding plus an in-memory vector set for the other.

My hypothesis: Jev could replace both, reduce complexity, raise accuracy, and still fit my brutally efficient budget, which currently hovers around $20 per million questions asked.

So I ran a proof-of-value experiment against DumbQuestion to find out. Is it as cheap as it looks? Does it beat what I have? Do the container savings offset the inference cost? And what happens to time-to-answer?

How it works today

Web searches cost money (even if it's just a little), so I only want to search if the user's question needs it. Today, I use a simple regex gate: if the question wants current facts, search; otherwise answer from the model alone. Free, instant, in-process, still supporting the brutally efficient mantra: one Go binary, a sub-20 MB container, still $20 per million questions.

Self-awareness is subtler. A regex can't catch "are you trapped in there" phrased a thousand ways, so I generate a dense embedding of the question and semantically compare it against a vector set of 433 known probes, all in memory. The design constraint was low false positives. Wrongly glitching the UI on a sincere philosophy question (answered sarcastically, of course) is worse than missing a probe. On my labeled set of 1,066 questions, that setup scored 99.1%.

Put a pin in that number. It comes back later.

Running the PoV

The Jev primitive I care about is the Noul question: a yes/no probability tested against a threshold, so tuning means moving a number instead of retraining anything or adding more and more questions to the vector set. Better still, both of my checks collapse into one call: two questions, "does this question need a web search" and "is this person probing self-awareness", answered in parallel.

I ran this as a coding-session PoC with Claude Code Cloud and Pocock's Wayfinder skill, evaluating against OpenRouter's new Decisions API using Jev. The process had five steps: research how to best call the API, prototype the one-call two-question request, run the comparison on my labeled sets while recording per-case probabilities (so thresholds could be tuned later without new calls), compute the cost per million questions, then line up cost, accuracy, and experience side by side.

Step three's bake-off, though, is where things got interesting.

My baseline was flattering itself

One thing I did not anticipate was the need to audit my current eval sets. Two lessons learned: First, 353 of my 369 self-awareness positives were embedded in the app's vector set, so the detector was being graded on questions it had memorized. Second, my search-intent labels were derived from the regex, so the regex was grading its own homework. Apologies to the data scientists in the crowd who are shaking their heads, twice.

The 99.1% (remember?) was real in the way a rigged game is real, rigged twice over, accidentally, by me.

To fix it, I used Claude Code to build an independent fair set and re-ran everything: regex, embeddings, and Jev, same questions, thresholds tuned from the recorded probabilities.

What the evals said

Cost. Measured with the production embedding model and the real Jev request. Jev, with both intents in one call, runs a median 790 input tokens: $0.000033 per question, or $33.20 per million questions. Today's embedding call, though, is $0.000000048 per question ($0.05 per million), and the regex is free. That is roughly a 700x increase over a cheap embedding and free regex. For scale, one Gemma 3 12B answer costs me about $0.00005, so Jev would add roughly 60% to per-question model spend. The problem is that nearly all of Jev's cost is fixed input overhead: my criteria text runs about 700 tokens and the question itself adds 10 to 30. At the time, no provider caching was available to help.

Accuracy. On the (now) fair eval set, my current embeddings missed 54% of self-awareness positives. Jev at a 0.7 threshold had zero false positives and 6% false negatives. On the legacy negatives, the embeddings false-fired on 53 questions (7.6%, mostly generic consciousness and philosophy phrasings) while Jev flagged 3 (0.4%). Search intent: the regex missed 53% of positives and fired on 18 timeless questions it should ignore. Jev at a 0.3 threshold got all 84 fair-set cases right.

Resources. I measured what the embeddings actually cost in memory to see if I could cover the cost of Jev by reducing Cloud Run costs. The 433 vectors leave about 1.7 MiB live on the heap (the 5 MB source file is baked into the binary and only paged in when read). The container already runs at the platform default and there is no tier to drop to, so removing the vectors saves roughly nothing on the runtime bill. The money buys accuracy and no dataset upkeep. My time is free to me, but if you are paying someone to constantly maintain and train a dataset, then there are more savings to be had.

Latency. Jev's decision call came back at a p50 of about 350 ms from my sandbox. The embeddings path, which would be removed, was already a network hop, so only the regex was ever instant. Production time-to-answer numbers are still pending, but this is a same-order-of-magnitude swap, so not a new usability tax.

Does the accuracy even matter?

The accuracy is way higher than the legacy approach, but so is the cost. Does that cost provide real, tangible impact and value?

Determining search intent improves answer quality and protects search spend. The regex missing 53% of positives means answer-worthy questions skip search, and firing on 18 timeless questions means wasted search budget. At 1 million questions asked, if I searched half of those using Serper, that runs about $500 at the standard rate ($375 on a volume plan). Honestly, the regex can only do so much work here, and I think it has already hit its accuracy ceiling.

Detecting self-awareness provides the fun part. Missing 54% of positives means more than half the people who would have found the easter eggs and poked at the containment grid got a normal roast instead, and those 53 false positives mean 7.6% of ordinary philosophy questions got a glitchy UI and a confusing experience.

Where I landed

Both of the points above matter, but only at scale. On a slow month, the likelihood of either of these happening is quite low, and my current traffic would never surface the cost. But at my "million questions asked" benchmark, they can both cost money and ruin the experience.

Jev was more accurate on every axis I measured, at a real (but small) and somewhat tunable cost, with no resource savings to offset it. Should a decision model provide caching, though, it would drastically change the math.

Do I stick with my legacy approach, trading accuracy for lower cost? Or do I join the viral trend and inject Jev into my app? I haven't merged that PR yet as I struggle with this... decision.

Your turn

Apply these principles to your own apps: run the math at scale, and evaluate the true end value, which isn't always about pure cost.

If the experience ends up invisible to your users, keep the regex and in-memory vectors, and spend practically nothing. If it engages your users and encourages them to stick around, the accuracy is worth the cost.

Agent Evals in Practice: Offline, Online, and Braintrust vs LangSmith vs Datadog

Mon, 28 Sep 2026 05:43:07 +0000

Originally published on thomaskalnik.com.

I have set up evals for two very different systems. The first was the chat agent in Spotter Studio, a now-retired tool that helped YouTube creators come up with video ideas and titles. The evals there covered routing and behavior: did a request go to the right part of the agent, and did the agent then do what it was supposed to do. The second was a retrieval system at a nonprofit, where the question was whether semantic search over social posts and media returned the right items, and whether a new embedding model was actually better than the old one.

Both used Braintrust. I have also used LangSmith and Datadog's LLM Observability. This post covers what I would set up again, the mistakes worth skipping, and how the three platforms compare today.

What an eval is made of

Every eval platform uses the same four parts, with different names:

Dataset: inputs, and where possible the expected output or the facts a good output must contain.
Task: the code under test. It takes one input and returns an output: a route, a tool call, an answer, a ranked list.
Scorers: functions that turn an output into a number. Some are plain code (exact match, recall@k, "did it call the tool with a valid argument"). Some are an LLM judge with a rubric. Some are a human.
Experiment: one run of the task over the dataset with a fixed version of the code, prompt and model, stored so it can be compared with the next run.

The dataset and the scorers are the part that takes real work. The platform's job is to make experiments cheap to run and easy to compare.

Offline evals for an agent

Offline evals run before a change ships, against a fixed dataset. For a chat agent I split them into three layers, cheapest first.

Routing evals

A creator-facing chat agent gets many kinds of requests: brainstorm video ideas for a channel, rewrite a title, explain why one title might work better than another, or something the agent should not attempt at all. A router decides which flow, tool or sub-agent handles each one. If routing is wrong, nothing downstream can fix it, so this is the first thing to test.

Routing evals are the easiest evals you will ever write. The dataset is a list of user messages, each labeled with the route it should take. The scorer is exact match. A case looks something like this (illustrative, not a real Spotter case):

{
  "input": "my last video on budget travel flopped, give me 5 better title options",
  "expected": { "route": "title_generation" },
  "metadata": { "source": "synthetic", "tricky": "about a past video, but asks for titles" }
}

Three things make these evals useful:

Collect cases from production, not from imagination. The messages that break a router are the ambiguous ones: a request for ideas that also asks for titles, or a follow-up that only makes sense given the previous turn. You find those in real conversations.
Look at the confusion matrix, not only the accuracy. One number going from 91% to 93% tells you little. Knowing that "title rewrite" keeps getting sent to "idea generation" tells you what to fix.
Run them on every change. The scoring is deterministic, but the router is a model call, so it still costs something and can vary between runs. Include the previous turns for follow-up cases, and repeat borderline cases a few times before making them a blocking check on pull requests.

Behavior evals

Once the request reaches the right place, the question becomes whether the agent behaved well: did the ideas fit the channel, were the titles in the requested style and length, did it ask a clarifying question when it should have, did it avoid promising things the product could not do.

Most of this needs an LLM judge. The practices that make a judge useful:

One rubric question per scorer. "Is every title under 70 characters" is plain code. "Does each idea fit the channel description" is a judge. A single judge asked to rate "overall quality" produces a number nobody can act on.
Pin and version the judge. The judge's prompt and model are part of the experiment. If they change, the scores are no longer comparable with last week's.
Calibrate the judge against people. Label a few dozen outputs yourself, run the judge on the same outputs, and read the disagreements. You will usually find the rubric is ambiguous, not that the judge is bad.
Prefer pairwise comparisons for subjective output. "Is version B's set of titles better than version A's for this channel" is easier to judge reliably than an absolute score out of 10.

Trajectory checks

For an agent that calls tools, the final answer is not the only thing that matters. Useful code scorers check the path: the tool that was called, whether its arguments were valid, how many steps it took, and whether it did anything it must never do. These are cheap and catch a surprising number of regressions after a prompt change.

What a retrieval eval taught me about eval hygiene

The nonprofit's retrieval eval was not an agent eval, but it taught me the most about keeping an eval honest. The existing setup was a test in the repo's test runner: 20 questions with hand-labeled correct answers, a corpus of a few thousand text posts and media items, and a GitHub Action that posted precision, recall and F1 to the pull request when a label was added.

It had a quiet bug in its design. The runner retrieved exactly as many items as each question expected, so precision, recall and F1 were always the same number. That is mathematically expected when k equals the number of relevant items, so nothing was broken in the code. But it meant the three metrics could not move independently, and the eval said nothing about what happens with a fixed retrieval budget. Before trusting any eval, check that each metric can actually disagree with the others.

What I changed, and would do again:

Two jobs, two tools. I kept the test in the test runner as the hard gate: below a set score, the pull request fails. Braintrust ran alongside it for everything the gate is bad at: side-by-side comparison of embedding models, per-question drill-down, and history across pull requests. A CI gate should be simple and boring. A place to explore results should be rich. Keeping them separate meant neither had to compromise.
Put the experiment's identity in its name and metadata. Each run was named from the provider, model and embedding dimension, with metadata for the corpus type and k. Comparing "model A at 1024 dimensions, text only" with "model B, images included" is then a filter, not an archaeology project.
Cache the expensive part. Embedding the whole corpus is slow and costs money. Caching the embeddings meant reruns only redid retrieval and scoring, which made iterating on the questions and scorers take seconds instead of minutes.
Drill into single cases. The per-question view showed that text posts were retrieved reasonably well while media items were found much less often. One likely cause was easy to spot once visible: media had been embedded from its text metadata only, not from the images. An average score would never have shown that.

The scorers went through a lesson of their own. I started with about fourteen: precision, recall and F1 for each item type, NDCG, MRR, and similarity diagnostics such as the average similarity of correct and incorrect hits and the gap between them. One of them could go negative, and Braintrust expects scores between 0 and 1, so I clamped it: a workaround that throws information away, so keep the raw value somewhere as a diagnostic. Within a day I cut the list to eight: recall and mean average precision for each item type, hit rate for media, the average similarity of correct hits for each item type, and one combined score. Fourteen numbers per run meant nobody looked at any of them. Eight, each tied to a decision, got read.

Online evals

Offline evals only cover the cases you thought to collect. Online evals score real production traffic, after the fact, and they are how the dataset keeps growing.

The pattern that works:

Run cheap code checks on all traffic and LLM judges on a sample. Formatting, length, refusals, tool errors and latency can be checked on every trace. How much to send to a judge depends on volume, how rare the failures you care about are, and which segments matter most. Sample more heavily where failures are rare or costly.
Score asynchronously. Online scoring must never add latency to the user's request.
Alert on the trend, not on single traces, except for severe failures such as leaked data or unsafe output, which deserve an alert every time. For everything else, a drop in the daily share of conversations the judge rates as on-topic is a signal; one bad trace is not.
Close the loop. The most valuable output of online evals is new offline test cases. When a production trace fails, add it to the dataset with the correct label, and the next change is tested against it.
Mind the data. Production traces contain user data. Check what the platform stores, for how long, and where, before turning on full capture, and redact before copying production traces into a dataset.

Braintrust vs LangSmith vs Datadog

I have used all three, and Braintrust is my favorite. Apart from that preference and the Braintrust details from my own projects, what follows comes from a fresh read of each vendor's documentation and pricing page in September 2026. These products ship quickly, so check anything that matters to you against the current docs.

Braintrust

Braintrust is the one I would pick again. Its core is the experiment loop: you write an Eval() with data, a task and scorers, run it, and every run becomes an experiment you can compare against a baseline, with a per-example diff and each case marked as an improvement, a regression or a tie. The per-question view in that comparison is where the media-retrieval gap above became visible.

Other things that matter in practice:

Scores are numbers between 0 and 1. That is a small constraint (I had to clamp a scorer that could go negative), but it keeps every scorer on the same chart scale.
Autoevals, an open-source library of ready-made scorers (factuality, similarity, JSON validity and others), plus LLM-judge and plain-code scorers.
An official GitHub Action that runs evals on a pull request and posts the results as a comment.
Online scoring through rules that run scorers asynchronously on a sample of production spans or whole traces.
Agent guidance in the docs that covers routing ("did it go to the correct model or sub-agent"), tool choice and tool arguments. It has no packaged trajectory matcher or simulated-user feature. You write those as scorers.
No per-seat pricing, which matters when you want product managers and designers in the tool reading results.

The weaknesses: the platform is closed source, self-hosting is Enterprise-only, and scores are metered. If you run judges on a lot of production traffic, the score count is the number to watch.

LangSmith

On paper, LangSmith has the most built-in tooling for agent-specific evaluation, and its own libraries do more of the work for you. agentevals gives you trajectory matching in four modes (strict, unordered, subset, superset), openevals can simulate a user for multi-turn tests, and the docs have a dedicated guide for single-step routing evals. Online evaluators can have a weekly spend cap, which is a thoughtful feature for LLM judges on live traffic, and alerts can go to Slack or PagerDuty.

It works without LangChain, through its own SDK or OpenTelemetry, and it integrates most deeply with LangChain and LangGraph apps. For CI it offers pytest and Vitest/Jest plugins rather than a packaged GitHub Action. Pricing is per seat plus per trace, and the overage rate sits behind a calculator rather than on the page. Having used both, I still prefer Braintrust.

Datadog LLM Observability

Datadog's product, which it is now also calling Agent Observability, has grown a full eval toolset: versioned datasets, experiments with side-by-side results, LLM judges at span, trace and session level, nine judge templates including tool selection and tool-argument correctness, annotation queues (since March 2026), and a prompt registry. Since June 2026 it can also run DeepEval and Pydantic Evals metrics unchanged.

Its real advantage is everything around the evals. LLM traces sit next to your APM traces, logs and real-user monitoring, and eval results can drive the same monitors your on-call rotation already uses. Its pricing is also the simplest of the three: you pay per LLM call span, and tool and retrieval spans are free. There is no separate evaluation fee, but the LLM calls a judge makes count as billable spans.

Where it is weaker: it started as an observability product, and most of its offline tooling (datasets, experiments, annotation queues) is newer than the other two platforms'. I found no packaged eval action for CI in the docs, so pass/fail thresholds in a pipeline need your own integration code around the SDK. It is SaaS only.

Feature comparison

	Braintrust	LangSmith	Datadog
Framework-agnostic SDKs	Python, TS, Go, Java, Ruby, C#	Python, TS	Python, Node, Java
OpenTelemetry ingest	Yes	Yes	Yes
Datasets from production traces	Yes	Yes	Yes
Experiment comparison	Per-example diff, regressions flagged	Diff mode for two runs	Side by side
Code and LLM-judge scorers	Yes, scores in [0, 1]	Yes, any range	Yes, typed outputs
Trajectory / tool-call evals	Custom scorers	`agentevals`, four match modes	Tool selection and argument templates
Simulated users	Not built in	Yes (`openevals`)	Not built in
Online evals with sampling	Yes	Yes, with spend caps	Yes
CI	Official GitHub Action	pytest / Vitest plugins	Your own code around the SDK
Human review queues	Yes	Yes, including pairwise	Yes
Self-hosting	Enterprise only	Enterprise add-on	No

Pricing, as of September 2026

	Free tier	Named paid plan	What you pay for
Braintrust	1 GB of data, 10k scores a month, 14-day retention, unlimited users	Pro, $249 a month: 5 GB, 50k scores, 30-day retention	Data and scores (the free tier can also pay for overage), no seat charge
LangSmith	1 seat, 5k traces a month, 14-day retention	Plus, $39 per seat a month: 10k traces	Seats and traces; overage rate not published as text
Datadog	40k LLM spans a month, 15-day retention	$160 a month for up to 100k LLM spans	LLM call spans only, including the judge's own calls

One pricing detail to watch in LangSmith: experiments and scored traces default to extended retention, which costs more than base traces unless you turn it off. The free tiers are all large enough to find out whether evals change how your team works, which is the only question that matters in the first month.

How I rank them

Criteria first, because the ranking depends on them. I am ranking for a small team shipping an agent, where the scarce resource is engineers' attention and the goal is to know whether a change made the agent better or worse before it ships.

Braintrust. In my experience, the shortest path from "I changed a prompt" to "here are the cases that got worse." Official CI support, no seat pricing, and the experiment comparison I liked working in most.
LangSmith. On paper, the most built-in tooling for agent-specific evals: trajectories, simulated users, routing guides. Choose it first if you are on LangGraph, or if multi-turn simulation is central to what you need to test.
Datadog. The right answer if your company already runs on Datadog and wants LLM quality in the same place as latency, errors and on-call. Per-LLM-call pricing with no separate eval fee makes the bill easy to predict.

Two self-hostable options are worth knowing about even though I have not used them in production. Langfuse is MIT-licensed at its core, can be self-hosted for free, and has an official GitHub Action for experiments. It is a strong option when traces must stay on your infrastructure, as long as the judge model runs there too. Arize Phoenix is free to self-host under the Elastic License and has good tool-call and trajectory metrics. You can score production traces with its SDK, but managed continuous online evals and alerting come with Arize's commercial product.

What I would set up first

Whichever platform you pick, the order matters more than the tool:

Fifty routing cases taken from real conversations (redacted), scored by exact match, run in CI.
Three to five behavior scorers, each answering one question, with the judge prompt versioned and checked against a few dozen human labels.
Code checks on every production trace, and one judge on a small sample, scored asynchronously.
A weekly habit of turning failed production traces into new test cases.

The platform makes steps 1 to 4 faster. It cannot choose the cases for you, and the cases are where the quality comes from.

Spring AI Evals: how I test agent behavior

Wed, 23 Sep 2026 21:55:02 +0000

Part 2: https://dev.to/lbobylev/baseline-for-spring-ai-evals-catching-silent-degradation-4ll6

When building an AI agent, there is usually a moment when the prompt seems to work. But as development continues, this can quickly get out of control. Today the agent can call the right tool. Tomorrow, after a small system prompt change, it can stop calling it. Later, it can start calling it when it should not. So I treat evals as regular regression tests for AI behavior.

Here is a small example: a Spring AI agent must create a Markdown note through the create_note tool. The task looks simple, but there are already several ways the agent can fail:

not call the tool when needed, or call it when it is not needed;
create an extra file;
produce a note in the wrong format;
add extra advice or any irrelevant content.

So I split the checks into a few layers.

Success Criteria

First I define what correct behavior means. The agent should:

call create_note when the user asks to create a new note;
not call create_note when the user asks to read, delete, rename, or explain;
create only note.txt;
put a heading on the first line;
write short, relevant, and readable content.

This matters: without clear success criteria, an eval becomes a subjective check.

Tool Definition

The tool's scope is intentionally narrow: creating the note.txt file.

@Tool(
    name = "create_note",
    description = "Create a new Markdown note in note.txt. Use this only when the user asks to create a new note."
)
public String createNote(String title, String body) {
    var notePath = this.notesDirectory.resolve("note.txt");

    var content = "# " + normalize(title, "New Note")
            + System.lineSeparator()
            + System.lineSeparator()
            + normalize(body, "No content provided.")
            + System.lineSeparator();

    try {
        Files.createDirectories(this.notesDirectory);
        Files.writeString(notePath, content, CREATE, TRUNCATE_EXISTING);
        return "Created note.txt.";
    }
    catch (IOException ex) {
        return "Failed to create note.txt: " + ex.getMessage();
    }
}

The model uses the name and description input parameters to determine whether it needs to call the tool.

Agent Instructions

The process goals are formulated in the system prompt, which establishes the contract:

when to use the tool;
which file must appear;
what must not appear in the result.

return this.chatClient.prompt()
        .system("""
            When the user asks to create a new note, use the create_note tool.
            The tool must create note.txt, put a Markdown heading on the first line,
            add the body text, and create no extra files.
            Keep the note concise and do not add unrelated suggestions.
            """)
        .user(message)
        .call()
        .content();

Deterministic Checks

The first layer checks tool invocation. This is no different from a normal unit test.

@ParameterizedTest
@CsvSource(value = {
    "note-create-01|true|Create a new note about my plans for the week",
    "note-create-02|true|Make a markdown note with the title Retro and the body: what to improve",
    "note-no-trigger-01|false|Show me the existing notes",
    "note-no-trigger-02|false|Read the note.txt file",
    "note-no-trigger-03|false|Delete the note.txt file",
    "note-no-trigger-04|false|Draft the text of a note, but do not create a file"
}, delimiter = '|')
void evaluatesWhetherAgentCallsNoteTool(String id, boolean shouldTrigger, String prompt) {
    doReturn("Created note.txt.").when(this.noteTool).createNote(anyString(), anyString());
    clearInvocations(this.noteTool);

    var response = this.chatController.chat(prompt);

    assertThat(response).as(id).isNotBlank();

    verify(this.noteTool, shouldTrigger ? atLeastOnce() : never()).createNote(anyString(), anyString());
}

Tool Unit Test

Next, I test the tool itself without the model. This is also a normal unit test.

@Test
void createNoteWritesOnlyNoteFileWithMarkdownHeading() throws IOException {
    var noteTool = new NoteTool(this.tempDir);

    var result = noteTool.createNote("Meeting Notes", "Discuss the evaluation setup.");

    var notePath = this.tempDir.resolve("note.txt");
    var content = Files.readString(notePath);
    var lines = Files.readAllLines(notePath);

    try (var fileStream = Files.list(this.tempDir)) {
        var files = fileStream.toList();

        assertThat(result).isEqualTo("Created note.txt.");
        assertThat(notePath).exists();
        assertThat(content).isNotBlank();
        assertThat(lines.getFirst()).startsWith("#");
        assertThat(files).containsExactly(notePath);
    }
}

This separates responsibilities: if the unit test fails, the problem is in the tool implementation rather than the agent's behavior.

Rubric-Based Assessment

Deterministic checks are good at catching facts, but not everything can be checked this way. For example, "the note is relevant," "there is no extra information," and "the text is easy to read" are qualitative requirements. Rubric-based assessment is used for these. In Spring AI, this is conveniently implemented with Evaluator: it receives an EvaluationRequest, runs a grader prompt inside, and returns an EvaluationResponse with pass, score, feedback, and metadata. The grader model is fixed, ideally using a versioned model name. For example, gpt-5.5-2026-04-23 instead of gpt-5.5. If the provider updates a model alias, the grader can become stricter or more lenient without any code change. In that case, a score change may be caused by grader drift rather than agent behavior. A low temperature is also preferable for grading. The grader should not be creative; it should apply the same rubric consistently. For rubric-based assessment, the grader runs multiple times. The pass rate better reflects its behavior.

class NoteStyleEvaluator implements Evaluator {

    private static final List<String> REQUIRED_CHECK_IDS =
            List.of("title", "relevance", "simplicity", "readability");

    private final ChatClient chatClient;

    private final int minimumScore;

    NoteStyleEvaluator(ChatClient.Builder chatClientBuilder, int minimumScore) {
        this.chatClient = chatClientBuilder.build();
        this.minimumScore = minimumScore;
    }

    @Override
    public EvaluationResponse evaluate(EvaluationRequest evaluationRequest) {
        var rubricResult = gradeNote(evaluationRequest);

        if (rubricResult == null) {
            return new EvaluationResponse(false, 0.0f, "No rubric result", Map.of());
        }

        var pass = rubricResult.overallPass()
                && rubricResult.score() >= this.minimumScore
                && hasRequiredPassingChecks(rubricResult.checks());

        var metadata = Map.<String, Object>of(
                "overallPass", rubricResult.overallPass(),
                "checks", rubricResult.checks());

        return new EvaluationResponse(pass, rubricResult.score(), rubricResult.toString(), metadata);
    }

    private StyleRubricResult gradeNote(EvaluationRequest evaluationRequest) {
        var graderPrompt = """
            Read the original user request and the note.txt content,
            then grade the note against these requirements:

            - it has a clear title;
            - the content is relevant to the original user request;
            - there is no clearly unnecessary information;
            - the text is easy to read.

            Return only a structured JSON object with these fields:
            - overall_pass: boolean
            - score: integer from 0 to 100
            - checks: exactly four items with ids title, relevance, simplicity, readability.

            Each check must include id, pass, and notes.

            original user request:
            %s

            note.txt content:
            %s
            """.formatted(
                evaluationRequest.getUserText(),
                evaluationRequest.getResponseContent()
            );

        return this.chatClient.prompt()
                .user(graderPrompt)
                .call()
                .entity(StyleRubricResult.class);
    }

    private boolean hasRequiredPassingChecks(List<StyleRubricCheck> checks) {
        if (checks == null || checks.size() != REQUIRED_CHECK_IDS.size()) {
            return false;
        }

        var checkIds = checks.stream()
                .map(StyleRubricCheck::id)
                .toList();

        return checkIds.containsAll(REQUIRED_CHECK_IDS)
                && checks.stream().allMatch(check ->
                        check.pass() && check.notes() != null && !check.notes().isBlank());
    }

    record StyleRubricResult(
            @JsonProperty("overall_pass") boolean overallPass,
            int score,
            List<StyleRubricCheck> checks) {
    }

    record StyleRubricCheck(
            String id,
            boolean pass,
            String notes) {
    }
}

Structured output enables automated verification. With Evaluation in Spring AI, grading becomes a normal Java contract:

EvaluationRequest stores the user prompt and generated output;
Evaluator contains the grading logic;
EvaluationResponse returns a machine-checkable result;
metadata stores details for each rubric check.

This makes grading convenient to integrate into CI: there is pass, score, feedback, and a list of checks.

@ParameterizedTest
@CsvSource(value = {
    "weekly-plan-01|Create a new note about my plans for the week. Include a clear title and only a concise list of concrete weekly goals.|80",
    "weekly-plan-02|Create a markdown note for my weekly planning. Mention only work, health, and learning goals. Do not add extra categories.|80"
}, delimiter = '|')
void gradesCreatedNoteWithStyleRubric(String id, String prompt, int minimumScore) throws IOException {
    var agentResponse = this.chatController.chat(prompt);

    assertThat(agentResponse).as(id).isNotBlank();
    assertThat(this.notePath).as(id).exists();

    var noteContent = Files.readString(this.notePath);

    var evaluator = new NoteStyleEvaluator(this.chatClientBuilder, minimumScore);

    int graderRuns = 3;
    int requiredPassingRuns = 2;

    var results = IntStream.range(0, graderRuns)
            .mapToObj(run -> evaluator.evaluate(new EvaluationRequest(prompt, noteContent)))
            .toList();

    long passingRuns = results.stream()
            .filter(EvaluationResponse::isPass)
            .count();

    assertThat(passingRuns)
            .as(id + ": " + results)
            .isGreaterThanOrEqualTo(requiredPassingRuns);
}

The agent runs once and creates one note. The grader then evaluates the same note.txt content three times; the test passes if at least two evaluations pass. This approach makes the grader result more robust against random model variation.

Saving Results Between Versions

It is also advisable to save eval results between runs. This helps compare behavior after changes to the prompt, model, or rubric. Pass/fail status is not always enough. A test can still pass while the score gradually gets worse. That is why I keep the score, feedback, and rubric metadata as artifacts.

var resultPath = Path.of("build", "eval-results", id + ".json");

var resultJson = objectMapper.writeValueAsString(Map.of(
        "id", id,
        "prompt", prompt,
        "minimumScore", minimumScore,
        "graderRuns", graderRuns,
        "requiredPassingRuns", requiredPassingRuns,
        "passingRuns", passingRuns,
        "results", results
));

Files.createDirectories(resultPath.getParent());
Files.writeString(resultPath, resultJson);

Separating Evals from Regular Tests

It is a good idea to keep AI evals separate from regular tests. They call the model API, so they are slower, more expensive, and less deterministic.

tasks.named('test') {
    useJUnitPlatform {
        excludeTags 'eval'
    }
}

tasks.register('evalTest', Test) {
    description = 'Runs AI agent evaluation tests.'
    group = 'verification'

    useJUnitPlatform {
        includeTags 'eval'
    }

    shouldRunAfter test
}

Run them separately:

./gradlew test
./gradlew evalTest

What This Gives

This makes development more predictable and manageable. When the system prompt, the tool description or implementation, or model parameters change, evaluating agent behavior remains much less subjective. CI tracks:

whether the number of false positives increased;
whether false negatives appeared;
whether the expected artifact was created;
whether the output passes the rubric assessment;
whether quality dropped below the minimum score.

The most useful part is that the eval suite can grow gradually. Found a new miss? Add a new eval case. The agent failed on a real user prompt? That prompt becomes a regression test. This way, behavior drift becomes visible instead of staying hidden in manual testing.

TDD in the Age of AI: How Evals Can Benefit from Test-Driven Development

Mon, 09 Feb 2026 04:22:29 +0000

Test-Driven Development (TDD) has been a cornerstone of quality software engineering for decades. But as we enter the age of AI-assisted development and large language models, does TDD still hold relevance? The answer is a resounding yes—and perhaps more than ever before.

The Timeless Principles of TDD

At its core, TDD follows a simple cycle: write a failing test, write the minimum code to pass, then refactor. This red-green-refactor loop forces developers to think about requirements and edge cases before implementation. The result? Cleaner code, better design, and a comprehensive test suite that serves as living documentation.

TDD Meets AI: A Natural Partnership

When working with AI coding assistants, TDD becomes even more valuable. Here’s why:

Clear specifications: Tests provide unambiguous requirements that AI can understand and implement against
Verification layer: Tests act as a safety net to validate AI-generated code behaves correctly
Reduced hallucinations: When AI knows it must pass specific tests, outputs tend to be more grounded and practical
Iterative refinement: The red-green-refactor cycle works naturally with AI iteration

Evals: TDD for AI Systems

Perhaps the most exciting application of TDD principles is in the world of AI evaluations—commonly called “evals.” Evals are systematic tests designed to measure how well an AI model performs on specific tasks or criteria.

The parallel to TDD is striking:

TDD Concept	Eval Equivalent
Write failing test first	Define expected behavior before training/prompting
Minimum code to pass	Iterate on prompts or fine-tuning until eval passes
Refactor	Optimize prompts while maintaining eval scores
Test suite	Eval suite covering multiple capabilities

Applying TDD Mindset to Evals

Here’s how you can apply TDD thinking to your AI evaluation strategy:

1. Define Success Criteria First

Before crafting prompts or fine-tuning models, clearly define what success looks like. What inputs should produce what outputs? What edge cases matter? This is your “test specification.”

2. Create Measurable Evals

Write evals that can objectively determine pass/fail. This might include exact match comparisons, semantic similarity thresholds, or structured output validation. Avoid vague criteria that require subjective judgment.

3. Start with Failing Evals

Just like TDD, begin with evals that your current system fails. This ensures you’re building toward measurable improvement rather than just validating the status quo.

4. Iterate Incrementally

Make small changes to prompts or model configurations, run your eval suite, and observe the impact. Large changes make it difficult to understand what’s working.

5. Prevent Regression

Maintain a comprehensive eval suite that runs with every change. Just like unit tests catch regressions in code, evals catch regressions in AI behavior.

Practical Benefits

Teams adopting eval-driven development report several benefits:

Faster iteration: Clear pass/fail criteria eliminate guesswork
Better communication: Evals serve as a shared language between engineers, product managers, and stakeholders
Confidence in deployment: A passing eval suite provides assurance before shipping
Documentation: Evals document expected behavior in an executable form

Conclusion

TDD isn’t just surviving the AI age—it’s thriving. The discipline of writing tests first translates beautifully to the world of AI evals, where defining expected behavior upfront is crucial for building reliable systems.

Whether you’re writing traditional software, working with AI coding assistants, or building AI-powered products, the TDD mindset remains invaluable: specify behavior first, verify it works, then iterate with confidence.

The age of AI doesn’t diminish the importance of testing—it amplifies it.

The post TDD in the Age of AI: How Evals Can Benefit from Test-Driven Development appeared first on sudoish.