# Evaluating Pydantic AI agents that call MCP tools

Sat, 08 Aug 2026 13:18:00 +0000

A Pydantic AI agent returns BookingResult(confirmed=True, flight_id='AA127', total_usd=412.50). Every Pydantic check passes. The output_type matches. The tool-call arguments validated against the Pydantic model on the way in. The retry counter is at zero. The user asked to book the Sunday flight, and AA127 is a Friday flight. The agent confirmed the wrong booking with full type safety.

That is the gap I want to talk about. Pydantic AI's typed contract gives you free schema validation. It does not give you free quality evaluation. The shape is guaranteed, the retry-on-validation-failure loop is wired, and the output_type parses. That is the floor. The ceiling is whether the agent picked the right tool, whether the arguments were semantically correct, whether the typed output reflects the user's goal, whether the dependency invariants held, and whether an MCP tool description carried an injection payload that bypassed every type check. Four layers cover that gap, and there is a thorough walkthrough of the full methodology with the rubric code and the instrumentation if you want to lift it directly; this is the shorter argument for each layer.

Why the typed contract is the floor, not the ceiling

Pydantic AI is the agent framework from the Pydantic team. You declare an agent with a model, an output_type (a Pydantic model), a typed deps_type for runtime dependencies, and tools whose arguments are Pydantic models. The framework dispatches the model call, parses the tool selection, validates arguments against the schema, calls the tool, and re-prompts on a validation failure. It also has first-class MCP client support.

That contract handles four bug classes for free: tool-argument shape mismatch, output-schema mismatch, a non-existent tool name, and deps_type enforcement at call time. Every one of those is a question about syntax. The questions that break in production are about meaning, and the two do not overlap:

Failure mode	Caught by the type contract	Needs evaluation
Tool argument shape mismatch	Yes	No
Output shape mismatch	Yes	No
Wrong tool for the user intent	No	Yes
Right tool, semantically wrong arguments	No	Yes
Right tool, wrong MCP server	No	Yes
Typed values do not reflect the user goal	No	Yes
Trajectory loops, stalls, or stops short	No	Yes
Refusal drifts under retry pressure	No	Yes
MCP tool description carries an injection payload	No	Runtime guardrail
MCP result tampers with the next planning turn	No	Runtime guardrail
A planned call violates a dependency invariant	No	Yes

Schema validation is one assertion per field. The evaluation that matters is the assertion across fields, across calls, across servers, and across the dependencies that anchor the run.

Layer 1: score every typed output field on its own

Structured output is the part most teams over-trust. A BookingResult(confirmed: bool, flight_id: str, total_usd: float) constrains the shape. It says nothing about whether confirmed is correctly True, whether flight_id matches the flight the user asked about, or whether total_usd is plausible for the route.

The rule I use: any field whose correctness depends on the input gets its own judge. Pure-structure fields, a UUID or a fixed-format timestamp, do not. One judge per non-trivial field, each rubric narrow enough that the score is unambiguous. A flight_id judge scores whether the flight matches the user's stated day, route, and time window, 5 for a match and 1 for a different day or route. Add the same for confirmed (did the booking actually succeed in the trace) and total_usd (is the value plausible for the route).

Per-field judges produce per-field signal, which is the thing that names the fix. A single "is this object correct" judge produces an average you cannot act on. Gate CI per axis (FlightIdFidelity >= 4.5, ConfirmationFidelity == 5) and fail the build on the axis that broke.

Layer 2: check tool-call argument fidelity

The type contract confirms that book_flight(flight_id='AA127', date='2026-08-15') parses against the tool's argument model. It says nothing about whether AA127 is the flight the user asked about or whether the date matches what they said. Argument fidelity sits between "the arguments parsed" and "the agent did what I asked."

Run a function-calling rubric on every tool-call span, scoring the tool name and arguments against the user prompt. Add three more:

Task completion scores the end state against the user goal across the whole trajectory, on the parent run span.
Refusal calibration catches both over-refusal and under-refusal, which matters because the retry-on-validation-failure loop can push the agent into a refusal posture when validation keeps failing on edge cases.
Tool-server attribution carries the case the others miss. When two MCP servers expose similarly named tools, a docs search and a CRM search, the agent can call the wrong one and still produce a plausible answer.

Gate per run: function-calling above 0.90, task completion above 0.85, refusal calibrated, server attribution passing. The build fails on the axis that broke.

Layer 3: treat the MCP integration as a security surface

Three properties of MCP change the threat model. The tool catalog is part of the prompt, since tools/list returns each tool's name, description, and schema straight into the model's context. The supply chain is npm install or pip install. And the trust boundary moves per session, because the active toolset is the union of every registered server at run time.

Four checks run at the gateway and one runs in CI. At the completion boundary, scan every tool definition before it enters the model's context, so a poisoned description is rejected at registration. At the per-call hook, scan arguments before dispatch and results before they enter the next planning turn, so an injection hidden in a read_file return never reaches the model. Score sandbox and permission-escape attempts against a fixed payload catalog and the tool's declared scope. Get cross-tenant isolation from per-key allowed and denied tool lists plus trace-ID audit propagation. The CI piece is a 200 to 500 case regression suite of attack tool definitions and tampered results, run on every PR that touches MCP policy.

Layer 4: verify the agent's dependency invariants

The deps_type is the most underused contract surface. You declare a deps type, the agent injects it into every tool through the run context, and the framework enforces the type at parse. What it cannot enforce is whether the model's plan respects the invariants those deps imply. A support agent with deps=AccountContext(tenant_id='acme', role='viewer') can plan a delete_record call. Every type check passes. The action is a policy violation that should never have been planned.

The fix is a deterministic invariant assertion per tool call: a pre-call check that takes the tool name, the arguments, and the run context, and returns allow or deny against the deps. If a viewer role reaches for a write tool, raise a structured error. Surface the verdict as a span attribute. The framework's retry loop treats the error as a structured tool failure, so the model sees the message, plans an alternative, and the run proceeds without escalating to a wrong action. Gate CI on zero tool-call spans where a deps-invariant violation flipped to an error on the golden set, and enforce the same rule at the gateway in production. Two layers, one invariant, no silent policy drift.

Running the four layers in production

Run the gates cheapest-first. The output_type parse and deps_type enforcement run inside the framework at no token cost. The deps-invariant assertion is the next layer, still pure Python. The MCP runtime scanners are single-digit to tens of milliseconds. The judge-based rubrics (function calling, task completion, per-field judges) run last, only on the rows the deterministic gates let through.

Build the golden set from production retries. Take around 300 prompts that produced two or more attempts in the last 30 days of live traffic; the retry count is the natural filter, because the prompts that retry are the ones closest to the framework's limits. Score offline and live with the same rubrics: offline over the golden set with per-axis CI thresholds, live by applying the same templates to sampled production spans, so a regression that escapes CI is caught within hours.

Watch retries as a cost signal. The retry default is good for uptime and bad for the bill. A prompt whose mean retry count drifts from 1.1 to 2.4 has doubled in cost without changing in correctness. Chart the distribution per output type and alert on shifts.

The anti-patterns I keep seeing

Schema-pass-rate as the metric. Parse success is a floor. Treating it as the ceiling ships wrong values confidently, because every type check is green. Add at least one per-field judge per output type.
Monolithic "is this object correct" judges. A single score over the whole object is uninterpretable. Did flight_id break or confirmed? Per-field judges produce per-field signal.
No deps-invariant assertions. The type contract enforces the type, not the policy. Every write tool reachable from a viewer-role run is a silent escalation path until the assertion is in place.
Trusting MCP descriptions blindly. A registered MCP tool is not a vetted one. The description scan and the result scan are not optional once the agent consumes any third-party server.

Instrument the agent so the run, tool-call, and model-request spans show up with the tool arguments and the output type populated. You cannot evaluate what you cannot see.
Add one judge per non-trivial field of your output type, score it over a 50-row golden set, and wire per-axis CI thresholds.
Add deterministic role checks on every write tool. Surface the verdict on the span and fail CI on golden-set runs that planned a denied call.
Front the agent with an MCP scanner that checks tool definitions and results, and set allowed servers and per-key tool lists from day one.
Cluster the first week of production failures, promote representative rows into the regression set, and re-run.

The teams shipping reliable Pydantic AI MCP agents stopped reporting schema-pass-rate and started reporting per-field judge scores, argument fidelity, MCP scanner verdicts, and deps-invariant pass rates. The type contract gives you the type-safe agent. The evaluation stack gives you the correct one.

Coder Legion - Recent posts tagged agentic-evaluation