How to Design Tool Errors AI Agents Can Recover From

Thu, 20 Aug 2026 14:34:07 +0000

“Something went wrong” is not an error contract. It is a request for another blind guess.

That is expensive inside an AI agent loop. The controller has to decide whether to retry the same call, repair an argument, ask for authorization, choose another tool, or stop. A human can infer some of that from a stack trace. A machine needs the next action to be explicit.

Human strategy, benchmark design, interpretation, and final editorial judgment led this article; AI assisted with research organization, drafting, and code review.

I built a deterministic benchmark around 12,000 synthetic tool operations to test three response formats: a generic message, an error code, and an actionable error envelope. The test does not measure model intelligence. It measures a smaller systems question: how much useful recovery can a controller perform when the error contract tells it what kind of failure occurred?

Original architecture visual. The raw message remains useful for people, while structured fields make the next machine action explicit.

A tool error is part of the agent's control plane

In a normal application, an error often ends a request. In an agent, the error becomes input to the next turn.

That changes the design requirement. The message should still help an operator, but it also needs to support a routing decision:

Retry: the same operation may succeed after backoff.
Repair: the operation can succeed after changing an argument.
Authorize: the call needs a credential or human approval.
Replace: another tool or resource should be used.
Stop: another attempt would waste time or repeat a side effect.

The Model Context Protocol makes a related distinction between protocol errors and tool execution errors. Malformed requests and unknown tools belong at the protocol layer. Validation failures, API failures, and business-logic errors can be returned as tool results with isError: true, giving the model actionable feedback it can use to adjust a later call (MCP tool error handling).

OpenAI's Agents SDK exposes the same underlying control problem from the runner side. An unresolved tool name can either raise an exception or be returned to the model so the loop can recover and select an available tool (OpenAI Agents SDK runner documentation).

The protocol can carry an error. The application still has to decide whether that error is useful.

I tested three error formats across 12,000 operations

The benchmark generated 60 seeded runs with 200 initial operations in each run. Forty percent of operations failed under an explicit scenario assumption. Failures were divided evenly among four types:

transient;
validation;
permission;
not found.

Each strategy received the same generated operations and the same recovery rolls.

The three strategies were:

Generic message: retry every failure once because the controller cannot classify it.
Error code only: retry transient failures, attempt to repair validation failures, and stop on permanent failures.
Actionable envelope: use the same routing, but include a field and repair suggestion that improves the modeled validation-repair probability.

Run the benchmark with:

npx tsx scripts/run-agent-tool-error-contract-benchmark.ts

The scenario assumptions are part of the result, not hidden setup. A transient retry succeeds 70% of the time. A validation repair succeeds 45% of the time when the controller receives only a code and 90% when it receives the failing field and a repair hint.

Those percentages are illustrative assumptions. They are not measurements of a production model.

The actionable envelope cut failed retries by 87.7%

Here are the aggregate results:

Strategy	Resolution rate	Attempts per operation	Failed retry attempts	Permanent-failure retries
Generic message	67.1%	1.402	3,953	2,449
Error code only	71.2%	1.198	1,005	0
Actionable envelope	75.5%	1.198	487	0

Original deterministic benchmark. Sixty seeded runs contained 200 initial operations each. Resolution is simulated controller success within one recovery attempt; no language model was called.

The generic strategy made 16,823 total attempts for 12,000 operations. It retried permission and not-found failures 2,449 times even though the retry could not change the outcome.

Adding a code eliminated those permanent-failure retries. Total attempts fell to 14,374, while the resolution rate increased from 67.1% to 71.2% because the controller could treat validation as a repair problem instead of repeating the same invalid call.

The actionable envelope used the same number of attempts as the code-only strategy but resolved 75.5% of operations. Its failed retry count fell to 487—87.7% below the generic-message strategy.

The benchmark does not prove that every model will recover at that rate. It shows what the controller can do when recovery policy is encoded rather than guessed.

Give the controller a small, stable error envelope

The error contract does not need to become a second API specification. It needs enough information to route the failure safely.

type ToolError = {
  code: "TRANSIENT" | "VALIDATION" | "PERMISSION" | "NOT_FOUND";
  message: string;
  retryable: boolean;
  field?: string;
  suggestion?: string;
  retryAfterMs?: number;
  traceId: string;
};

Each field has one job.

code selects a known failure class.
message explains the problem to the model and operator.
retryable stops the controller from inventing its own retry rule.
field identifies the argument that needs repair.
suggestion supplies a bounded correction rather than an open-ended rewrite.
retryAfterMs prevents an immediate retry from amplifying throttling.
traceId connects the agent-visible failure to logs and traces.

Do not put secrets, raw database errors, or internal stack traces into a model-visible message. The public contract should expose the recovery surface, not the server's internals.

Separate routing policy from error prose

A controller can route the envelope without asking a model to interpret every sentence.

function nextAction(error: ToolError): "retry" | "repair" | "authorize" | "stop" {
  if (error.code === "TRANSIENT" && error.retryable) return "retry";
  if (error.code === "VALIDATION" && error.field && error.suggestion) return "repair";
  if (error.code === "PERMISSION") return "authorize";
  return "stop";
}

This is intentionally boring. A deterministic router is easier to test than a prompt that says, “Use your judgment and try again if appropriate.”

The model still matters when a validation error requires semantic repair. It may need to convert “next Friday” into an ISO-8601 date or select a replacement resource. But the controller should decide whether a repair turn is allowed before spending another model call.

GitHub's MCP Scripts specification takes this idea further for execution errors by requiring a data.recoverable boolean. It limits recoverable: true to transient failures where the same invocation may succeed on a later attempt (GitHub MCP Scripts error handling). That is the kind of field a controller can enforce without semantic guesswork.

The prompt trace should preserve the boundary

Here is the exact interpretation constraint used for the benchmark summary:

Prompt:
Compare the three tool-error strategies using only summary.json. Separate
measured simulation output from scenario assumptions. Do not claim LLM accuracy.

Input:
Generic message: 67.1% resolved, 3,953 failed retry attempts
Error code only: 71.2% resolved, 1,005 failed retry attempts
Actionable envelope: 75.5% resolved, 487 failed retry attempts

Output constraint:
Report resolution rate, attempts per operation, failed retries, and permanent-
failure retries. State that all recovery probabilities are explicit synthetic
assumptions and that no model was called.

That last line prevents a common reporting error. A controller simulation can test policy logic. It cannot tell you how reliably a model will interpret a repair hint.

Retry safety needs more than a boolean

retryable: true is necessary and insufficient.

A production retry path should also answer:

Is the operation idempotent?
Has this workflow already produced a side effect?
What is the maximum attempt budget?
Should the controller use exponential backoff and jitter?
Did the server provide a rate-limit reset time?
Does the repaired call need a new idempotency key or the old one?
Which failures require a human decision?

A retryable payment lookup and a retryable payment creation are different risks. The first may waste capacity. The second may charge twice.

For state-changing tools, store the operation ID and side-effect status outside the model context. A model should not be the source of truth for whether a transaction already happened.

Where this benchmark stops

The benchmark is deterministic and reproducible, but its scope is narrow.

It does not measure latency, tokens, model interpretation, parallel tool calls, ambiguous repair hints, or side-effect safety. The failure mix is synthetic. The recovery probabilities are declared assumptions. Resolution means success within one modeled recovery attempt, not completion of a full agent task.

Real evaluations should replay production-shaped traces, vary models and prompts, measure incorrect repairs, and test that terminal failures really stop. They should also include adversarial errors: misleading suggestions, mismatched codes, missing trace IDs, and a server that marks everything retryable.

The small benchmark is still useful because it tests the first layer without model noise: whether the controller has enough structure to choose a permitted next action.

Make failure actionable before making it eloquent

Start with four routes: retry, repair, authorize, and stop. Define the codes that lead to each route. Add an explicit retry flag, the smallest useful repair hint, and a trace ID. Then test the controller with fixed seeds before asking a model to interpret the message.

The practical test is simple: if the model were unavailable, could the runtime still avoid repeating a permission error, respect backoff, and identify the argument that needs repair?

If not, the error contract is still prose. Give it enough structure to become control flow.

Coder Legion - Recent posts tagged model-context-protocol