Coder Legion - Recent posts tagged tool-calling

How to Design Tool Errors AI Agents Can Recover From

Thu, 20 Aug 2026 14:34:07 +0000

“Something went wrong” is not an error contract. It is a request for another blind guess.

That is expensive inside an AI agent loop. The controller has to decide whether to retry the same call, repair an argument, ask for authorization, choose another tool, or stop. A human can infer some of that from a stack trace. A machine needs the next action to be explicit.

Human strategy, benchmark design, interpretation, and final editorial judgment led this article; AI assisted with research organization, drafting, and code review.

I built a deterministic benchmark around 12,000 synthetic tool operations to test three response formats: a generic message, an error code, and an actionable error envelope. The test does not measure model intelligence. It measures a smaller systems question: how much useful recovery can a controller perform when the error contract tells it what kind of failure occurred?

Original architecture visual. The raw message remains useful for people, while structured fields make the next machine action explicit.

A tool error is part of the agent's control plane

In a normal application, an error often ends a request. In an agent, the error becomes input to the next turn.

That changes the design requirement. The message should still help an operator, but it also needs to support a routing decision:

Retry: the same operation may succeed after backoff.
Repair: the operation can succeed after changing an argument.
Authorize: the call needs a credential or human approval.
Replace: another tool or resource should be used.
Stop: another attempt would waste time or repeat a side effect.

The Model Context Protocol makes a related distinction between protocol errors and tool execution errors. Malformed requests and unknown tools belong at the protocol layer. Validation failures, API failures, and business-logic errors can be returned as tool results with isError: true, giving the model actionable feedback it can use to adjust a later call (MCP tool error handling).

OpenAI's Agents SDK exposes the same underlying control problem from the runner side. An unresolved tool name can either raise an exception or be returned to the model so the loop can recover and select an available tool (OpenAI Agents SDK runner documentation).

The protocol can carry an error. The application still has to decide whether that error is useful.

I tested three error formats across 12,000 operations

The benchmark generated 60 seeded runs with 200 initial operations in each run. Forty percent of operations failed under an explicit scenario assumption. Failures were divided evenly among four types:

transient;
validation;
permission;
not found.

Each strategy received the same generated operations and the same recovery rolls.

The three strategies were:

Generic message: retry every failure once because the controller cannot classify it.
Error code only: retry transient failures, attempt to repair validation failures, and stop on permanent failures.
Actionable envelope: use the same routing, but include a field and repair suggestion that improves the modeled validation-repair probability.

Run the benchmark with:

npx tsx scripts/run-agent-tool-error-contract-benchmark.ts

The scenario assumptions are part of the result, not hidden setup. A transient retry succeeds 70% of the time. A validation repair succeeds 45% of the time when the controller receives only a code and 90% when it receives the failing field and a repair hint.

Those percentages are illustrative assumptions. They are not measurements of a production model.

The actionable envelope cut failed retries by 87.7%

Here are the aggregate results:

Strategy	Resolution rate	Attempts per operation	Failed retry attempts	Permanent-failure retries
Generic message	67.1%	1.402	3,953	2,449
Error code only	71.2%	1.198	1,005	0
Actionable envelope	75.5%	1.198	487	0

Original deterministic benchmark. Sixty seeded runs contained 200 initial operations each. Resolution is simulated controller success within one recovery attempt; no language model was called.

The generic strategy made 16,823 total attempts for 12,000 operations. It retried permission and not-found failures 2,449 times even though the retry could not change the outcome.

Adding a code eliminated those permanent-failure retries. Total attempts fell to 14,374, while the resolution rate increased from 67.1% to 71.2% because the controller could treat validation as a repair problem instead of repeating the same invalid call.

The actionable envelope used the same number of attempts as the code-only strategy but resolved 75.5% of operations. Its failed retry count fell to 487—87.7% below the generic-message strategy.

The benchmark does not prove that every model will recover at that rate. It shows what the controller can do when recovery policy is encoded rather than guessed.

Give the controller a small, stable error envelope

The error contract does not need to become a second API specification. It needs enough information to route the failure safely.

type ToolError = {
  code: "TRANSIENT" | "VALIDATION" | "PERMISSION" | "NOT_FOUND";
  message: string;
  retryable: boolean;
  field?: string;
  suggestion?: string;
  retryAfterMs?: number;
  traceId: string;
};

Each field has one job.

code selects a known failure class.
message explains the problem to the model and operator.
retryable stops the controller from inventing its own retry rule.
field identifies the argument that needs repair.
suggestion supplies a bounded correction rather than an open-ended rewrite.
retryAfterMs prevents an immediate retry from amplifying throttling.
traceId connects the agent-visible failure to logs and traces.

Do not put secrets, raw database errors, or internal stack traces into a model-visible message. The public contract should expose the recovery surface, not the server's internals.

Separate routing policy from error prose

A controller can route the envelope without asking a model to interpret every sentence.

function nextAction(error: ToolError): "retry" | "repair" | "authorize" | "stop" {
  if (error.code === "TRANSIENT" && error.retryable) return "retry";
  if (error.code === "VALIDATION" && error.field && error.suggestion) return "repair";
  if (error.code === "PERMISSION") return "authorize";
  return "stop";
}

This is intentionally boring. A deterministic router is easier to test than a prompt that says, “Use your judgment and try again if appropriate.”

The model still matters when a validation error requires semantic repair. It may need to convert “next Friday” into an ISO-8601 date or select a replacement resource. But the controller should decide whether a repair turn is allowed before spending another model call.

GitHub's MCP Scripts specification takes this idea further for execution errors by requiring a data.recoverable boolean. It limits recoverable: true to transient failures where the same invocation may succeed on a later attempt (GitHub MCP Scripts error handling). That is the kind of field a controller can enforce without semantic guesswork.

The prompt trace should preserve the boundary

Here is the exact interpretation constraint used for the benchmark summary:

Prompt:
Compare the three tool-error strategies using only summary.json. Separate
measured simulation output from scenario assumptions. Do not claim LLM accuracy.

Input:
Generic message: 67.1% resolved, 3,953 failed retry attempts
Error code only: 71.2% resolved, 1,005 failed retry attempts
Actionable envelope: 75.5% resolved, 487 failed retry attempts

Output constraint:
Report resolution rate, attempts per operation, failed retries, and permanent-
failure retries. State that all recovery probabilities are explicit synthetic
assumptions and that no model was called.

That last line prevents a common reporting error. A controller simulation can test policy logic. It cannot tell you how reliably a model will interpret a repair hint.

Retry safety needs more than a boolean

retryable: true is necessary and insufficient.

A production retry path should also answer:

Is the operation idempotent?
Has this workflow already produced a side effect?
What is the maximum attempt budget?
Should the controller use exponential backoff and jitter?
Did the server provide a rate-limit reset time?
Does the repaired call need a new idempotency key or the old one?
Which failures require a human decision?

A retryable payment lookup and a retryable payment creation are different risks. The first may waste capacity. The second may charge twice.

For state-changing tools, store the operation ID and side-effect status outside the model context. A model should not be the source of truth for whether a transaction already happened.

Where this benchmark stops

The benchmark is deterministic and reproducible, but its scope is narrow.

It does not measure latency, tokens, model interpretation, parallel tool calls, ambiguous repair hints, or side-effect safety. The failure mix is synthetic. The recovery probabilities are declared assumptions. Resolution means success within one modeled recovery attempt, not completion of a full agent task.

Real evaluations should replay production-shaped traces, vary models and prompts, measure incorrect repairs, and test that terminal failures really stop. They should also include adversarial errors: misleading suggestions, mismatched codes, missing trace IDs, and a server that marks everything retryable.

The small benchmark is still useful because it tests the first layer without model noise: whether the controller has enough structure to choose a permitted next action.

Make failure actionable before making it eloquent

Start with four routes: retry, repair, authorize, and stop. Define the codes that lead to each route. Add an explicit retry flag, the smallest useful repair hint, and a trace ID. Then test the controller with fixed seeds before asking a model to interpret the message.

The practical test is simple: if the model were unavailable, could the runtime still avoid repeating a permission error, respect backoff, and identify the argument that needs repair?

If not, the error contract is still prose. Give it enough structure to become control flow.

Fine-Tuning vs Prompt Engineering vs RAG: When to Use Each (And the Fourth Option Nobody Mentions)

Sun, 09 Aug 2026 14:33:21 +0000

You have an AI model that hallucinates with your company's data. You open a support ticket and ask the chatbot about your returns policy. The chatbot, powered by GPT-4 or Gemini 2.5, responds with a fabricated policy that sounds perfectly plausible but has nothing to do with your company's reality. Your boss stares at you. Your customer complains. You open Google and search "how to connect LLM to my data".

The top 10 results offer three contradictory answers: "use RAG," "do fine-tuning," "improve your prompt." All three are right. All three are wrong. Because the correct answer isn't any of the three in the abstract — it's the one that fits your specific use case. And there's a fourth option that almost nobody mentions, and which, in my experience, is the right call in more cases than the industry admits.

This article is the decision tree I wish I'd had when I started building the AI systems behind this blog. I've distilled it after implementing all four techniques in real production: Prompt Engineering across the entire Autopilot series, RAG in the Ops Copilot with Algolia, pure Tool Calling in the Obsolescence Radar, and experimental fine-tuning in industrial classification pipelines. It closes the trilogy that began with RAG: 7 Anti-Patterns and continued with MCP Protocol.

The Question Nobody Asks

Before choosing a technique, ask yourself this: Does the knowledge your LLM needs change, or is it static?

If the answer is "it changes frequently" (product documentation, inventory, prices, regulations), you need a technique that accesses data in real time without retraining the model. If the answer is "it's static or changes very slowly" (brand tone, formatting rules, domain nomenclature), you can consider techniques that incorporate that knowledge into the model.

This distinction is the first node of the decision tree. It seems obvious written this way. Yet most teams I've seen jump straight to whatever technique is trending (RAG in 2024, fine-tuning in 2023, prompt engineering always) without asking this fundamental question.

Option 1: Prompt Engineering — 80% of Cases

The uncomfortable truth the AI tooling industry doesn't want you to know: for 80% of use cases, a well-designed prompt is sufficient. You don't need RAG. You don't need fine-tuning. You need a system prompt that clearly defines the role, context, constraints, and expected output format.

When it's sufficient:

The required knowledge fits in the model's context window (Gemini 2.5 handles up to 1 million tokens; Claude up to 200K).
The task is generic but needs structure (drafting emails, summarizing documents, classifying text, generating code).
You don't need updated proprietary data — the model's general knowledge is enough.

Advanced techniques that make the difference:

Structured system prompts: Define the role ("You are a senior supply chain engineer"), constraints ("Always respond in technical English"), and output format ("Return a JSON with fields: analysis, recommendation, confidence").
Few-shot prompting: Include 3-5 correct input-output examples in the prompt. In the Autopilot series, CrewAI agents use few-shot to maintain style consistency across generated articles.
Chain-of-thought (CoT): Instruct the model to "think step by step" before giving the final answer. Dramatically improves accuracy in reasoning, calculation, and multi-step analysis tasks.
Prompt chaining: Break complex tasks into sequential subtasks, each with its own optimized prompt. This is exactly what CrewAI does with its agent architecture: each agent has a specialized prompt for its role.

Cost: Virtually zero (only API token cost). A well-designed prompt can take hours of iteration, but operational cost is minimal.

Fatal limitation: The context window has a limit. If you need the model to "know" about 10,000 documents from your knowledge base, you can't inject them all into the prompt. This is where RAG enters.

Option 2: RAG — Updatable Proprietary Knowledge

RAG (Retrieval-Augmented Generation) is the right answer when you need the LLM to respond about your proprietary knowledge and that knowledge updates frequently.

When it's necessary:

Product documentation, technical manuals, internal knowledge bases that update weekly or monthly.
The user can ask unpredictable questions about a broad document corpus (you don't know in advance which fragment the LLM will need).
You need citability: the answer must include the sources it draws from (critical for compliance, as we documented in the EU AI Act, Article 13 on transparency).

When NOT to use it: When data is structured (SQL tables, APIs with defined schemas) or when you need numerical precision. As I extensively documented in Anti-Pattern 7 of the RAG article, RAG over structured data generates narrative hallucinations where you need exact figures.

Correct architecture (summarized):

Semantic chunking (not fixed-length — Anti-Pattern 1)
Evaluated embeddings with your domain benchmark (Anti-Pattern 2)
Reranking between retriever and LLM (Anti-Pattern 3)
Generous context (top-10/15, not top-3 — Anti-Pattern 4)
Evaluation with RAGAS/DeepEval before production (Anti-Pattern 6)

Real cost: Moderate. The vector store (Pinecone, Weaviate, Algolia) has a monthly cost (€0-100 depending on volume), plus embedding cost (low) and generation cost (API tokens). In the Ops Copilot, the total RAG cost with Algolia was under €3/month for the blog's ~70 posts.

Real case at Datalaria: The Ops Engineering Copilot (Autopilot Part 8) uses RAG with Algolia Agent Studio to answer questions about blog content. Posts are indexed as semantic records (one record per section), and the copilot retrieves relevant fragments before generating the response. Works well for semantic search over free text.

Option 3: Fine-Tuning — The Scalpel, Not the Hammer

Fine-tuning is the most powerful technique and the most misused. It involves partially retraining a base model (Gemini, Llama, Mistral) with your own data so the model internalizes specific knowledge, style, or behavior.

When it's essential:

You need the model to adopt a very specific tone or style consistently (a brand with strict voice & tone, a domain with very particular technical jargon).
The task is highly specialized and generalist models don't solve it well even with advanced prompting (industrial defect classification, proprietary nomenclature entity extraction, specialized medical diagnosis).
You need to reduce latency and cost in production: a fine-tuned smaller model (7B-13B parameters) can match the quality of a large model (70B+) on your specific task, at a fraction of the cost and latency.

When NOT to use it (the most widespread myth):

Don't use fine-tuning to "teach the model data." Fine-tuning is not a database. If you need the model to know your product catalog, use RAG. Fine-tuning "burns in" behavioral patterns, not updatable facts.
Don't use fine-tuning if your knowledge changes frequently. Each update requires retraining, which can cost hours and hundreds of euros. RAG is instant: update the document and the retriever finds it immediately.

Modern tools:

LoRA (Low-Rank Adaptation): The standard technique. Instead of retraining the model's billions of parameters, LoRA trains only low-rank matrices "attached" to the model's layers. Reduces training cost by 90%+ and stores the fine-tuned model as a few MB of "adapters."
QLoRA: LoRA applied to a 4-bit quantized model. Enables fine-tuning 70B-parameter models on a single consumer GPU (24GB VRAM). Democratized fine-tuning for startups and teams without GPU clusters.
Vertex AI Tuning / OpenAI Fine-Tuning API: Managed services where you upload your training dataset (instruction-response pairs) and the platform runs the fine-tuning without you managing GPU infrastructure.

Real cost: Variable. Fine-tuning with LoRA on a 7B-parameter model with 10,000 examples costs €5-20 on cloud (Google Cloud, AWS). A 70B model can cost €50-200 per training session. Plus the cost of preparing the dataset (hours of human work). As we analyzed in The Hidden Economics of AI, fine-tuning's hidden cost isn't compute — it's training dataset curation.

Option 4: Tool Calling / MCP — The One Nobody Mentions

This is the option I discovered by elimination after RAG failed spectacularly on the Obsolescence Radar. Tool Calling means the LLM doesn't try to "know" the answer; instead, it knows who to ask — that is, which tool to execute to get the information with deterministic precision.

When it's the right option:

Data is structured (SQL databases, REST APIs, spreadsheets with schemas).
You need absolute numerical precision (financial calculations, inventory metrics, sensor data).
The operation requires actions, not just answers (create a ticket, send an email, run a query, call an external API).
You want to standardize connections between the LLM and tools to avoid vendor lock-in — exactly the problem solved by MCP (Model Context Protocol).

Architecture: The LLM (Gemini 2.5, Claude) acts as a semantic orchestrator: it understands the user's natural language intent, decides which tool(s) to execute, constructs the parameters, executes the tool(s), and interprets results for the user. Tools are deterministic Python functions (decorated with @tool in CrewAI) that execute precision operations: SQL queries to Supabase, supplier API calls, linear programming calculations with PuLP.

Cost: The lowest of all four options. You only pay for LLM tokens (typically few, since the prompt is short) and tool execution (SQL queries, API calls). In the Obsolescence Radar, the cost per complete execution (analyze a component, traverse the BOM graph, calculate financial impact, generate executive report) was under €0.02 per query.

Real case at Datalaria: The Agentic Obsolescence Radar uses Tool Calling exclusively. The LLM (Gemini 2.5 via CrewAI) understands the obsolescence alert in natural language, but all data operations — SQL query to the component catalog, BOM graph traversal, P&L calculation, PDF generation — are executed by deterministic Python tools. Result: executive reports in 4 seconds with 0% numerical hallucination.

The Decision Matrix

Criterion	Prompt Engineering	RAG	Fine-Tuning	Tool Calling
Initial cost	⭐ Minimal	⭐⭐ Low-medium	⭐⭐⭐ High	⭐⭐ Low
Operational cost	⭐ Low	⭐⭐ Medium	⭐ Low (small model)	⭐ Minimal
Precision (free text)	⭐⭐ Medium	⭐⭐⭐ High	⭐⭐⭐ Very high	⭐ N/A
Precision (structured data)	⭐ Low	⭐ Low	⭐ Low	⭐⭐⭐ Exact
Data freshness	⭐⭐⭐ Instant	⭐⭐⭐ Instant	⭐ Requires retraining	⭐⭐⭐ Real-time
Implementation effort	⭐ Hours	⭐⭐ Days-weeks	⭐⭐⭐ Weeks-months	⭐⭐ Days
Maintenance	⭐ Minimal	⭐⭐ Medium	⭐⭐⭐ High (data drift)	⭐⭐ Medium
Traceability (EU AI Act)	⭐ Difficult	⭐⭐⭐ High (citable sources)	⭐ Opaque (black box)	⭐⭐⭐ Total (deterministic)
Ideal use case	Generic tasks with clear instructions	Updatable proprietary text knowledge	Specific style/tone, ultra-specialized tasks	Structured data, numerical precision, actions

The 3-Question Framework

If the matrix seems dense, I've distilled a 3-question framework that resolves 90% of decisions:

Question 1: Does the data the LLM needs fit in the prompt?

Yes → Prompt Engineering. Inject context directly. Simpler, cheaper, faster.
No → Next question.

Question 2: Is the data free text or structured?

Free text (documentation, manuals, posts) → RAG. Semantic retrieval is superior for searching unstructured text.
Structured (SQL, APIs, tables, calculations) → Tool Calling. Deterministic tools that execute exact queries.
Both → Hybrid architecture (RAG for textual context + Tool Calling for structured data, as we proposed in the RAG article).

Question 3: Do you need a behavior or style the base model can't reproduce even with the best prompt?

Yes (unique brand tone, ultra-specific domain jargon, task that generalist models consistently fail) → Fine-Tuning on a base model.
No → Go back to Prompt Engineering and refine your prompt before considering more complex techniques.

What I Learned Implementing All Four

The most valuable lesson from operating these four techniques in production fits in a single sentence: always start with the simplest technique that could work.

The temptation is to jump straight to RAG or fine-tuning because they're more "sophisticated." But sophistication doesn't correlate with effectiveness. In the Autopilot, the majority of output quality comes from Prompt Engineering — carefully designed system prompts, few-shot examples, and Chain-of-thought. RAG added marginal value in the Ops Copilot for blog search. Fine-tuning wasn't necessary in any case. And Tool Calling was the transformative technique in the Obsolescence Radar, where RAG had failed.

The evaluation order should always be:

Prompt Engineering (hours, ~€0)
Tool Calling if data is structured (days, ~€0)
RAG if you need access to proprietary text (days-weeks, ~€3-50/month)
Fine-Tuning only if the previous three consistently fail (weeks, €50-500+)

And in the era of the EU AI Act, there's a fifth consideration that isn't technical but legal: traceability. Article 10 of the regulation requires that training data for high-risk systems be "relevant, representative, and to the extent possible, free of errors and complete." This applies directly to fine-tuning: if you fine-tune a model with biased or incorrect data, and that model makes decisions in a regulated domain, you're exposed to sanctions. RAG and Tool Calling, being transparent in their sources, offer traceability that fine-tuning cannot match.

As we wrote in The Hidden Economics of AI, the 10x Rule applies: if a more complex technique doesn't give you a result 10 times better than the previous one, it probably doesn't justify its added cost and complexity. Start simple. Measure. Scale only when the data demands it.