A team decides its agent costs too much. Someone reads that small models are the future of agent work, picks a 3B model, and points the whole agent at it. Ticket classification holds up. The open-ended replies fall apart within a day, and the team ro...

A team ships a small Python agent with two tools. A month later someone renames an argument on one tool function and forgets the hand-written JSON schema sitting a few lines above it. The model keeps sending the old argument name, the function raises...

A support agent passes every eval in staging. In production the same agent is reached three ways: from an IDE over MCP, from the web app over the OpenAI Responses API, and from a triage agent over A2A. The dashboard stays green for weeks. Then a cust...

Most teams optimize prompts by hand: change a line, rerun, eyeball the output, repeat. It works until you have more than a handful of cases, then it stops scaling. A prompt optimizer replaces that with a scored search you run on demand, and you do no...

An MCP-connected agent passes staging and then fails in production. A user sends a query slightly outside the test distribution, the agent picks the wrong tool, passes malformed arguments, chains three unnecessary calls, and returns garbage. The reas...

A search-augmented agent looks like RAG until the citation list breaks. The first user clicks through, lands on a 404, and the trust budget spends in one impression. Three months of eval work, groundedness and context adherence and factual accuracy a...

Your multi-agent system works locally. Three agents coordinate, call tools, pass context, return a clean answer. In production the final output is wrong, and you cannot tell which agent failed, which tool call returned garbage, or where the reasonin...

Your custom metric is scoring noise unless it has these three parts A medical agent passes your faithfulness and refusal gates all quarter. A week after launch, a review flags outputs with unverified drug-interaction claims. The faithfulness rubric...

A Pydantic AI agent returns BookingResultconfirmed=True, flightid='AA127', totalusd=412.50. Every Pydantic check passes. The outputtype matches. The tool-call arguments validated against the Pydantic model on the way in. The retry counter is at zero....

A support model picks refundorder, fills the schema, validates clean, and pushes $54 to the right account. The user asked about $5,400. Tool selection scored 1.0. The arguments passed schema validation. The number was off by an order of magnitude, an...

You have eleven versions of the same prompt in a doc, three of them named "final," and no real idea which one scores best on last week's failures. So you tweak it again by hand, read ten outputs, decide it seems better, and ship. That is not optimiza...

I have sat in this incident more than once. A user reports a regression on Tuesday. On-call pulls the trace, and the span tree is complete: provider, model, tokens, finish reasons, retrieval chunks, eval scores. None of it carries the prompt version....

Every MCP server I have worked on is really two products under one binary. The first is the functional surface: tool descriptions that match what the tools actually do, calls that succeed with the right arguments, results that flow into the next age...

I keep seeing the same failure land on teams that do everything right. The CI eval gate has been green for six weeks, faithfulness at 0.87, task completion at 0.91. Then a support thread surfaces a hallucination on a feature shipped in March. The tra...

How to evaluate a voice agent before launch with simulation Evaluating a voice agent used to mean listening to recordings, then it meant scripts that replayed transcripts at the agent. Neither catches much. The approach that works now is to simula...

The six conversation metrics that catch what voice agent dashboards miss Most teams have shipped voice agents and most are not happy with them. In Hamming AI's State of Voice AI 2026, 87 percent of companies have deployed voice agents but only 12 p...

A trace is one of the more useful things you can capture from an LLM app, and many teams store it in a format they cannot move later. If your AI tracing uses its own private format, the spans sit in a silo, a backend change means re-instrumenting, an...

I want to start with a trace that still bothers me. An agent fails to book a flight. The model called searchflights with departuredate="next Friday". The endpoint expected an ISO date, returned a 400, and the agent retried four times with the exact ...