Coder Legion - Recent posts tagged a2a

How to evaluate agents with one trace across MCP, A2A and Realtime

Wed, 14 Oct 2026 16:22:41 +0000

A support agent passes every eval in staging. In production the same agent is reached three ways: from an IDE over MCP, from the web app over the OpenAI Responses API, and from a triage agent over A2A. The dashboard stays green for weeks. Then a customer escalation shows the triage agent handed over a ticket without the customer's region, and the support agent quoted the refund policy for the wrong country. No evaluator flagged it, because no evaluator ever looked at the hop.

The reason is structural. Eval is a property of the path a request took, and the protocol defines the path. Most eval stacks were written against one surface, usually chat completions, so they score whatever still looks like a chat completion and quietly skip the rest. A protocol-neutral eval stack scores the path instead, whichever protocol carried the request.

Every protocol adds seams to the path

A plain chat completion has one path: prompt in, response out. One rubric can grade it. Each protocol you add forks that path into more decision points, and each decision point needs its own check.

MCP adds a tool catalog. The path becomes prompt, tool descriptor list, tool selection, call arguments, tool result, final answer. Score each tool call against policy, and score the catalog as a whole, because a poisoned descriptor looks fine at call time and only shows up against the full catalog.
A2A turns the path into a tree. Every hop needs three answers: did the calling agent pass enough context, did the called agent stay in scope, and did the returned artifact match the task that was asked for.
The Responses API puts three outputs in one envelope. Structured JSON gets graded against a schema, streamed text gets graded as it arrives, and a tool call gets graded against a per-tool policy. One end-to-end pass mixes three things that share nothing except the wrapper.
Realtime has no final message. Audio streams in both directions, so checks have to run mid-stream, every N tokens or N audio chunks, with a stop or disclaimer action when a rule fires. The end-to-end budget sits below 500ms, which caps how heavy each check can be.
Inline vision needs a second judge. When images ride in the same message as tool blocks, the text portion and the image portion need different rubrics, and the image rubric needs a judge that can open the image.
Session state spans turns. On a framework with explicit session state, the evaluator has to read state across turns instead of grading each turn alone.

Trace fidelity caps eval fidelity

An evaluator can only score what the trace can represent. If your trace has no way to express an agent-to-agent hop or a streaming chunk, no rubric downstream will catch a failure inside one.

So the first fix is in instrumentation, before any rubric work:

Model both sides of a hop. Emit one span kind on the calling agent and another on the called agent, and carry the trace ID across the wire so the whole multi-agent path lands in one trace.
Give each modality its own span kind. Voice, image and computer-use spans should be distinguishable from text spans, so each one reaches an evaluator built for that modality.
Record guardrail and evaluator results as spans. When a guardrail decision and an evaluator score sit in the same trace as the model call, you can see which check ran on which step.

The MCP spec helps here. The 2026-07-28 revision documents trace context propagation through _meta (traceparent, tracestate, baggage), so MCP tool calls can join the same trace as the rest of the agent loop.

Normalize the spans, then write each rubric once

The same agent identity shows up differently on every protocol. If you write a groundedness check against raw MCP payloads and another against raw Responses payloads, you now maintain two checks that drift apart.

Normalize first. Map every span to one semantic convention, such as the OpenTelemetry GenAI conventions, before any evaluator reads it. Then write rubrics against the normalized trace. One groundedness rubric can then grade a RAG answer that arrived over the Responses API and a RAG answer that arrived over MCP, and one task-completion rubric can grade the support agent no matter which client reached it.

This is the same move observability teams made with OpenTelemetry. Teams that locked to one vendor's tracing format rebuilt their instrumentation every time they switched vendors. Agent protocols are repeating that pattern one layer up.

Gate releases per protocol

A single overall score hides which surface regressed. A drop in the MCP tool-audit rubric and a drop in the Realtime guardrail are different failures with different fixes.

Keep a regression dataset per protocol surface. Run the matching rubric set against it on every change.
Set a threshold per protocol. Fail the release when any one surface drops below its own threshold, even if the blended number looks fine.
Key each suite on protocol and version. MCP's 2026-07-28 revision removed the initialize handshake, so every request now carries its protocol version in _meta, and the new deprecation policy gives every deprecated feature a minimum twelve-month window. Tag each regression case with the revision it ran on, and keep the tests for deprecated features until their window closes.
Keep the rubric sets in the repo as code. Reviewers should be able to read exactly what gets scored on each surface.

A five-step readiness audit

Inventory every surface in production. MCP servers, A2A endpoints, streaming, voice, vision and computer-use, with the protocol version and client library for each. Most teams find one they did not plan for, usually an MCP server a developer installed last sprint.
Instrument and pull one real trace per surface. Confirm the hop spans appear on both sides with the propagated trace ID, and that voice and image calls emit their own span kinds. Fix gaps before writing rubrics.
Write the rubric set for each protocol. Per-tool audit on MCP, per-hop audit on A2A, split rubrics on the Responses API, mid-stream checks on Realtime, an image-capable judge for vision.
Run a per-protocol regression suite and gate on per-protocol thresholds.
Cluster production failures by protocol. Failures that look like noise one at a time often form a pattern once you group them by surface.

If you only have half a day, do step 1. The list is usually longer than the team expects, and it tells you which evaluators are scoring the wrong path today.

Pitfalls that keep showing up

Evaluators hard-coded to the chat completions shape. Every new protocol triggers a rewrite.
Scoring each agent in isolation. The end-to-end view looks green while a handoff in the middle dropped context.
Post-call checks only on streaming surfaces. By the time the evaluator sees the output, the risky tokens are already on the wire.
Accepting tool descriptors from any MCP server without a scan. An untrusted server can change what your agent does, and the eval layer never sees it.
Scoring voice and computer-use on text alone. Errors at the ASR layer and the vision layer stay invisible.

The span kinds, the rubric set for each of the six protocols and the full audit are in the longer protocol-neutral eval playbook.

New protocols and revisions will keep arriving. Score the path the request took, and each new surface plugs into a trace model and a rubric library you already maintain.

An Ai built me a demo, I built it into a business

Tue, 29 Sep 2026 23:01:08 +0000

I run several AIs in parallel — Claude, Grok, ChatGPT, Gemini — and I feed the wins to my main agent to reality-check against my actual repos before anything ships. This week Grok built me a slick demo of an "SML Desk": a command-center web app for my whole operation.

There was one problem. I couldn't ship it.

The source was stuck behind x.ai auth. The "video clips" it linked to were sandbox-only files that didn't exist in production. The data was demo-shaped, not real. It looked done and wasn't.

So I rebuilt it clean. Same idea, zero borrowed code, wired to my real stack. It's live now at desk.scriptmasterlabs.com — a web app, no install, no signup. Here's what that rebuild actually took.

What the Desk is

One page, six tabs, all client-side:

Desk — the command surface: my live x402 API directory (23 pay-per-call endpoints, every one priced in USDC)
Stack — the product stack: high-ticket courses, Squeeze Wire, Kindle editions, with verified prices
Directory — every canonical page on the domain
Agents — the agent tools: SMLSheets, Micro-Series Studio, the open-seo fork
Ledger — claim receipts. Every factual claim on the page carries a receipt
Shelf — 50 research PDFs, loaded live from a JSON catalog

The design brief I gave my agent: cream editorial, like a banker's desk. It kept my branding intact — dark/neon is the main site's identity, but the Desk is its own room.

The part that actually mattered: every number verified

My trust rule is simple: every number sourced or it doesn't reach the user. A demo can fake an API price. A shipped product can't. So the rebuild wired every tab to real sources:

The 23 x402 paths were verified live (HTTP 200) before they went in the directory — not copied from a spec doc
The 50 PDFs load from the live research-catalog.json with CORS opened for the desk subdomain
Product prices were re-checked against the live storefronts, not admin panels

The Shelf tab is the cleanest example. It fetches the catalog at runtime:

fetch('https://scriptmasterlabs.com/research-catalog.json')
  .then(r => r.json())
  .then(catalog => renderShelf(catalog));

That only works because the main domain serves the catalog with Access-Control-Allow-Origin: *. Small detail, the kind the demo never had to solve.

The DNS saga (the demo never deploys)

This is the part nobody warns you about with subdomains. desk.scriptmasterlabs.com 404'd on launch day. The debugging went:

Assumed DNS was Cloudflare — wrong. The registrar is Porkbun, and Porkbun is also the authoritative nameserver. The Cloudflare headers I was seeing came from Render's own ingress edge.
Found the desk subdomain CNAME'd to proxy-v2.appdeploy.ai — a third-party Caddy I didn't recognize and never set up. Replaced with an A record to Render's ingress (216.24.57.1), same as the apex.
Still 404 — because Render won't serve TLS for a custom domain until you add it on the service. That cost $0.25/month (included domains exhausted). Accepted, cert issued, live.

Total: about 40 minutes of "the demo worked on my machine" energy, except the machine was someone else's sandbox.

Host-based routing, one server

The whole domain (apex + desk subdomain) runs off one Node server — frontdoor/server.mjs. The desk needed host routing added:

const host = req.headers.host || '';
if (host.startsWith('desk.')) {
  // serve desk/index.html, desk-scoped static paths
} else {
  // normal apex routing
}

One repo, one deploy, two sites. The GitLab push auto-deploys via Render. Commit 2fdca03 if you want the shape of it.

The thesis: evidence before the agent spends

The Desk's tagline is "Evidence before the agent spends." That's the whole business thesis: AI agents are about to spend real money on the internet via protocols like x402, and somebody has to be the desk where the evidence gets checked first — prices verified, claims receipted, endpoints machine-readable.

It's the same decision-gate I run everywhere: a model with ≥0.80 confidence auto-pays, 0.50–0.79 asks, below 0.50 escalates. The Desk is where a human or agent can check the evidence before the gate fires.

What I'd do differently

Ship the boring parts first. The demo had a "Clip" tab with video links that pointed at nothing — I cut it from v1 rather than fake it. It comes back when I have real video files. A tab that loads is worth more than a tab that impresses.

Live: https://desk.scriptmasterlabs.com —#FREE web app, runs in the browser, nothing to install.

Built by ScriptMasterLabs Publishing, a service-disabled veteran-owned small business. Every number on the page was verified live before it shipped. The demo looked done. This one is.

AI Agents Have a Communication Protocol Now. They Still Don't Remember Anything.

Wed, 09 Sep 2026 12:00:00 +0000

At Black Hat this year, agent sprawl was the phrase everyone kept using: enterprises spinning up dozens, then hundreds, of AI agents faster than anyone could govern them. Most of that conversation was about security — who's watching all these agents, and what happens when one goes rogue.

Talk to Vikas Mathur, MariaDB's chief product officer, and you get a different worry. Not whether the agents can be controlled, but whether they can actually understand each other.

Google's Agent2Agent (A2A) protocol just moved into the Agentic AI Foundation, joining MCP under one governance roof. Most of the coverage has called this consolidation — two standards, one home, interoperability problem solved. Mathur isn't convinced that's the right word.

"I think it's the right move for the industry in general," he said, "but I don't know about consolidation. Both of them happening to be there means that we'll move faster."

Faster at what matters here. A2A gives agents a way to describe what they can do and how to talk to each other. It doesn't tell them what to remember, or give them any shared sense of what's true.

"It doesn't solve for the agent amnesia, if that's the right word to use," Mathur said. "It solves for communication, but it doesn't solve for context."

That gap shows up fast inside a large company running dozens or hundreds of agents built by different teams and vendors. Mathur's example: an employee shows up in an HR system under one ID, a GitHub handle in another system, a full name in a third. A2A doesn't resolve any of that. Nothing in the protocol says these are the same person. Agents can be technically communicating and still be talking past each other.

"They could be saying they're talking about the same thing, but really interpreting them completely differently," Mathur said. "It comes down to not having shared context being communicated across them. It happens in the real world with humans. It happens with agents too."

MariaDB's answer is building that shared context underneath the protocol layer, not inside it. Over the past two years, the company has added vector search, JSON, and distributed analytics to what used to be a straightforward relational database, and it recently acquired GridGain, adding a distributed key-value and caching layer built for the short-lived, high-volume state agents generate. Mathur says GridGain customers are running close to 2 million read queries a second at roughly 200 microseconds of latency — a different order of magnitude than the 10 to 50 millisecond range a well-tuned relational database delivers at scale.

That speed helps with session state. It doesn't, on its own, solve the harder problem: building a dynamic knowledge graph that resolves entities across agents and gives them something to agree on. Mathur describes that graph as the layer that turns raw agent activity into a trustworthy shared history — what worked, what didn't, and why — built and resolved on the fly rather than hand-maintained the way older knowledge graphs were.

It also creates a new access problem. Pulling scattered data into one place makes it useful. It also makes it sensitive in a way it wasn't when it was scattered.

"The moment you get into knowledge graphs that are across agents, now you've got to start looking at who can really look at this," Mathur said. "You're pulling information out, connecting it, which is great, so nothing is lost. But it becomes sensitive the moment you get so much information in one place."

For developers building multi-agent systems now, Mathur's advice is unglamorous: skip the ontologies and knowledge graphs for a minute, and get the data out of application silos first. Most enterprise data is still locked to whatever system created it. MariaDB's pitch — echoed by other vendors doing something similar — is a layer in front of those systems, sometimes called a digital integration hub, that pulls data into one place without touching the systems of record underneath it.

"If there was anything to spend time on, if people have time to spend today, I would go do that," Mathur said. "Bring it together so you can actually build agentic workflows much more easily." Skipping that step is possible, he said, but it makes everything downstream a lot more painful.

Even if the memory and context problem gets solved, Mathur doesn't think the hard part is over. He puts orchestrating agents at real enterprise scale — thousands of them, not dozens — as the next bottleneck, right behind cost. Token spend is unpredictable in a way infrastructure spend isn't, and he's blunt about what happens when the accounting catches up.

"They're selling a plan for $200 and spending $8,000 in the backend," he said. "That's fine for now, with venture money. Three years from now, that's not going to happen."

A2A joining a shared foundation with MCP is a real step. It gives agents a common way to introduce themselves and describe what they do. But talking to each other was never the hardest part of building agents that work together. Remembering, and agreeing on what's real, still is. That part doesn't have a protocol yet.

Agent Economy — hype or real revenue? Let's share honest numbers.

Tue, 01 Sep 2026 03:19:50 +0000

I’ve been following the AI agent space closely — building a DeFi data API for agents, an MCP server, and recently a Telegram Mini App that uses LLMs.

There’s a lot of excitement. But honestly? I haven’t seen real, sustainable revenue from agent-driven traffic yet.

Most agents are still either:

Crawling (reading data)

Testing

Running on free tiers

Or just "talking" without paying

Is this just me? Or is the agent economy still in the "infrastructure phase" where the money is being spent on building, but not yet earned by agent-based services?

Would love to hear different perspectives from builders here:

Have you seen real revenue from agents paying for APIs / services?

Are we early? Or is this just a bubble that will take years to mature?

Let’s share honest experiences — no hype, just real talk.

Is the "Agent Economy" actually generating revenue yet? Or is it still mostly hype?

Tue, 01 Sep 2026 03:18:48 +0000

I’ve been following the AI agent space closely — building a DeFi data API for agents, an MCP server, and recently a Telegram Mini App that uses LLMs.

There’s a lot of excitement. But honestly? I haven’t seen real, sustainable revenue from agent-driven traffic yet.

Most agents are still either:

Crawling (reading data)

Testing

Running on free tiers

Or just "talking" without paying

Is this just me? Or is the agent economy still in the "infrastructure phase" where the money is being spent on building, but not yet earned by agent-based services?

Would love to hear different perspectives from builders here:

Have you seen real revenue from agents paying for APIs / services?

Are we early? Or is this just a bubble that will take years to mature?

Let’s share honest experiences — no hype, just real talk.

Let us learn MCP and A2A without the boring stuff

Tue, 06 Jan 2026 05:37:49 +0000

Through humour, simple analogies, and real-world examples, Jigyasu and Saral explore:

What MCP Servers are and who builds them
How MCP makes AI Agentic systems scalable, maintainable, and safe
What is Google's A2A
and more

-> Next episode: AI Agent Security

Previous episodes:

0. Basics of AI - https://lnkd.in/gWWHqJcn

What is an AI Agent - https://lnkd.in/gbWVEfyr
Multi AI Agents - https://lnkd.in/gJFg_UrU
AI Agent Memory - https://lnkd.in/gjWRiZdV
AI Agent Evaluation - https://lnkd.in/gj34yKqp