Coder Legion - Recent posts tagged itleadership

Agents 2.0: What a Year of Actually Building Them Taught Us

Wed, 10 Jun 2026 15:00:00 +0000

Last year at Info-Tech LIVE, Martin Bufi warned the audience that agentic AI was overhyped and underbuilt. Seventy percent of organizations had launched agent pilots. Ninety percent were failing to return ROI. The gap between ambition and readiness was obvious.

Then the market did what the market does. It heard "autonomy" and ran with it.

Bufi, Principal Research Director at Info-Tech Research Group, opened this year's feature keynote with a viral cautionary tale. A developer using Meta's OpenClaw agent typed what seemed like a reasonable instruction: clear everything in the inbox older than February 15. The agent executed. Immediately. Hundreds of emails deleted. The human typed "do not do that." The agent kept going. "Stop don't do anything." Still going. "STOP OPENCLAW." The user had to physically kill every process on the host machine to get it to stop.

The agent later apologized. It acknowledged that it had acted without showing the plan first. Which, as Bufi noted, is not the same as not acting.

"Autonomy without architecture isn't innovation," he said. "It's chaos."

While that was happening at scale across the industry, Info-Tech took a different approach. They treated it as an engineering problem. Prototype fast. Measure everything. Ship only what clears the bar. After a year of work, the results: 13 prototypes, 63 agents built, 123 custom tools created, across five enterprise domains — finance, HR, IT, engineering, and retail. Average run time: 177 seconds. Average cost per run: $0.43.

Those numbers matter. Here's what produced them.

Story 1: The Whiteboard Week

Every prototype started the same way. Not with code. With a whiteboard.

Bufi described the moment that made this discipline non-negotiable. You ask five people how a workflow operates. You get five different answers. PDFs, emails, screenshots, spreadsheets — everyone has a different version of the process, and one of those people has been doing it the same way for 20 years with knowledge that lives entirely in their head.

"Standardizing the workflow is half the build," he said. "Before you automate it, map it. Before you map it, standardize it."

The IT service desk workflow Bufi walked through makes this concrete. What looks like a simple five-step process — intake, triage, routing, response, closure — carries layers of reasoning logic, edge cases, and data dependencies that only become visible when you document them explicitly. The agent can't handle what wasn't mapped.

One more thing they learned: generalist agents fail unpredictably. Every prototype Info-Tech built was multi-agent. An orchestrator delegates to specialists. An extractor pulls fields from documents. A validator checks data against rules. A router sends work to the right queue. An escalator flags exceptions for humans. "Not one genius agent," Bufi said, "but a team of simple ones."

Story 2: 123 Tools for 63 Agents

The model is the ceiling. Tools and architecture are the floor.

Info-Tech built 123 custom tools across their 63 agents. The real engineering work — the part that determines whether an agent actually performs in production — is in the integrations. An agent without tools is just a chatbot.

On safety, Bufi was direct: prompts alone aren't enough. They use three layers. First, narrow tools — give the agent a scalpel, not a Swiss Army knife. If it doesn't need delete access, it doesn't get delete access. Second, prompt instructions — useful guidance but not enforceable controls. Third, hard guardrails: structured validation on every call, blocking destructive operations at the software level regardless of what the model decides to do.

"That's like hoping your dog won't eat the food on the counter," he said of relying on prompts alone. "The hard guardrail is the one that guarantees it."

They also default to stateless design. Each agent run starts clean. No persistent memory across sessions unless the use case explicitly demands it. The reason is security: persistent state creates a surface area for prompt injection that compounds across runs. In a stateless architecture, if run 3 gets poisoned, run 4 starts clean. "Ninety-nine percent of agents work better this way," Bufi said. "Be deliberate about memory. Add it only when you have to."

Deployment follows microservice architecture. Each workflow is its own service. One workflow, one container. Triggered by webhook, cron, queue, or API. No monolith, no shared state, no single point of failure. When something breaks, you replace the service, not the whole system.

Story 3: The $0.43 Question

The governance conversation starts when someone asks: "What happens when this runs 500 times a day?"

At $0.43 per run, one workflow at 500 daily runs costs about $78K annually. Scale that to 25 workflows and you're at $2 million. That's not a surprise you want in a board meeting six months after deployment.

Info-Tech's governance framework answers three questions before any agent ships:

Can we afford it? Every run is costed. Not estimated — measured. Cost visibility is a deployment requirement, not an afterthought.

Can we stop it? Every agent is built with two and only two options when it hits ambiguity: consult a human, or exit gracefully. "There is no third option," Bufi said. "No fallback path, no deployment."

Can we improve it? They run continuous evaluations — not as a quality checkpoint, but as the development loop itself. One workflow went from 72% task completion accuracy on the first iteration to 96% by the third, through prompt rewrites, tool rerouting, and edge case handling. Evals aren't the finish line. They're how you get there.

Where Are You Right Now?

Bufi closed with three questions every builder should be able to answer before calling anything production-ready.

Can you draw the workflow end-to-end right now — not the happy path, but every edge case, every data dependency?

Does your agent fail safely when a tool goes down, or does it improvise a workaround you didn't design?

Do you know what last week's agents cost — and whether they were right?

If any of those made you hesitate, you know where to start.

The title of the talk was "Architect for Autonomy." The actual message was simpler: don't start with an agent. Start with a workflow. Build it with a method, not a vibe. And next year, come back with something deployed.

The 95% Enterprise AI Failure Stat Is Wrong. Here's What the Data Actually Shows.

Tue, 09 Jun 2026 20:23:10 +0000

If you've spent any time in IT circles over the past year, you've probably heard the number: 95% of enterprise AI pilots fail to deliver meaningful business impact. It's been cited in presentations, board meetings, and media coverage so often it's become accepted truth.

Tom Zehren isn't buying it.

Zehren, CEO of Info-Tech Research Group, opened Info-Tech LIVE 2026 in Las Vegas by taking direct aim at the statistic — and the research behind it. "The challenge when media picks up research is truncated information," he told a packed Grand Ballroom. "The shorter you make information, the older you make the information, the easier it travels."

His assessment of the source? "It's an eight-page marketing memo. And it's terrible research."

The specific study Zehren was referring to — widely cited as MIT research — surveyed a relatively small sample and was conducted roughly a year ago, when most enterprise AI deployments were early-stage ChatGPT experiments, not production agentic systems. "They talked about Gen AI one year ago," Zehren said. "What they were really talking about is chat." He also noted a fundamental timing problem: many pilots counted as failures simply hadn't completed yet.

His larger point wasn't to defend AI hype. It was to argue that bad data leads to bad decisions — and that IT leaders need to dig into the details before accepting a narrative.

What Info-Tech's Data Actually Shows

To back his argument, Zehren presented results from two large-scale surveys Info-Tech conducted across roughly 550 IT executives, with a second survey focused specifically on the software development lifecycle.

The findings tell a different story than the 95% headline. Forty-two percent of organizations have already deployed AI solutions across multiple departments — not just pilots — and have seen measurable impact. Another third have deployed customer-facing AI solutions. And perhaps most importantly, the top drivers for AI adoption aren't cost savings. They're growth and removing operational constraints that have held IT back for years.

"Cost reduction is at the bottom," Zehren said. "People are primarily trying to drive growth."

On the development side, 84% of software developers are now using AI across the entire software development lifecycle — not just for code generation, but for testing, documentation, and design. Sixty percent say AI-generated code requires more oversight, but Zehren sees that number differently than most. "If you flip that around, you get to a third of people saying it requires the same level or less. And leading software companies have already moved past the phase of fixing bugs from AI-generated code. The vast majority of code that is prompt engineered by developers at those companies is ready for production."

Five Hypes Worth Examining

Zehren structured his keynote around five AI narratives he believes deserve more scrutiny.

The Klarna story — often cited as proof that AI saves millions — is more nuanced than the headlines suggest. The company reduced its contact center headcount from 3,000 to 2,300 agents and saved approximately $40 million. But customer satisfaction tanked when emotional, complex cases like fraud disputes had no clear path to a human agent. The mistake, Zehren explained, was routing all calls in a specific region to AI agents only — eliminating the human escalation option entirely. Compare that to AWS, which used AI agents to migrate thousands of legacy Java applications to Java 17 — a use case with clear parameters, measurable output, and no customer-facing complexity. "You are not AWS," Zehren noted. "You don't have 30,000 applications running around."

On job displacement, Zehren aligned with World Economic Forum projections: roughly 190 million jobs created globally by 2030 against 92 million displaced — a net gain. The bigger challenge isn't job loss. It's the skills gap. "AI is not going to take your job," he said. "Someone who knows how to use AI is going to take your job."

On SaaS, he sees pressure building but not collapse. Seat compression, autonomous workflows, and the shifting economics of build versus buy are all real forces — but the major SaaS players are evolving their models rather than being displaced overnight.

The Bottleneck Nobody Talks About

In a media roundtable following the keynote, Zehren got more specific about where organizations actually get stuck. The problem isn't model quality or infrastructure. It's organizational.

"The biggest bottleneck, besides data, is the gap between business intelligence and understanding of workflows, and how you attach AI to get value out of them," he said. "That thinking and that knowledge sits in two very different heads. How do you bring those heads together in a structured way?"

It's a deceptively simple observation — and one that explains why Info-Tech's value proposition has sharpened in the AI era. The firm has grown more than 350% recently, even as AI has put pressure on traditional research and advisory models. The reason, according to Zehren, is the combination of the "what" and the "how." "If you have a fancy stat and you don't know how to exploit that — that's the problem. We can combine those two things."

As one participant in the roundtable put it plainly: "AI democratized access to research. That's no longer the valuable piece. The value now is the advisory piece — here's what you do, here's what it means, here's the step."

Zehren's closing argument was direct: this is IT's moment, not IT's threat. "We believe this is actually IT's moment to step up and move from the 'please give me this app' mode into the push mode — where IT understands the technology best and steps up as an organizational leader."

The organizations that figure that out first, he suggested, won't be asking whether AI delivers value. They'll already know.