Count your agent's narrow calls before you move any of them to a small model

Wed, 28 Oct 2026 13:46:09 +0000

A team decides its agent costs too much. Someone reads that small models are the future of agent work, picks a 3B model, and points the whole agent at it. Ticket classification holds up. The open-ended replies fall apart within a day, and the team rolls everything back.

The reason is structural. The team swapped the whole agent, while the savings live in individual calls.

Some calls repeat with little variation. Others need broad reasoning. Until you know which is which, you cannot know what a small model can take.

Many agent calls are narrow and repetitive

NVIDIA Research and Georgia Tech made this case in a 2025 position paper, Small Language Models are the Future of Agentic AI (arXiv 2506.02153). Many agent calls are narrow, specialized tasks. For those calls, the paper argues, small models are sufficient, better suited and more economical. It counts most models under about 10B parameters as small.

The cost gap is large. The paper states that serving a 7B model is 10 to 30 times cheaper in latency, energy and FLOPs than a 70B to 175B model. In a multi-step chain, that saving applies at every step.

Large models keep a role. The authors argue for heterogeneous agents. Small models take calls by default. A large model is called when general reasoning or open-domain dialogue matters.

The published percentages are estimates, not your numbers

The authors studied three open-source agents. They estimate that about 60% of MetaGPT's LLM queries could move to specialized small models. For Open Operator the estimate is about 40%. For Cradle it is about 70%.

Those numbers come from a position paper. They are not benchmark results. They also vary widely across three agents, and that spread is the useful part. Your agent's share depends on what your agent does.

So treat them as a reason to measure. Log a week of calls and count how many are narrow tasks that repeat with little variation.

Log every call that is not talking to a person

The paper's conversion recipe starts with logging. Capture the prompt, the output, the tool-call contents and the latency of every call that is not a direct exchange with a person.

The exchange with the user is usually the open-ended part. The calls behind it are where the repetition lives: classify the ticket, pull the order ID, pick the tool.

Record these as OpenTelemetry spans. Wrap the agent, each tool and the overall chain, so every call shows up under its parent. You need that structure later, when a bad answer has to be traced back to one step.

Cluster the calls into jobs

Once the logs fill up, group the calls into recurring tasks. Typical clusters are intent recognition, data extraction and document summarization.

Count each cluster. Its size tells you how much traffic a small model could take over.
Read a sample from each one. A cluster that looks narrow can still hide long or unusual inputs.
Strip sensitive data first. Remove PII, PHI and other sensitive fields before any call becomes a training example.

The paper's rule of thumb for fine-tuning is 10K to 100K examples per task. Check how close each cluster gets to that range before you plan a fine-tune for it.

Pick and train one small model per job

For each cluster, pick candidate models. Weigh capability, license and deployment footprint. Then test them on your own eval set, because a public benchmark measures general skill and your agent step is narrow.

Several current small models document tool calling on their model cards. They include Qwen3.5 (0.8B to 9B), Gemma 4 E2B and E4B, Ministral 3, Phi-4-mini and SmolLM3-3B. In that set, Qwen3.5-0.8B is the smallest with documented tool calling.

Fine-tune with LoRA or QLoRA on a single GPU. Another option is distillation: train the small model to mimic the large model's outputs on the task data. Either way, build each dataset to look like production traffic, with the same length, noise and label distributions.

Score every swap against the model it replaces

Before any traffic moves, run the small model and the model it replaces through the same evals. Use a metric that fits the job:

Router. Routing accuracy on a held-out set.
Extractor. Exact-match or structured-field accuracy.
Reasoning step. Faithfulness against the supplied context.

Then run end-to-end scenarios across the whole chain. Per-agent evals miss handoff bugs. One agent emits a malformed handoff token, the next silently drops the request, and the workflow returns a wrong but plausible answer. A full-chain test is what catches it.

Keep a fallback and retrain one piece at a time

Route the easy calls to the small model, and keep a frontier model as a fallback behind the same endpoint. Small models are less reliable on adversarial or out-of-distribution inputs, so low-confidence cases go to the stronger model.

The last step of the recipe is iteration. Retrain on fresh data as usage shifts. When one agent regresses, retrain or revert that one component and leave the others alone. Each change stays small and easy to roll back.

Pitfalls that keep showing up

Swapping the whole agent at once. The open-ended calls fail and take the narrow wins down with them.
Quoting someone else's percentage. The 40% and the 70% came from different agents. Yours needs its own count.
Training on raw logs. Logs carry PII and PHI. Strip them before a call becomes a training example.
Picking by leaderboard. The paper lists generalist benchmarks as one of the barriers to adoption. Your own eval set decides.
Skipping end-to-end tests. Each agent can pass its own suite while a handoff between them drops requests.

The model lineup, a LoRA sketch and the tracing setup are in this guide to small language models for agentic AI.

Count the calls first. A small model earns its place only on a job you have measured, scored and can roll back.

Coder Legion - Recent posts tagged small-language-models