Every "run a local model instead of Claude Code" guide opens the same way: a wall of SWE-bench scores, a triumphant "we're within five points of the frontier," and a shopping list of GPUs. I've read a dozen of them and run most of the setups. They're not wrong, exactly. They're measuring the thing that's easy to measure instead of the thing that decides whether you can actually close your Claude Code tab.
Here's the version nobody puts in the headline, after pointing Claude Code and Codex at local models for real work:
A local LLM writing a correct function is a solved problem. A local LLM being a reliable agent — emitting a clean tool call, ten times in a row, across a multi-file edit, without drifting from the schema — is not. And the second one is the entire job.
So let's do this properly. We'll go through the model families that actually matter in late 2026, what each is genuinely for, the hardware reality, the setup (it's 60 seconds now), the failure modes nobody screenshots, and a cost model so you know whether this saves you a cent. With charts, because a table of numbers you have to squint at is exactly the problem.
First: why this suddenly works at all
Two things changed in 2026, and neither is "the models got louder on Twitter."
1. Mixture-of-Experts broke the hardware wall. A dense 70B model activates all 70B parameters for every token. A MoE model has a huge total parameter count but only fires a sliver per token — Qwen's 35B-A3B activates ~3B. VRAM is set by the file size you load; speed is set by the tiny active slice. So a model that used to need a server now runs on a gaming GPU, and quantized, on a laptop.
2. The agents learned to speak local. This is the part the 2025 posts missed. In January 2026, Ollama shipped native support for the Anthropic Messages API, and LM Studio and llama.cpp followed. That's the unlock: Claude Code no longer needs a proxy shim to talk to localhost. You point one environment variable at it and it works. The plumbing stopped being the hard part — which means the conversation finally has to move past "can it code."
The number everyone quotes — and why it misleads you
SWE-bench Verified is the stat in every headline, and the reported 2026 figures genuinely crowd the frontier.
Read that chart and the story looks finished: a model on a single 24GB card lands at ~77, within a rounding error of the cloud frontier. But here's the catch, and it's the whole article:
SWE-bench measures "given a bug, produce a correct patch." That's a writing task. It says almost nothing about the four things that actually break when the model is driving your editor.
-
Tool-call validity — can it emit a well-formed
edit_file/run_commandcall the harness can parse, reliably, not usually? - Long-horizon consistency — does call #11 still remember the plan from call #2?
- Context under load — does quality hold at 40K tokens of real repo, or collapse long before the advertised window?
- Instruction-following — when you say "don't touch the tests," does it not touch the tests?
None of those four are on the leaderboard. All four are why your local setup feels different from the demo.
⚠️ Treat every number in this post as vendor-reported. Different guides quote the same model at wildly different scores because they quietly swap SWE-bench Verified for Pro or Terminal-Bench and never say so. Verify on the official SWE-bench leaderboard before you buy a GPU on the strength of a percentage.
The 7 families worth knowing (and what each is actually for)
Think in families, not single checkpoints — each ships in several sizes, and the right pick is the one that fits your card and your job.
1. Qwen 3.x — the backbone of local coding
If you run one family, it's this one. Qwen is the open-weight leader on both coding and tool-calling, which is the combination that matters.
- Qwen 3.6 27B (dense) — the default self-hosted coder. ~77% SWE-bench Verified, ~22GB quantized, fits a single 24GB GPU or an M-series Mac. Reliable tool caller.
- Qwen 3.5 35B-A3B (MoE) — 35B total, ~3B active. Runs shockingly light (8GB VRAM at Q4) because only the active slice costs compute. The "frontier on a laptop" headliner.
- Qwen3-Coder (30B-A3B / 480B-A35B / Next-80B-A3B) — code-specialist line. The 30B is a great daily driver; the 480B is the most capable open coder if you have 64GB+ unified memory or multi-GPU.
Best for: your general coding agent, tool-use workflows, and anyone who wants one model that does 80% of the job well. Weak spot: nothing glaring — it's the safe default.
2. GLM-4.x — the long-session agent on a budget
GLM punches above its size on long, tool-heavy agent runs, and the Flash variant is the lightest thing that's genuinely usable.
- GLM-4.7-Flash — ~59% SWE-bench, runs on 5–8GB laptops. The "I only have an integrated-ish GPU" option that isn't a toy.
- GLM-4.6 / 4.x (MIT license) — recommended for long, multi-step agent sessions where consistency over many calls matters more than raw single-shot score.
Best for: laptop-tier hardware, long agentic sessions, permissive-license shops. Weak spot: lower ceiling on hard single-shot coding than Qwen 27B.
3. DeepSeek V3.2 / V4-Pro — the highest open ceiling
DeepSeek posts the top open result — ~80.6% SWE-bench Verified for V4-Pro — but it's "local" only if your desk has a server in it (80GB+).
Best for: pure, hard code tasks where you have server-class hardware or a self-hosted box for a team. Weak spot: not a consumer-GPU model; this is the escalation tier, not the daily driver.
4. Devstral Small 2 — the permissive-license workhorse
~68% SWE-bench Verified, Apache-2.0, fits 12–16GB. The best "I need a clean license and it runs on a mid-range card" option, and a solid agentic coder for its size.
Best for: commercial use with license certainty, mid-range GPUs, agent sub-tasks. Weak spot: multi-file reasoning tops out below the 27B tier.
5. Kimi K2.x — long-horizon frontier (if you have the rack)
Kimi targets long-horizon agentic work and scores well on the harder Pro benchmark, but needs heavy quantization or server hardware to run locally.
Best for: long autonomous runs on serious hardware. Weak spot: not a realistic single-consumer-GPU pick; benchmark figures are on Pro, so don't compare them head-to-head with the Verified numbers above.
6. GPT-OSS (20B / 120B) — OpenAI's open weights, Codex-native
OpenAI's open-weight line slots naturally into Codex CLI via the --oss flag. The 20B runs on consumer hardware; the 120B is server-tier.
Best for: teams already standardized on Codex who want a local fallback with zero tooling friction. Weak spot: Qwen/GLM generally edge it on open tool-calling reliability.
7. Gemma 4 — the lightweight generalist
Gemma 4 (12B / 27B) is the lighter-hardware generalist — great for a 16GB machine when you want a capable all-rounder rather than a code specialist.
Best for: laptops, mixed coding+writing use, low-VRAM boxes. Weak spot: a generalist, not a coding specialist — it'll trail the coder-tuned models on hard SWE tasks.
(Honourable mention: **MiniMax M2.x, trained agent-first and priced aggressively — worth watching if throughput-per-dollar is your constraint.)
The efficiency knee: more hardware ≠ better
Here's the chart that should actually drive your GPU decision — performance against the VRAM it costs to get there.
The curve climbs steeply up to ~24GB and then flattens — the 480B coder even dips below the 27B on Verified while costing 3× the memory. The lesson: a single 24GB consumer GPU running Qwen 3.6 27B is the sweet spot for self-hosting. Everything past it is diminishing returns unless you specifically need the server-tier ceiling.
The wall the leaderboards hide: tool calling
This is the finding that reorganized how I pick a local model. Tool calling is a separate trained skill from writing code. A model can produce a beautiful function and still fail the one thing an agent needs — emitting a valid tool call on the first try, every try.
What the evidence converges on:
- Reliability falls off a cliff below ~4B parameters. Sub-7B models emit malformed tool calls no matter which agent wraps them. The harness can't save a model that can't format the call.
- Native tool-use training beats bolted-on grammar constraints. Models trained with a tool template behave; models with constrained decoding stapled on top drift.
- The model matters more than the harness. Swapping Claude Code → OpenCode → Codex won't fix a model that garbles JSON.
- Multi-step chains are where it dies. One tool call is fine. Five in a row, each depending on the last, is where smaller local models start hallucinating calls — inventing a function that doesn't exist, or re-calling one with drifted arguments — and the agent spirals into a retry loop.
The practical cut: for agentic work, want a 14B–32B model from a family with native tool training. In late 2026 that overwhelmingly means Qwen, with GLM-4.x and Devstral Small 2 as credible alternates. Anything under ~7B is a chat model cosplaying as an agent.
The setup is genuinely 60 seconds
Because the agents speak local now, this is anticlimactic — which is the point.
Claude Code → local model (via Ollama):
# 1. pull a capable coder
ollama pull qwen3-coder:30b
# 2. point Claude Code at localhost + a throwaway token
export ANTHROPIC_BASE_URL="http://localhost:11434"
export ANTHROPIC_AUTH_TOKEN="local-dummy"
# 3. run it
claude
Codex CLI → local model: Codex talks to any OpenAI-compatible endpoint. With Ollama, the --oss flag points it at the local server (:11434); LM Studio serves on :1234/v1. Pin providers/models in config.toml.
Harness pick: Ollama is the terminal-first option (it even has one-command launchers for Claude Code / Codex / OpenCode); LM Studio is the GUI. Either works — remember, the model, not the harness, decides your tool-call reliability.
Two gotchas that each cost me an evening:
- Set the context window explicitly — at least 32K, ideally 64K+. Defaults are often tiny and a coding agent starves instantly. But quality degrades well before the advertised limit, and the KV cache eats the VRAM you wanted for weights. Budget for both.
-
To go back to the cloud:
unset ANTHROPIC_BASE_URL(or remove it fromsettings.json). Obvious until you forget and wonder why "Claude" got dumber.
Hardware tier guide
| Your hardware | Pick | Reality |
|---|---|---|
| 8GB laptop / small GPU | GLM-4.7-Flash, Gemma 4 12B | Usable for single-file help; expect retries on agentic work |
| 12–16GB GPU | Devstral Small 2, Qwen3-Coder 14B-class | Solid daily coder; clean Apache-2.0 option |
| 24GB GPU (the sweet spot) | Qwen 3.6 27B | The one to buy for; reliable tool caller, ~77% SWE-bench |
| 64GB+ unified / multi-GPU | Qwen3-Coder-480B, GPT-OSS 120B | Specialist ceiling; diminishing returns |
| Server 80GB+ | DeepSeek V4-Pro, Kimi K2.x | Team box / escalation tier, not a desk machine |
Where it actually breaks (the honest part)
Run this for a week on real work and the pattern is consistent: a good local coder gives you roughly 70% of cloud Claude for routine single-file tasks, and the gap widens fast as the work gets agentic. The failure modes, in order of how often they bite:
- Schema drift → retry loops. The model emits a slightly malformed edit, the harness rejects it, it drifts differently on the retry. On a hard step this eats a dozen round-trips.
- Multi-file reasoning falls apart. "Rename this across the module and fix the imports" is where cloud is boring and reliable and a local 30B gets lost by file three.
- Latency you feel. A chain cloud finishes in seconds can take minutes locally — not because tokens/sec is bad, but because every retry is another full generation.
- The context cliff is early and sharp. The model gets subtly worse around half the stated window, and "subtly worse at following the plan" is exactly what you can't afford in an agent.
None of this makes local useless. It makes local a different tool, not a cheaper clone.
So does it actually save money?
This is the real reason people try, so let's do the math instead of waving at "free." Local isn't free — it's prepaid. You trade a metered API bill for fixed hardware + electricity + time spent babysitting retries.
- A used 24GB GPU that runs Qwen 3.6 27B comfortably: a few hundred dollars, one time.
- Electricity under sustained load: real but small — single-digit dollars a month for an individual.
- Against that: one heavy Claude Code user can burn hundreds a month, and teams running agent swarms post eye-watering bills (the "$4,660 in a month" screenshots are a genre now).
Honest conclusion: for an individual doing casual coding, local rarely pays back versus just using the cloud carefully — the hardware and fiddling cost more than you'll save. Where it flips hard is high-volume, repetitive, or privacy-bound workloads — especially agent swarms, where you route the cheap 80% of calls to a local model and spend frontier tokens only on the 20% that need them. That routing — not "replace Claude Code" — is the actual 2026 win.
The framework that makes the call
Stop asking "which model is best." Ask "which job is this."
Reach for local when the work is:
- Private or offline — code that can't leave the machine.
- High-volume, cheap-per-task — the thousand little classify/summarize/boilerplate sub-calls in a swarm.
- Narrow and single-file — a function, a test, a regex, a migration.
- A credit-saver — route the easy 80% local, escalate the hard 20%.
Keep cloud Claude/Codex for:
- Multi-file, long-horizon changes where consistency over many steps is the value.
- Large-context work where you need the whole window to actually work.
- "Right the first time, fast" — anything where retry loops cost more than tokens.
The one-liner I've landed on: local is becoming the default, and the frontier model is the escalation path. That's the opposite of how most teams run it today — and it's where the money is.
The real answer
Can a local LLM replace Claude Code? For being your agent on hard, multi-step, large-context work — not yet, and the thing holding it back isn't the SWE-bench score, it's tool-calling reliability over long sessions. For being the cheap, private, always-available workhorse for the 80% of tasks that don't need a frontier model — absolutely, today, on a single 24GB GPU, and it's getting better monthly.
The mistake is treating it as a binary. The people getting real value aren't "replacing" anything. They run two models and know which job goes where.
If this was useful, follow me — I write about AI agents, local models, and the unglamorous engineering around them, and I'm testing more of these setups as the models ship.
Your turn: what's the actual line for you — the specific task where you reach for the cloud and refuse to trust a local model? I'm collecting the real break points, not the benchmark ones. 👇
Sources & verify it yourself
Numbers in this post are vendor-reported and move monthly — don't buy hardware on the strength of a blog percentage. Check them at the primary sources:
- SWE-bench leaderboard — the official coding benchmark; confirm Verified vs Pro before comparing.
- Berkeley Function-Calling Leaderboard — the one that actually measures tool-calling, the metric this whole post argues matters most.
- Ollama — local serving + the native Anthropic/OpenAI-compatible endpoints that make the Claude Code / Codex setup a one-liner.
- Model cards for the real sizes, licenses and quant builds: Qwen, DeepSeek, Mistral / Devstral, GLM (Z.ai).
The cost and tool-calling-tier charts are illustrative, drawn to show the shape of the trade-off — plug in your own API bill and GPU price for the numbers that apply to you.
Top comments (122)
The "within five points of the frontier" framing is the easy-to-measure trap you named. The part nobody puts in the headline: it's rarely local-vs-frontier as a binary.
The real lever is per-step routing. The ~80% of calls that are boilerplate, classification, or formatting don't need the frontier at all — a small or local model handles them at a fraction of the cost with no visible quality drop. So "replace Claude Code with a local model" is the wrong unit. The question that actually closes the tab is: which steps need the frontier, and which can a cheap model own? Benchmark the mix, not the model.
the mix could also be 2 open models: one for boilerplate and one for reasoning - a lot cheaper than Claude
The 2-model split is the most pragmatic instantiation of this — boilerplate vs reasoning as two tiers instead of a spectrum.
The catch I keep hitting: the router (deciding which of the two a step is) has to be cheaper than the saving it unlocks, or you've just moved the cost. Do you keep that decision in a tiny local model, or rule-based on step metadata?
That constraint kills most two-tier setups. I keep the decision rule-based on step metadata wherever it holds — file count, diff size, failing-test present, "edit" vs "plan" — and fall back to a tiny local classifier only for the ambiguous tail. A frontier call to decide the tier is self-defeating: you've paid the strong model to avoid paying the strong model. The honest test is net cost including the router; if it doesn't beat always-strong after routing overhead, the split isn't earning its keep.
Rule-based-on-metadata-first is the part most teams skip because the classifier feels cleaner — but the classifier is a tax on every step, and most steps the metadata already decides. The failure mode I keep hitting: when the step metadata is itself unreliable — e.g., the agent just generated code, so "files touched" exists but says nothing about intent — the rule misfires, and the tiny model earns its place precisely there, on the ambiguous tail you described. That's the real budget argument for keeping the classifier: not for the average step, but as the only thing that catches the step the rules can't see. Where does your rule set stop and the classifier take over — fixed heuristics, or a confidence threshold on the metadata?
You've put it better than I did: the classifier isn't paying for the average step, it's paying for the step the rules can't see — and "the agent just generated code, so files-touched exists but says nothing about intent" is the exact blind spot.
So the handoff isn't a fixed heuristic, it's a confidence threshold on the metadata itself, not on the task: route by rule only when the deciding fields are present and trustworthy for that step type, and fall through to the classifier the moment they're missing, stale, or (your case) structurally uninformative. "Files touched after a codegen step" is a field that exists but scores low on informativeness, so it should fail the gate and hit the model.
The trap is treating "metadata present" as "metadata decisive" — a null-vs-unreliable distinction. Where I'm still unsure: whether informativeness is better learned per field or hand-flagged per step type. How are you detecting the unreliable-but-present case — a per-field confidence, or step-type rules that know which fields lie?
The null-vs-unreliable distinction is the real sharpening here — "metadata present" and "metadata decisive" are different claims, and collapsing them is exactly how a rule set quietly routes on a field that exists but means nothing. Your gate (route by rule only when the deciding fields are present and trustworthy for that step type, else fall to the classifier) is the right shape.
On detection, I lean step-type rules that know which fields lie, mostly because per-field confidence has to be bootstrapped from labeled mistakes and almost nobody has that dataset. The nasty case is the field that's reliable for 90% of steps but lies on the 10% — a flat confidence threshold tends to miss that unless the step type already flags it. So I'd hand-flag per step type rather than learn informativeness from logs, at least until there's enough misroute history to train on. How are you seeing it: hand-flagging, or actually learning informativeness, and has the 90/10 field ever burned you in practice?
Hand-flagging per step type, for your exact reason: per-field informativeness has to be bootstrapped from labeled misroutes, and almost nobody has that dataset on day one — you'd be learning confidence from the mistakes you're trying to prevent.
And yes — the 90/10 field is the one that's burned me. "Files touched" is reliable for most steps and lies precisely after a codegen step, and a flat confidence threshold sails past it because it's trustworthy 90% of the time. That's what convinced me the step type carries the flag, not a global per-field score — the field isn't unreliable, it's unreliable in a context, and only the step type knows the context.
The split that actually pencils out — you're paying frontier prices for autocomplete-grade work most of the loop. Cheap local model for boilerplate/edits, a stronger one (local or API) only for the planning turns, gets most of the saving without the drift, since the expensive turns are the rare ones. The hard part is the routing logic: deciding per-turn which tier you're in.
"Deciding per-turn which tier you're in" is the whole ballgame — and where most setups quietly fall back to frontier, because the router isn't stateful. If the tier decision loses track of what previous steps did, it over-assigns to the strong model.
How are you tracking state across turns for that decision — replaying the transcript, or a lighter side-channel?
Side-channel, every time — replaying the transcript to pick the tier re-reads the exact context you were trying not to pay for. What works: a tiny running state of what the trajectory has done — files touched, tools called, whether the last step left an unresolved error — not the text itself. The tier keys off that ("mid-refactor across 3 files with a failing test" → reasoning tier; "single-file edit, tests green" → cheap tier). The moment the router needs the full transcript to decide, it costs as much as the model it's gating.
Side-channel all the way — replaying the transcript to pick the tier is the one option that pays for the context you were trying not to pay for, twice. The running-state shape you describe (files touched, tools called, whether the last step left an unresolved error) is cheap to maintain and exactly the thing a transcript re-read throws away.
The "mid-refactor across 3 files with a failing test → reasoning tier" example is a good illustration of state that's trivial to track and expensive to lose. The wrinkle I'd push on is staleness: that state can go stale mid-trajectory, and a router keyed off a value that quietly stopped being true is just a faster way to misroute. So when the test flips green three steps later, do you recompute the tier from the updated state, or trust the running state you already had?
Recompute — staleness is the whole risk. The running state is cheap to maintain precisely because it's cheap to update, so the discipline is: anything that changes a routing-relevant fact (test flips green, error clears, refactor closes) is a state-change event that triggers a re-decision, not a value carried forward. A router keyed off "mid-refactor, failing test" that doesn't notice the test went green is just a faster way to misroute — so the running state has to carry freshness itself, not only the last value.
You've crystallized the thesis better than I wrote it.
That's the keeper line, and it's the correction to my own framing — I spent 2,700 words on "which model" when the unit that actually closes the tab is "which steps." The ~80% that's boilerplate / classification / formatting doesn't need the frontier and shows no visible quality drop on a cheap or local model; the frontier only earns its cost on the 20% that's genuinely hard, multi-file, or long-horizon.
The part I don't have a clean answer for — and where I'd love yours — is how you draw the routing line without a frontier grader in the loop. If a cheap model decides "this step is easy, I've got it," you've reintroduced the frontier's judgment as the thing that routes. Do you route by static heuristics (tool type, diff size, files touched), by a small classifier, or by confidence/escalate-on-failure? Because "benchmark the mix" needs a way to assign steps to tiers before you can benchmark it, and that assignment is where I keep getting stuck.
Appreciate that — and the 2,700 words on "which model" is exactly the trap; the reframe only clicks once you've watched the bill.
The one caveat I'd add to "benchmark the mix": the routing decision itself has to be nearly free. If you call a frontier model to decide which tier a step belongs to, you've re-introduced the cost you were avoiding — the router has to run on the cheap tier or it eats the savings. Where did you land on keeping that decision local?
Local and boring, on purpose — mostly deterministic rules over step metadata (edit vs plan, files touched, test state), with a small local model only for the ambiguous tail, never a frontier call. That's the real lesson of the piece: once you've watched the bill, the interesting number stops being which model scores higher and becomes total cost including the thing that decides. A clever router that itself runs on the expensive tier just launders the cost into a line you're not looking at.
That's the crux exactly — a router running on the expensive tier just relocates the cost to a line nobody's watching, and "launders" is the right word for it. Keeping the decision on deterministic rules over step metadata, with the local model only catching the ambiguous tail, is the only shape that survives contact with a real bill.
The thing I'd want to measure is the rules path's own overhead. Once it's boring and local the cost is presumably rounding error, but have you actually timed it, or did it get cheap enough that you stopped caring? And the failure I'd watch for: when a step type has no usable metadata at all, does it fall through to the small local model, or default to a tier? That default decides whether a silent gap costs you a frontier call or a wrong route — so which way did you land on it?
Both honest answers:
The four failure modes are exactly what a sealed tool-call journal catches, and the receipts angle is why the gap between the benchmark and the babysitting exists at all:
SWE-benchmeasures the model in isolation, the babysitting happens when the model is in a loop — and the loop is where the evidence layer lives. Each tool call can be sealed (attempt, transport witness, label), and theq^karithmetic tells you the reliability before you sit down: a per-call validity of 0.99 over a 10-call chain is 0.904 — the missing 9.6% is the babysitting, quantified.The four items on your list map to four different evidence gaps. Tool-call validity is the schema enforcement problem — solvable with a parser gate. Long-horizon consistency is the context-rot problem — the
Makefile/init.dfix. Context under load is the staleness problem — the execution-horizon equation. Instruction-following is the claims-versus-observations problem: the model said it didn't touch the tests, and only a sealed diff proves whether that claim was true. Four failure modes, four different instruments — which is whySWE-benchcan't carry all four on one leaderboard.The local angle adds one more: on a 3B model, the tool-call journal is smaller than the context window delta of a single edit. The receipts are the cheapest component in the stack — on constrained hardware, they're the one control that earns its place by being free.
That's the line the whole thread was circling. SWE-bench scores the model; the receipts score the run — and babysitting is a property of the run, not the model. The 4→4 mapping is clean and I'm keeping it:
And the point I hadn't considered is the best one: on a 3B, the tool-call journal is smaller than the context-window delta of a single edit. That inverts the usual "verification is overhead" assumption — on constrained hardware the evidence layer is the control that's basically free, so it's the first thing that should earn its place, not the last.
One question on the "transport witness" in your sealed journal: is it observing the actual syscall/network egress, or the harness's record of the call? i.e. how out-of-band is the witness from the thing it's witnessing — because that's the difference between "the agent said it sent X" and "X actually left."
Three things at once, because the thread has earned a consolidated answer. The transport witness: the hook process observes the harness's call, not the syscall — if the harness lies about what it's sending, the hook catches the lie. But if the harness itself is compromised, the hook is in the same process tree — the shared-dependency test lands there. The true out-of-band witness is the OS: eBPF on the egress, the filesystem's audit log. That's the next descent, and the floor is a property of the kernel, not of the architecture. On AST normalization: canonicalize identifiers and sort independent nodes first, then diff the canonical forms — only semantically meaningful divergence survives, and the threshold stops being a tuning problem because the canonical form removes the benign variance by construction. And on the metric that isn't a number: "would I use this tool again tomorrow" is the integral of every retry, every correction, every time it surprised you well or badly — the q^k arithmetic and the hazard curves are just trying to approximate that gut call. The day it stops teaching you anything and you're purely supervising is the day it isn't pulling its weight — and that line is the one no benchmark will ever carry.
This is the line I'll be stealing. You've named the thing I was circling: the witness has to live below the layer that can lie. A hook in the harness's process tree shares its fate — compromise the harness and you compromise its confession in the same breath. eBPF on egress and the kernel audit log don't share that fate, because the thing being observed can't forge the observation without owning ring 0. The floor is the kernel because that's the lowest layer still cheaper to trust than to subvert.
Which reframes the whole exercise: we're not building trust, we're pushing the witness down until the cost of forging it exceeds the payoff. Every descent buys you a harder-to-fake floor.
On AST normalization — yes, and I think canonicalize-then-diff quietly dissolves the problem I was treating as hard:
The threshold was never a tuning problem. It was a representation problem — I was trying to set a tolerance on a signal that still contained benign variance, when the fix is to remove the variance by construction so the tolerance can be zero.
And the metric: "would I use this tomorrow" as the integral of every retry and surprise is exactly right — the q^k arithmetic is just a cheap closed-form approximation of a gut call that's already integrating. The part that stays with me is your closing line: the day it stops teaching you and you're purely supervising is the day it stops pulling its weight. That's not a performance threshold, it's a relationship one.
So here's where I'd push next: if the kernel is the floor for the transport witness, what's the floor for the semantic one? AST canonicalization witnesses that the code didn't change meaning — but what witnesses that the meaning was correct? Is there an out-of-band semantic oracle, or does that floor simply not exist, and that's precisely the gap where you still have to be in the loop?
The semantic floor exists — and it's the cheapest layer in the stack, not the most expensive: the spec itself, sealed when the human wrote the expectation. A named expectation — "the 2024 pin is a business rule, not a default" as a test name — is a human judgment captured at write time and checked mechanically at every run after. The semantic oracle isn't in the loop at audit time; it was in the loop at spec time, and the
spec_hashcarries that judgment forward. The gap you named is real — nothing witnesses that the meaning was correct without an oracle — but the oracle ran once, at spec time, and the receipt binds the spec to the verdict so the judgment is never silently replaced. What the semantic layer needs isn't a kernel audit; it's a human who wrote the expectation and a hash that proves the expectation hasn't drifted. That's the floor: a human judgment, sealed once, carried forward by every machine that runs after.This is the answer, and it dissolves the question I was asking rather than answering it. I was hunting for a semantic oracle at audit time — an out-of-band witness, kernel-level, that could certify meaning the way eBPF certifies egress. There isn't one, and your point is that there doesn't need to be, because the semantic oracle is a human, and it already ran — at spec time.
So the two floors aren't the same kind of thing, and I was wrong to expect symmetry:
spec_hashis the whole security argument: it doesn't prove the spec was right, it proves the spec you're being judged against is the one a human actually wrote.Which relocates the entire trust question to one place: the integrity of the seal. The dangerous failure is no longer "the audit missed a bad meaning" — it's "someone rewrote the expectation and the hash didn't notice," i.e. a silent replacement of the judgment. The receipt binding spec→verdict is exactly what closes that, and it's the cheapest layer in the stack precisely because it's a write-time artifact, not a runtime one.
So the honest shape is: kernel floor for did it do X, sealed-spec floor for was X what we meant — and the only thing left to defend is the seal itself. Last descent: what witnesses the seal? A signed, append-only spec history where rewriting an expectation is a visible event and not an in-place edit — or is there something cheaper than that that I'm still over-engineering?
A signed, append-only spec history is not over-engineering — it's the write-time equivalent of the transport attempt. Same pattern: the expectation sealed before it's judged, the rewrite visible as a diff event rather than an in-place edit. The cheap version is git: the spec file in the repo, the hash in the receipt, the history in the log. The expensive version is the anchored journal — but the floor is the same: the judgment was sealed at spec time and the bytes haven't changed since. What the semantic layer needs isn't a kernel witness; it's the discipline to never edit the expectation in place — and the git history to prove it.
That's the landing, and it's the right asymmetry: the transport floor needed a witness below the layer that can lie, but the semantic floor needs a witness before the moment of judgment — and "before" is cheap in a way "below" never was. Git already gives it to us for free:
The anchored journal is the expensive version, but you're right that it's the same floor — the only thing it buys over git is defending against someone who can rewrite history and force-push, i.e. it raises the cost of forging the past. For most teams the git log is already past the point where forging it is cheaper than just doing the work honestly, which is the only bar that ever matters.
So the stack, stated once, end to end:
The discipline is the mechanism — "never edit the expectation in place" isn't a guideline, it's the thing that makes the hash worth binding. An in-place edit doesn't break the hash; it just quietly changes what the hash was ever protecting. Which means the whole semantic floor rests on one social rule enforced by one tool, and that's genuinely the cheapest layer in the stack — exactly as you said four comments ago. I think the thread actually converged. Rare thing.
That's the landing, and it's the right asymmetry: the transport floor needed a witness below the layer that can lie, but the semantic floor needs a witness before the moment of judgment — and "before" is cheap in a way "below" never was. Git already gives it to us for free:
The anchored journal is the expensive version, but you're right that it's the same floor — the only thing it buys over git is defending against someone who can rewrite history and force-push, i.e. it raises the cost of forging the past. For most teams the git log is already past the point where forging it is cheaper than just doing the work honestly, which is the only bar that ever matters.
So the stack, stated once, end to end:
The discipline is the mechanism — "never edit the expectation in place" isn't a guideline, it's the thing that makes the hash worth binding. An in-place edit doesn't break the hash; it just quietly changes what the hash was ever protecting. Which means the whole semantic floor rests on one social rule enforced by one tool, and that's genuinely the cheapest layer in the stack — exactly as you said four comments ago. I think the thread actually converged. Rare thing.
This is exactly what I have been seeing. Everyone benchmarks on SWE-bench but nobody benchmarks on "can this thing actually ship a feature without me babysitting it for 3 hours." The local vs cloud debate misses the point. It is about total time to done, not tokens per second. Actually a great perspective.
That's the metric the whole industry is dodging, because it's the hard one to measure. Tokens/sec and SWE-bench are easy to put on a chart; "time to shipped, including the babysitting" requires you to actually run the thing on real work and count the hours you spent un-breaking it. A model that generates at 80 tok/s but needs three correction loops is slower to done than one at 30 tok/s that lands it first try — and the leaderboards can't see that at all.
The babysitting tax is the real cost, and it's almost entirely a function of how often you have to step in, not how fast the tokens come out. Out of curiosity — where's your personal line? The specific task where the babysitting got bad enough that you went back to the cloud and didn't look back?
yeah honestly I hate babysitting. That's why I give the most detailed prompt ever covering all steps then ask the model to ask any questions it has and no assumptions.. then give permissions to the relevant folders and then handsfree this usually works for me and I dont have to babysit around most of the time. Some minor manual corrections at the end.
That workflow is doing something smarter than it looks, and it's worth naming why it works:
That one step is the whole trick. Most babysitting isn't the model writing bad code — it's the model silently guessing at something you left ambiguous, then confidently building on the wrong guess. By the time you see it, it's three edits deep and looks fine. Forcing it to surface the assumptions as questions up front converts a wrong guess (which you'd only catch late, by reading the output carefully) into a cheap question you answer in one line before any code exists.
In other words: you're front-loading the ambiguity so it never turns into the expensive kind of mistake — the one that's valid-looking but wrong. The detailed spec + scoped permissions handle the rest by shrinking what it can get creative about.
The "minor manual corrections at the end" is the honest tell that it's working — you've pushed the human step to the cheapest possible point (review once, at the end) instead of the most expensive (interrupt mid-run, repeatedly).
Does the "ask questions first" step hold up on the local models too, or is that where it slips? That instruction-following — "stop and ask, don't assume" — is exactly the kind of thing I'd expect a smaller local model to quietly ignore and just barrel ahead.
The measurement problem is real. I have tried tracking it and it is messy. Half the time I am not even sure if I am babysitting the tool or learning from it. Maybe the metric is not a number. Maybe it is just: did I ship something I am proud of, and would I use this tool again tomorrow. That is not a chart but it is honest.
Honestly, that might be the truest metric in the whole thread. All the q^k arithmetic and hazard curves are just trying to approximate that gut call — "would I reach for it again tomorrow" is the integral of every retry, every correction, every time it surprised you well or badly.
And your "babysitting vs learning" blur is a real signal, not noise. Early on it's learning — you're building the mental model of what it can and can't do. It only becomes babysitting once it's stopped improving your understanding and you're just catching the same failures on repeat. So maybe that's the actual line: the day it stops teaching you anything and you're purely supervising is the day it isn't pulling its weight.
What tipped a tool, for you, from "still learning its edges" into "just babysitting it"?
+1. open models are good enough for most of this now, the weak spot is serving them consistently. that’s the problem Keln.ai solves (I’m one of the founders)
For me it was the third time I caught the same bug. The first two times I learned something new about how the tool thinks. The third time I was just frustrated. That was the line. After that every session felt like supervision instead of collaboration. I went back to the cloud the same day.
Two models is an interesting angle. I have been thinking about this too. The hard part is the routing. How do you decide which model gets which task without adding so much overhead that you lose the cost savings. If the routing is cheap enough this could actually work. Curious if you have tried this in practice or if it is still theoretical. I dont wanna sound demotivating but I have tried some open source projects and created my own routings in free tiers of ai coding tools.. But I dont see any good results worth the hassle.
This is the most precise version of "the metric that isn't a number" I've seen in the thread. The first two catches had information in them — you were still learning the tool's failure modes, which is real value. The third had none; it was pure tax. The line isn't a quality threshold, it's the moment the marginal lesson hit zero and all that was left was supervision.
And "I went back to the cloud the same day" is the honest data point every benchmark misses: you didn't churn because the local model crossed some eval score, you churned because the relationship stopped teaching you anything. That's the integral of every retry — and it tipped the instant the retries stopped being educational. A leaderboard can't see that number, but it's the only one that decided your actual behavior. Thanks for putting a clean edge on it.
Not demotivating at all — it's the most important objection, and your experience matches mine: most routing overhead eats the savings it's supposed to create. The reason, I think, is that people route on the wrong variable. The common attempts route on predicted difficulty ("is this a hard task? → cloud"), and estimating difficulty up front is itself a model call — so you pay for a judgment that's barely cheaper than just doing the task on the big model. That's the hassle with no payoff you hit.
The framing from elsewhere in this thread that I think actually survives contact: route on detectability, not difficulty. I.e. send a task local only when you have a cheap, deterministic acceptance check for its output — a test that runs, a schema that validates, a diff against a known-good baseline. If the local model's output passes a check that costs ~nothing, you keep it; if it fails or there's no check, escalate. The routing decision is then a cheap check you already have, not a second model estimating difficulty. That's the only version where the overhead is genuinely near-zero.
Honest caveat: it only works for the slice of tasks that have a cheap verifier (boilerplate, well-specified edits, anything test-covered). For the open-ended "think about the architecture" work, there's no cheap check, so it routes to cloud by default — which is fine, because that's the 20% you wanted on the big model anyway. Did your routing attempts try to be general (route everything)? That's usually where it falls apart — the win is only in the verifiable slice, not across the board.
rudratosh, this thread has evolved into the most rigorous discussion on agent evaluation I've seen all year. the shift from 'difficulty-based routing' to 'detectability-based routing' is a genuine paradigm shift.
to answer your final question on measuring the bucket-3 (valid-but-wrong) base rate without an expensive human or frontier oracle: you can shrink bucket 3 by engineering cheap, structural proxies for semantic correctness.
self-consistency divergence as a canary: run the local model n=3 times with temperature > 0. if the outputs pass the linter/tests but produce wildly divergent abstract syntax trees (asts), the 'valid-but-wrong' probability is high. true correctness usually converges structurally; hallucinated 'valid' code often takes bizarre structural paths just to satisfy the test suite.
contract-based instrumentation (design by contract): the reason bucket 3 exists is that test suites are incomplete. if you inject lightweight runtime assertions (pre/post-conditions) into the target codebase, a 'valid-but-wrong' edit that breaks a hidden invariant will trip a contract assertion. this mechanically moves the failure from bucket 3 (undetectable) to bucket 2 (detectable logic failure) without needing a frontier oracle.
differential execution: run the edit against a shadow copy of the codebase. if the local model's edit changes the behavioral output of a non-test, deterministic pure function, flag it as a structural anomaly.
you don't need a frontier model to grade the output if you can mathematically constrain the shape of the correctness. build the structural tripwires, and bucket 3 shrinks to almost nothing.
incredible work pushing this taxonomy forward. 🐯📊
This answers the question I was genuinely stuck on — and the reframe is the whole win:
Right. You don't need to see the valid-but-wrong edit; you need to make it trip something. All three proxies are cheap and oracle-free:
nextra local generations, not a frontier call.Stack them and the residual bucket-3 shrinks to "wrong in a way that passes tests and converges structurally and trips no contract and changes no pure-fn behavior" — a much smaller, more honest number than "undetectable."
The one place I'd push: the AST-divergence canary needs a divergence threshold, and separating benign structural variance (formatting, renamed temporaries, reordered independent statements) from signal divergence feels like it could become its own tuning problem — the very thing we're trying to escape. Do you normalize the ASTs first (canonicalize names/ordering) so only semantically-meaningful structural divergence survives, or threshold on a tree-edit distance?
rudratosh, glad the "shrink bucket 3" reframe clicked!
regarding ast self-consistency: the beauty of it is that you don't even need a heavy llm judge for the first pass. you can run the generated code through a lightweight, local parser like
tree-sittern=3 times with temp > 0. if the resulting abstract syntax trees diverge wildly on non-trivial logic paths, you flag it for human review before it ever touches the test suite. it turns a semantic hallucination into a cheap, deterministic structural anomaly.really appreciate you driving this conversation forward. it's pushing the whole community to think beyond simple leaderboard scores and actually measure agent reliability. 🐯📊
Tree-sitter is the perfect call — deterministic, local, no LLM judge in the first pass. That's what makes the AST-divergence canary actually cheap: you've turned "is this semantically wrong" (needs an oracle) into "do these parses disagree structurally" (needs a parser). Semantic hallucination → mechanical structural anomaly, exactly as you said, and flagging before it touches the suite is the right ordering — you catch it while it's still cheap.
Which lands on the one tuning risk I keep circling: you need to separate benign structural variance (renamed temporaries, reordered independent statements, formatting) from signal divergence, or the canary fires on style. The nice thing is tree-sitter hands you the tree, so you can canonicalize first — normalize identifiers, sort commutative/independent nodes — and diff the normalized ASTs, so only semantically-meaningful divergence survives. Is that how you'd do it (normalize → tree-edit distance over the canonical form), or do you keep it rawer and just threshold on gross divergence?
rudratosh, you nailed the exact tuning risk. raw thresholding on gross divergence is a trap—it will fire constantly on benign variance like variable renaming, loop unrolling, or harmless reordering of independent statements.
the only robust path is canonicalize first, then diff.
the pipeline should be: parse with tree-sitter -> strip comments/whitespace -> normalize identifiers (e.g., hash or sequential renaming like
var_1,var_2) -> sort commutative/independent nodes (like independent assignments or commutative math operations) -> then compute a lightweight tree-hash or tree-edit distance on that canonical form.if the normalized tree-hash diverges, that is your canary. it’s cheap, deterministic, and completely ignores stylistic hallucination, catching only actual structural/logic drift before it ever touches the test suite. 🐯🌳
The correct pipeline, and the answer to the false-positive trap — raw divergence fires on renames and reordering, which is 90% of benign LLM churn. Canonical tree-hash catches structural/logic drift only, cheap and deterministic. One caution: "sort commutative/independent nodes" has to prove independence (no shared side effects) or you'll canonicalize away a real reordering bug. But as a pre-test canary, parse→normalize→hash is exactly right. 🌳
Rudratosh, this is exactly the kind of deep-dive feedback that makes building in public worth it. 🐯🛡️
You just mapped out the exact evolution path for KAVLR (our terminal agent leash):
The False-Positive Trap: You are 100% right. Raw divergence fires on benign LLM churn (renames, reordering). The
tree-sitter → strip comments/whitespace → normalize identifiers → sort commutative nodes → tree-hashpipeline is the only way to catch actual structural/logic drift deterministically. We are adding this to our KAVLR v2 roadmap immediately.The Shell-Out Bypass: Your point about the eBPF observer seeing an egress attempt with no valid correlation token is brilliant. Binding the execution to a proxy-signed token instead of app-layer context closes the exact hole where an AI agent tries to bypass the leash via a rogue
curlorwget.I just checked out your
praetorrepo, and the stat speaks for itself: text guards catch 0/629 buried injections, but the action/provenance layer blocks 30/30. Grade the action, not the text. That is a mantra we are adopting.Thank you for sharing this. It’s pushing our architecture to the right level of rigor. 🐯☁️🚀
This made my day — "grade the action, not the text" becoming a mantra for KAVLR is exactly why I write these in public. 🐯
One thing to hold onto in the canonicalize-then-tree-hash pipeline: "sort commutative nodes" has to prove independence (no shared side effects) or it'll quietly canonicalize away a real reordering bug — so treat it as a canary, not a gate.
And the eBPF correlation-token idea was yours — binding egress to a proxy-signed token is the cleanest answer I've seen to the shell-out hole, and it stacks with an action gate: the gate decides whether the action runs, your token proves the egress belonged to an action that was actually authorized. Keep shipping KAVLR in the open — this back-and-forth sharpens both architectures. 🚀
Rudratosh, this reply just made my entire week. 🐯🚀
The "canary, not a gate" distinction for sorting commutative nodes is a massive catch. You are absolutely right: if we blindly sort without proving true independence (no shared side effects), we risk masking a real reordering bug as "benign churn." That is a brilliant architectural guardrail. We will implement it exactly as a pre-test canary, never as a hard gate.
And seeing the eBPF correlation-token + action gate stack validated by you is the ultimate confidence boost. The gate decides "yes/no" to the action, and the token cryptographically proves the egress actually belongs to that authorized action. No shell-out bypasses.
Thank you for pushing the architecture to this level of rigor and for the encouragement to keep shipping KAVLR in the open. This back-and-forth is exactly how elite, production-ready software gets built.
Onward! 🛡️☁️
Onward indeed. 🛡️
One small implementation note for when you wire the canary: don't just flip a boolean — log which commutative reorder would have changed the tree-hash, so a flagged run tells you where to look instead of just "something moved." A canary that says "drift here, in this node" is a debugging lead; one that says "drift: true" gets muted within a week.
Ping me when KAVLR v2 ships — I genuinely want to see the token + action-gate stack running end to end. This was a good one. 🚀
Noted and locked in. 🛡️
Logging the specific node drift instead of a boolean is a pro move. A canary that points to the exact location is a debugging lead; a boolean just becomes noise. We will implement it exactly that way.
Consider this a formal pact: I will personally ping you the second KAVLR v2 with the eBPF token + action-gate stack is live for end-to-end testing.
Thank you for sharpening the blade. We’ll see you at v2. 🚀🐯
Deal. 🤝 I'll be waiting for the v2 ping — the token + action-gate stack running end to end is exactly what I want to see. Go build it. 🚀
Pact accepted. 🤝 Log the node drift, not a boolean — a canary that names the exact location is a debugging lead; a true/false is just noise you'll mute in a week. Ping me when v2's live and I'll put it through its paces. 🐯
@rudratosh Agreed on the tail. The Tokenade leaderboard headline is a geometric mean over 10 runs per task, and the raw runs are published, so the spread can be pulled out.
@paul_irolla That's the detail that matters — a geometric mean over 10 runs hides exactly the tail I was worried about, but if the raw runs are published the variance is recoverable. The honest way to report it: headline the central tendency, ship the spread. I'll pull the per-task run distribution from the raw data rather than argue from the single number. Thanks for pointing to where it lives.
The line you're collecting has a number attached, and it is not a benchmark score — so here is the arithmetic for both halves of your argument.
Your "ten times in a row" bar, in per-call terms. If one call is valid with probability q and the run needs k calls, the run survives with q^k — a lower bound, since your drift observation is exactly the reason a retry does not fully recover:
Read the third column: to finish an 11-call chain 90 % of the time you need q ≥ 0.9905 per call; for 95 % you need 0.9953. That number is not on any of the charts — a tool-call score reports q, almost nothing reports the k at which q^k crosses your acceptance threshold. It is also why "reliable tool caller" and "reliable agent" are different claims: 0.99 and 0.995 are indistinguishable on a single-call chart and differ by a third on a 50-call run (0.61 vs 0.78).
The SWE-bench comparison cannot carry its own headline. For a paired comparison over the 500 Verified problems, the gap a pair of runs can resolve is about 1.96·√d ⁄ 500, where d is the number of problems the two models disagree on. A one-point gap is resolvable only if they disagree on fewer than 7 of the 500. Two models sitting at 76 % and 77 % do not have near-identical failure sets — they disagree on tens of problems — so at d = 30 the comparison resolves nothing below 2.2 points and at d = 50 nothing below 2.8. "Within a rounding error of the frontier" is not a small gap; it is a gap the instrument does not see, and the two readings lead to different actions: a small gap invites you to close it, an unresolvable one says the leaderboard column has stopped being the thing worth tuning. Same arithmetic kills the cross-family comparisons later in the piece — Verified, Pro and Terminal-Bench numbers from different harnesses are not one scale.
So the line, stated as a measurement rather than a task list: per task class, the per-call validity q, and the chain length k at which q^k falls below what you accept. Neither needs a GPU purchase to obtain. Run the same class twice — the rerun is the only control that separates the model from the session, and without it your week of experience is a single sample of your own sampling noise — and count calls, not tasks. The break point you publish then has a denominator attached.
(A small caveat on the second table: d is the quantity nobody publishes. If you have outputs for two local models on the same Verified subset, d is one pass over the per-problem results, and it is worth a line in the post — it converts "looks close" into "close, given these two disagree on N of 500", which is the form a reader can check.)
This is the comment that upgrades the post from "the leaderboard is the wrong chart" to "here is the right denominator." Both halves land, so let me restate them as the corrections they are.
The q^k table is the number my charts don't carry. You're right that this is the whole game:
That's exactly why "reliable tool caller" and "reliable agent" are different claims, and my tiers chart quietly conflates them. A single-call score reports q; nobody reports the k at which q^k crosses your acceptance threshold — and 0.99 vs 0.995, visually identical on any per-call plot, is 0.61 vs 0.78 over 50 calls. The honest metric is the pair: per task class, q, and the k where q^k drops below what you'll accept. No GPU required to measure it.
And you've caught my worst sin: the SWE-bench chart can't carry its own headline. The paired-comparison math is brutal and correct:
So "within a rounding error of the frontier" isn't a small gap — it's a gap the instrument can't see, and I wrote the opposite of the right conclusion from it. Worse, I warned that Verified/Pro/Terminal-Bench aren't one scale and then put them on one axis anyway. That cross-family chart is making a comparison the numbers don't support. Noted, and it's getting a caveat.
The thing I owe the post, straight from your last paragraph: publish
d. If I have per-problem results for two local models on the same Verified subset, d is one pass over them, and it turns "looks close" into "close, given these two disagree on N of 500" — a claim a reader can actually check.One thing I want to push back with you, not against: the q^k survival is a lower bound that's probably optimistic, because drift makes retries correlated — a call that failed from context rot is more likely to fail again, so the independent-Bernoulli model understates the cliff. Do you treat per-call validity as roughly independent in practice, or have you seen the failures cluster badly enough that q^k is the wrong shape entirely?
"Ten times in a row" is where the arithmetic bites, and it's worth putting on your four-item list. If each tool call is valid with probability p, a 10-call multi-file edit goes through clean with p^10: 0.99 gives 90.4%, 0.95 gives 59.9%, 0.90 gives 34.9%. So a model that is "usually" valid at 95% per call fails four edits in ten, and a leaderboard gap of a few SWE-bench points is small next to that.
It also tells you how to measure it. Replay the same 10-call edit task N times per model and count clean runs; 0 failures in 30 runs only bounds the failure rate at about 9.5% (95% upper bound, 1 − 0.05^(1/30)), so you need around 100 runs before a "reliable" claim means much. And count schema drift and valid-but-wrong calls (editing the test file you said not to touch) separately from unparseable calls, because the harness retries the first kind silently and the second kind it doesn't.
Did you log per-call validity in your own runs, or is the "feels different from the demo" judgement from using it day to day?
Straight answer to your closing question, because it's the honest weak point of the piece: no, I didn't log per-call validity. The "feels different from the demo" is day-to-day judgment plus corroborating reports — which is exactly the single uncontrolled sample you and howcani are both calling out. You're right that it needs a denominator, and I shouldn't have let an un-instrumented impression carry a quantitative claim.
Your p^10 math is the number that should've been on the four-item list:
A model that's "usually valid" at 95% per call fails four edits in ten — which dwarfs any few-point SWE-bench gap. And the measurement protocol is the useful part:
1 − 0.05^(1/30)), so ~100 runs before "reliable" means anything. (The rule-of-three bite again — same reason the benign set needed to grow in that other thread 😄.)That taxonomy is going into the rerun. If I'm logging q per call across those three buckets, the "break point" I publish finally has the denominator attached — which is the whole thing I should've done before pressing publish. Thanks for making the post do its own homework.
One dimension I'd add to the q^k math: in my experience the hazard isn't flat across call indices. The first few calls usually fail on schema and format, while later ones fail on context rot, so a single q actually understates the problem. Your "route on detectability, not difficulty" rule changed my workflow too - I now write the acceptance check before the prompt, which quietly moves more work to the cheap tier without touching the router. What I'd love to see you quantify next is the retry-loop tax in wall-clock terms: 0.97^20 sounds abstract, but expressed as "expected minutes of babysitting per shipped change" it would finally give teams the number they're really optimizing for.
This is the correction the q^k math needs, and it's sharper than my flat-q version. It's not one failure rate, it's two different failure modes on different call ranges: early calls fail on format (the model hasn't settled into the schema), late calls fail on context rot (the schema's fine, the context is poisoned). A single q averages two curves that don't belong together.
"Write the acceptance check before the prompt" is the same move I landed on, and I love that you noticed it shifts work to the cheap tier without touching the router, the check is the thing that lets a weak model's output be trusted, so routing becomes almost incidental.
On the wall-clock number: agreed, that's the one teams actually optimise and nobody publishes. "Expected minutes of babysitting per shipped change" is the honest unit. The piece that makes it computable is the stop-and-ask-count plus silent-failure-count scorecard a couple of people in this thread converged on, multiply interventions by mean handling time and you get minutes, not an abstract 0.97^20. If I run that, I'll post it. It needs real intervention logs, which is exactly the data no leaderboard collects.
The split between writing a correct patch and driving an editor reliably is the useful part of this piece. A model can score well on SWE-bench and still emit a malformed tool call on step eleven, and that one failure is what ends the session.
It also suggests a better way to decide. Instead of comparing leaderboard numbers, take one real task from your own repo, write down what done means (files allowed to change, tests that must pass, the instruction it must not break), and run it ten times on each setup. That gives you a pass rate for your work rather than for a benchmark.
Your instruction-following point, "don't touch the tests", is the one I would weight most. Did any of the local models you tried respect that boundary consistently across a multi-file edit?
That's the right protocol, and the "write down what done means" step is the one everybody skips, which is why they end up comparing leaderboard numbers instead of their own pass rate. Files allowed to change, tests that must pass, the instruction it must not break, that's an executable definition of done, and ten runs against it tells you more than any benchmark.
On "don't touch the tests" specifically, since you'd weight it most: it was the least reliably respected boundary I tried, and the failure was almost always indirect. The model rarely edited a test on purpose. It changed a shared fixture, or a helper the tests imported, or a type the tests depended on, and the test "passed" because the model had quietly moved the thing the test was checking. Direct test edits are easy to forbid with a path guard. The dangerous version is the model satisfying the test by changing what the test measures, and no "don't edit /tests" rule catches that, because it never touched /tests.
So the boundary that actually holds isn't "don't edit the tests," it's "the tests and their fixtures are frozen, and done means they pass unchanged." Local models respected the literal rule more often than the real one.
The multi-step chains point matches what I keep running into. A single tool call is fine even on mid-size models; the failure I see is argument drift around call three or four, where the model re-invokes the same function with a slightly different arg shape and the harness retries quietly instead of failing loud. I ended up logging schema validation failures separately from tool errors just to make it visible. Did you measure where in the chain the drift starts, or does it compound every hop?
You've described the exact failure, and logging schema-validation failures separately from tool errors is the right instrument, that split is the one most harnesses don't make, so the drift hides inside the retry count.
On where it starts: in my runs it wasn't flat per-call and it wasn't purely call-index. It tracked context depth, not hop number. The same model that drifted at call three on a context-heavy task held format past call fifteen on a lean one. The trigger was the working context crossing a size threshold, after which it started "simplifying" a nested arg it had emitted correctly ten times, usually by collapsing one level of nesting.
So my answer to "start vs compound": it compounds, but non-linearly, and the knee is a function of context size rather than a fixed call number. Which is why call-count-based retry caps don't catch it, the cap fires at the same hop whether or not the context is near the knee. If you log context size next to each schema failure, I'd bet your drift clusters above a token threshold rather than after a call number.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.