Coder Legion - Recent posts tagged benchmarks

How I tried to write an article about slow Chinese LLMs

Thu, 06 Aug 2026 15:51:59 +0000

Recently, I've added a bunch of hype-monsters to my AI Werewolf:

Kimi K3
Qwen 3.8 Max, Qwen 3.7 Plus, Qwen 3.7 Flash
MiniMax M3

Plus the ones I've had for a while

DeepSeek V4 Pro and Flash
GLM-5.2
Sakana Fugu base and Ultra

The last one is Japanese, not Chinese, but it was in the news a month or two ago and it belongs in this story.

I sat down to write about how slow Chinese models are. Because they are all annoyingly slow even with a small context. I had the numbers, the thesis wrote itself, and then... I discovered something I didn't expect.

Okay, let me go straight to the problem

All Chinese official APIs are extremely slow. DeepSeek got better with their v4, the rest are just terrible. So slow that they are barely usable in a text game. No, seriously, take a look.

Time to produce one four-sentence vote:

Kimi K3: 29 to 34 seconds
MiniMax M3: 25 to 30 seconds
Qwen 3.8 Max: 25 to 27 seconds, and only because I capped its thinking. Uncapped it hit 100.
DeepSeek V4 Pro: 14 to 22 seconds, the best of the group

Same prompt, same afternoon: Claude 5 Opus answers in 5.9 seconds.

I have a test which simulates the voting at the end of a game day. Some chat has happened, some players have already cast their votes, and now the model under test has to do the same. The prompt contains all of that: 36,000 characters, which works out to 8-13k tokens depending on the model's tokenizer. Not much, assuming those models have 1M contexts.

More US/EU models:

Model	Time	Input	Output tokens	Avg cost
GPT-5.6 Luna	3.2-3.6s	12,124	190-231	$0.0020
Mistral Large 3	2.9-5.2s	8,175	73-134	$0.0042
Claude Fable 5	4.8-6.0s	13,027	183-209	$0.1399
Gemini 3.6 Flash	4.6-7.1s	7,979	297-532	$0.0152
GPT-5.6 Sol	6.7-7.3s	12,124	150-156	$0.0471
Claude 5 Opus	5.9-7.8s	13,027	220-387	$0.0726
Gemini 3.1 Pro	11.6-12.1s	7,979	821-940	$0.0263
Claude 5 Sonnet	6.8-14.8s	13,027	252-821	$0.0464
Grok 4.5	10.0-14.8s	8,190	428-607	$0.0149
Mistral Medium 3.5	2.9-35.0s	8,175	57-109	$0.0128

Worth staring at the input column for a second. The same 36,000 characters is 7,979 tokens to Gemini, 8,175 to Mistral, 12,124 to OpenAI and 13,027 to Claude. Claude is billing me for 63% more prompt than Google is, for the identical text, before anyone has thought about anything.

Frontier models sit in the 3-7 range. Grok is lagging behind. Mistral is weird - their Large is way faster than the newer Medium. Gemini Pro is also not great - Google, where are your TPUs?

Mistral Medium deserves its own note. Seven runs of the same prompt across one day: 3.3s, 3.6s, 24.1s, 35.0s, 27.9s, 11.1s, 2.9s. Output was 57 to 109 tokens every single time. That's their infrastructure having a day. Whichever single measurement I had taken, it would have been wrong about this model.

Now the other group.

Model	Time	Input	Output tokens	Avg cost
Qwen3.7 Flash *	8.4-10.5s	8,216	1,093-1,174	$0.0004
GLM-5.2	16.4-21.2s	8,150	1,111-1,406	$0.0138
DeepSeek V4 Pro	14.4-22.2s	8,164	773-1,392	$0.0009
Qwen3.8 Max *	25.4-27.3s	8,250	1,120-1,159	$0.0175
DeepSeek V4 Flash	13.3-30.1s	8,243	990-2,275	$0.0005
MiniMax M3	25.3-30.4s	8,037	1,216-2,381	$0.0045
Qwen3.7 Plus *	12.9-31.0s	8,216	693-1,158	$0.0038
Kimi K3	28.9-33.5s	8,192	744-1,103	$0.0310

* Qwen rows are already capped at 1,024 reasoning tokens. Uncapped numbers are further down, and they are worse.

The only one under 15 seconds needed a thinking cap to get there. The rest need 13 to 34 seconds to produce a four-sentence vote. Twelve bots take turns talking in my game. At 30 seconds a turn, one round of discussion is six minutes of watching a spinner.

That was going to be the article.

Except for one thing

Claude 4.5 Haiku was slow.

Model	Time	Input	Output tokens	Avg cost
Claude 4.5 Haiku	19.7-42.0s	8,906	1,769-4,095	$0.0231

Very slow. Why? Isn't it the smallest, the cheapest, and therefore the fastest model? Apparently, it's one of the slowest in the roster.

What is going on there?

The pipes are fine

My first guess was - the hardware. The US has better hardware, right? And its servers are closer - less network latency.

Well, not really. Divide output tokens by seconds and you get a rough throughput per call, prefill included, across every run I have of each:

Qwen3.7 Flash: 104-145 tokens/s
Claude 4.5 Haiku: 90-98 tokens/s
DeepSeek V4 Flash: 73-76 tokens/s
Gemini 3.1 Pro: 70-78 tokens/s
GLM-5.2: 57-74 tokens/s
MiniMax M3: 48-78 tokens/s
Claude 5 Opus: 37-49 tokens/s
Qwen3.8 Max: 42-44 tokens/s
Kimi K3: 26-35 tokens/s

Look at DeepSeek V4 Flash. Three runs, 990 then 1,415 then 2,275 output tokens, and the generation rate never moves off 73-76 tokens/s. The connection is steady. The model just decided to write twice as much the third time, and the wall clock went from 13.3s to 30.1s.

DeepSeek and MiniMax generate as fast as anything Google runs. They take five times longer because they write eight times more. Only Kimi and Qwen Max are also slow per token, and even for them the token count does most of the damage.

Funny enough, Haiku types faster than every model here except Qwen 3.7 Flash, and it is the slowest model I have. 90 tokens a second, 42 seconds to answer. It isn't struggling to produce text. It produces four thousand tokens of it to say four sentences.

So the whole table reduces to one number: how many tokens does the model decide to write before it answers a question that needs four sentences. Opus wrote 220 to 387. Haiku wrote 1,769 to 4,095.

They think differently

All the latest Anthropic models support adaptive reasoning: the model reads the request and decides for itself how much thinking it deserves. Haiku doesn't, it's too old. Opus 5 outputs 220 to 387 tokens on this prompt, and that includes the answer and the reasoning. GPTs are the same, Sol outputs even less (~150 tokens). Gemini 3 does it too, though it lands on a bigger number - Google calls it dynamic thinking, and the old thinking budget is deprecated on that generation. Grok doesn't do it at all, it runs at fixed high effort, and that's why it takes 10-15 seconds.

Every Chinese model in my roster reasons at close to full tilt on every request, whatever the request is:

Kimi K3 always runs at max effort. There is no supported way to turn it down. Roughly 85-90% of its output tokens are reasoning.
Qwen offers a thinking_budget you set up front. The default is 4096, and that default is what produced the 100-second runs below.
DeepSeek and GLM ship thinking-only entries with no depth control I can reach.
MiniMax M3 advertises adaptive thinking, and this is the interesting one. It has the feature by name and still spent 1,216 to 2,381 tokens deciding who to vote for.

Every one of those is defensible if you're building a coding agent or answering a hard question. A reasoning model is graded on benchmarks where thinking longer nearly always scores better and nothing charges it for the tokens. Left alone, thinking hard on everything is the behavior you get, and it's the behavior that wins the leaderboard I'm not running.

My game is a different animal. Waiting for 30 seconds for a bot to cast a vote is devastating. Imagine my frustration seeing this slowness in a basic chat, while the rest of the world is celebrating Chinese AI winning the race.

The lottery

Oh, and this performance is super inconsistent. Qwen 3.8 Max shows it best. Before I capped it, four runs of the identical prompt: 26.3s, 30.6s, 81.9s, 100.5s. The output tokens went along with it: 1,068, 1,149, 3,363, 4,169.

Line those up and the generation rate is 40.6, 37.5, 41.1 and 41.5 tokens per second. Close to constant. The model was working at the same speed the whole time. It just decided, twice, that this question deserved three times more thought.

Nothing changed between those runs. Same prompt, same day, same server, same everything. What moves is the chain of thought itself, and a chain of thought is a sequence the model writes one token at a time with a temperature attached. I run Qwen at 0.7. Early in the trace it either reaches for "let me re-read what Kenji said on day one" or it doesn't, and that one fork decides whether you get 1,100 tokens or 4,100.

So it isn't a spread around some typical value, it's closer to two outcomes. Max's four runs: 1,068, 1,149, then 3,363, 4,169. Two short, two long, nothing in between. Qwen Flash did the same - 1,831, 1,897, 1,912, and then one run at 3,035. Haiku went 1,769, 2,635, 4,095.

You can't provision for that. A model with a 27-second typical case and a 100-second bad case needs a timeout set for the bad case, and a player who has to sit through it.

Low thinking budget

Qwen's API takes a thinking_budget. I capped all three Qwen models at 1,024 reasoning tokens and Max settled into 25.4-27.3s. Worse reasoning, bearable latency. That is a trade I will take for a game where a human is sitting there waiting.

The cap only helps the models that were over it. Flash was writing 1,831 to 3,035 tokens uncapped, so the cap cut it to 8.4-10.5s. Plus was already choosing 760 to 1,262, mostly under the limit, so capping it changed nothing and it still turned in a 31-second run.

For comparison, Gemini 3.1 Pro writes about as much as capped Qwen Max, 821 to 940 tokens, and comes back in 11.6-12.1 seconds every single run. Similar output, less than half the wait, and none of the spread. Nobody set a budget on it either. It runs at high effort and picks that number itself.

MiniMax M3 has no budget parameter at all. Its adaptive setting is the only throttle, and the only stronger lever is turning thinking off entirely. Kimi K3 has an undocumented K2-era thinking: disabled toggle I don't want to build on. For those two, what you measure is what you get.

The money is the good news

The cheap ones are properly cheap:

Qwen3.7 Flash: $0.0004 a vote
DeepSeek V4 Flash: $0.0005
DeepSeek V4 Pro: $0.0009

DeepSeek V4 Flash wrote 990 to 2,275 output tokens against Mistral Large's 73 to 134, and it still cost eight times less.

Then there is Kimi K3. It bills $3 in and $15 out, the same rate card as Claude 5 Sonnet. Ignore the averages in the table for a moment, since one of K3's three runs caught a warm cache: on cold runs K3 cost $0.036 and $0.041, Sonnet $0.043 and $0.045. Near enough the same money for a reply that took four times longer, and 85-90% of K3's output tokens were reasoning that nobody ever sees. Note also that K3 is being billed for 8,192 input tokens where Sonnet is billed for 13,027 of the same text.

The Qwen cap helped here too. Max's runaway 4,169-token think cost $0.0415 against $0.0229-0.0235 for a normal run. The lottery is a billing lottery as well as a latency one, and capping the thinking capped both.

Caching works on all of them, which matters when you re-send a growing conversation every turn. Those wide cost ranges in the tables are this effect. I ran the sweep twice nine minutes apart, and on the second pass Kimi went from $0.0411 to $0.0163 and GLM from $0.0167 to $0.0083, with the same prompt and more output tokens than the first time.

MiniMax reported cache hits on my very first probe call, before I had done anything deliberate about it. Qwen's implicit cache cut a real call from $0.0229 to $0.0115 between two runs. DeepSeek reads cached input at 2% of the normal rate, the deepest discount in my whole roster.

And then there is Fugu

Sakana's Fugu Ultra is Japanese, and it is seven times slower than the slowest model in that second table.

It is so slow that my tests were timing out after 3 min of waiting for a response.

The bill for one successful vote was $0.585. About 65% of that was orchestration_input_tokens and orchestration_output_tokens, internal drafts and re-reads that never reach me. I don't know what it is orchestrating, but this is not usable for me at all. Four times the price of Fable, and the waiting time is forever.

Cache hit rate on the byte-identical prompt sent minutes earlier: zero. Not low. Zero.

Conclusion

Chinese models reason more - this is the main cause of their slowness.
Per token they are slower too, but only by about 20-40%. Kimi K3 and Qwen 3.8 Max are the exceptions at roughly 2x, and Qwen 3.7 Flash is the fastest generator in the whole roster. The token count does most of the damage, not the throughput.
Some Chinese models are cheap. Kimi K3 and Qwen 3.8 Max are not so cheap.

And this is why I hate benchmarks.

One late addition: while I was writing this, DeepSeek emailed me about a billing adjustment. Continue using the service and you accept the new terms. So the cheapest column in my tables has a shelf life, though I doubt it moves enough to change the ranking.

5 coding-agent turns vs 1 command_run workflow: full example

Mon, 03 Aug 2026 21:04:04 +0000

Disclosure: I maintain Tura.

Here is the simple idea. A normal coding agent may use five separate LLM turns for one predictable workflow.

Turn 1 — inspect

rg -n "TODO|command_run|handler" crates/
rg --files crates/runtime/src crates/tools/src

Turn 2 — apply the patch

- // old command handler logic
+ // patched command handler logic

Turn 3 — build

cargo build -p runtime

Turn 4 — test

cargo test -p runtime --lib

Turn 5 — lint

cargo clippy -p runtime --all-targets

The overhead is not only the shell commands. The model wakes up five times and receives the growing conversation again.

Tura exposes one macro tool called command_run, so the agent can send the same workflow once:

{
  "name": "command_run",
  "arguments": {
    "commands": [
      { "step": 1, "command_type": "shell_command", "command_line": "rg -n \"TODO|command_run|handler\" crates/" },
{ "step": 1, "command_type": "shell_command", "command_line": "rg --files crates/runtime/src crates/tools/src" },
      { "step": 2, "command_type": "apply_patch", "command_line": "*** Begin Patch\n*** Update File: crates/tools/src/command_run/handler.rs\n@@\n-    // old command handler logic\n+    // patched command handler logic\n*** End Patch" },
      { "step": 3, "command_type": "shell_command", "command_line": "cargo build -p runtime" },
      { "step": 4, "command_type": "shell_command", "command_line": "cargo test -p runtime --lib" },
      { "step": 4, "command_type": "shell_command", "command_line": "cargo clippy -p runtime --all-targets" }
    ]
  }
}

Build, test, and lint still run. The model simply does not need a new turn between predictable steps.

In the full DeepSWE comparison, Balanced used 35.8% fewer turns and 31.1% fewer tokens than Codex CLI. Direct used 69.1% fewer turns and 77.5% fewer tokens.

GitHub: https://github.com/Tura-AI/tura

Benchmark: https://turaai.net/benchmark

Parameterization Is Distillation

Tue, 28 Jul 2026 03:56:17 +0000

Climate modeling and AI compression are one engineering move with two names: replace an expensive process with a cheap function fit to its behavior, and inherit the consequences of what you threw away. One field has been living with the failure modes for fifty years.

In November 2020, a team of climate scientists published a result that should be pinned above the desk of everyone who has ever shipped a compressed model. Brenowitz and colleagues had trained two machine-learning emulators to stand in for expensive atmospheric physics inside a climate model: a neural network and a random forest. On the offline test set, the neural network won. It fit the held-out data better. It was, by the numbers every ML practitioner reports, the better model.

Then they plugged both into the running atmosphere. Coupled into the climate model at 200-kilometer resolution, the random-forest version ran stably. The neural-network version crashed the simulated planet within seven days.

The mechanism, when they traced it, is the part worth memorizing (Brenowitz et al., arXiv:2011.03081). Coupled to the model's wave dynamics, the network's small errors fed back into its own inputs, and the feedback loop marched the system into states unlike anything in the training data. The network then did what networks do on inputs far from their training distribution: something confidently wrong, which fed back again. The offline benchmark had tested the model in the one condition that could not reveal this failure, because a static test set never lets the model's outputs become its next inputs. The benchmark measures the open loop. Deployment is the closed loop. And in the closed loop, the more expressive model was the unstable one.

If you build AI systems, that story should feel less like a curiosity from another field and more like a postcard from your own future. Because the two fields are, in a precise sense, doing the same thing.

The same move, twice

A global climate model cannot resolve a cloud. Its grid cells run roughly 100 to 250 kilometers on a side; the convection that actually makes weather happens at the scale of meters, and the droplet physics at microns. Nor can you simply refine the grid: halving the horizontal spacing of a coupled model multiplies compute cost by roughly sixteen, because you get four times the cells and the timestep must shrink too, and a century-long high-resolution run already costs millions of CPU-hours. So climate modelers do something called parameterization: they replace the unresolvable fine-scale physics with a cheap function of the coarse variables the model does track. You do not simulate the cloud. You simulate what the cloud does to the grid cell.

A frontier network cannot be cheaply served. So engineers distill: they train a small student to reproduce the input-output behavior of a large teacher. You do not reproduce the computation. You reproduce what the computation does to the output.

Same epistemic move, made independently, under different names: replace an expensive process with a cheap function fit to its behavior, and inherit the consequences of what you threw away. The most famous production example in AI is DistilBERT, which kept 97% of BERT's language-understanding performance on GLUE with 40% fewer parameters and 60% faster inference (Sanh, Debut, Chaumond and Wolf, 2019). The right way to think about that artifact is not “a small BERT someone designed” but “BERT seen through a small aperture.” Climate modeling has been building small apertures onto expensive physics for fifty years.

An honest boundary before going further: classical parameterizations are not distillation. A scheme like Arakawa-Schubert was derived by hand from physical reasoning, closer to an expert-written heuristic than to a student fit to a teacher. The identity gets tight in two places. One is the machine-learning era of climate modeling, roughly 2018 onward, where networks are literally trained on the outputs of high-resolution physics. The other is older and stranger, and it shows how deep the parallel runs.

In 2001, two papers (Khairoutdinov and Randall in Geophysical Research Letters; Grabowski in the Journal of the Atmospheric Sciences) introduced what the field calls superparameterization: embed a small two-dimensional cloud-resolving model inside each grid column of the coarse global model, and have the coarse model query it for the effects of convection. Look at that architecture with machine-learning eyes. There is an expensive teacher that knows the fine-grained truth. There is a cheap student that runs at scale. The student consults the teacher for the signal it cannot compute itself. The modern ML variant simply swaps the embedded teacher for a neural network trained to mimic it. Two communities built the same scaffolding, named it different things, and did not notice each other for over a decade.

When Hinton, Vinyals and Dean formalized distillation in 2015 (generalizing a 2006 idea from Buciluǎ, Caruana and Niculescu-Mizil), the heart of it was “dark knowledge”: train the student not on hard labels but on the teacher's softened probability distribution, which encodes the relative judgments the teacher makes about non-winning classes. Climate science even has an analog of that. A grid cell's mean temperature and humidity are its hard labels; the full distribution inside the cell (which parcels are saturated, how vertical velocity covaries with buoyancy) is the soft structure that determines whether convection actually fires. Schemes like SHOC (Bogenschutz and Krueger, 2013) and CLUBB explicitly track those subgrid probability distributions. Climate scientists were writing down dark knowledge years before machine learning handed them the word.

So the thesis is not an analogy hunting for a rhyme. It is one engineering move with two names, and the useful consequence is this: climate science has been living with the failure modes of that move for decades, at planetary stakes, and has written down lessons AI compression is only now rediscovering.

The two tails

Here is the lesson I find most valuable, because at first glance the two literatures flatly contradict each other.

From climate: in 2018, Rasp, Pritchard and Gentine published a landmark result in PNAS. Their neural-network parameterization, trained on a superparameterized model, produced a precipitation distribution that “closely matches that of SPCAM, including the tail,” where conventional hand-written schemes show “too much drizzle and a lack of extremes.” Read that carefully: the compressed model beat the hand-crafted one precisely on extreme events. Compression preserved the tail. (It was also around 20 times faster than the superparameterized physics it replaced, and roughly 8 times faster at the whole-model level once radiation is included. Not the four-orders-of-magnitude speedup that sometimes gets attached to this literature; the honest figures are impressive enough.)

From AI: in 2019, Hooker, Courville, Clark, Dauphin and Frome asked what compressed networks forget, and found that compression's damage is not spread evenly. It concentrates on the underrepresented long tail of the data distribution: the rare, atypical examples. They even gave the casualties a name, Pruning Identified Exemplars. Compression destroyed the tail.

Both results are solid. The resolution is that they concern two different tails, and the distinction is worth more than either finding alone.

Rasp's preserved tail is a tail of the output distribution: extreme rainfall intensities. Those extremes were densely sampled in training, because the teacher simulated plenty of storms. The network had every opportunity to learn them, and it did. Hooker's lost tail is a tail of the input distribution: rare kinds of cases the model seldom saw. And the climate paper confirms this reading inside its own results, because the same network that nailed the rainfall extremes was, in the authors' words, “unable to generalize to much warmer climates.” Pushed about four kelvin beyond its training climate, it broke down. Familiar extremes: preserved, even mastered. Unfamiliar inputs: lost.

Now say the alarming part plainly. The entire purpose of a climate model is to project a climate that has not happened yet. The compressed model was strongest exactly where the teacher was well-sampled and weakest exactly where the model exists to be used: the novel case. The same sentence transfers to AI word for word. A distilled student shines on the distribution it was fed and fails on the inputs nobody thought to include, and no aggregate benchmark number distinguishes the two, because the aggregate is dominated by the well-sampled middle. When someone tells you their compressed model “preserves performance on hard cases,” the question that matters is: which tail? Hard outputs it saw often, or hard inputs it barely saw? Those are different claims, and only one of them survives contact with novelty.

Lessons from a field that tuned first

Climate science also ran ahead of AI into a subtler trap: what happens when you optimize a lossy model against a benchmark for years.

In 2013, Suzuki and colleagues showed something uncomfortable in Geophysical Research Letters: a climate model could best match observed temperature trends while adopting parameter values that worst reproduced satellite-observed cloud microphysics. The system got the right answer through compensating errors, one wrong process canceling another. The definitive statement came in Hourdin and colleagues' 2017 survey of model tuning in BAMS, which describes the compounding trap: improving a component of a heavily tuned model can make its measured skill worse, because a genuine fix breaks the web of compensations, and the harder a model has been tuned, the harder it becomes to demonstrate that any real improvement is one.

Sit with that in AI terms. A model tuned hard enough to a metric becomes resistant to genuine improvement. Every real fix looks like a regression, because the errors it removes were load-bearing. The AI community is currently rediscovering pieces of this under the names benchmark contamination and eval overfitting; the climate literature stated the mechanism more precisely, a decade earlier. And it points at the same discipline: skill on the tuned metric is evidence about the tuning, not about the model.

Which leads to the healthiest habit the climate community has and AI mostly lacks. The defining climate quantity, equilibrium climate sensitivity (the long-run warming from doubled CO2), is dominated by exactly these parameterizations, above all clouds. Across the current CMIP6 generation of models, the raw spread runs from 1.8 to 5.6 kelvin (Zelinka et al., 2020), wider than the previous generation's 2.1 to 4.7. Volodin (2021) showed that reasonable changes to cloud parameterization inside a single model could span 1.8 to 4.1 kelvin by itself. The students, in other words, disagree wildly, and more compute has not closed the gap, because the bottleneck was never the grid. Yet the IPCC's assessed range in AR6 is narrower than the raw model spread: likely 2.5 to 4.0. Notice the direction of that move. The assessment did not average the models and call it truth. It brought in observations, paleoclimate evidence, and process understanding, and used them to discount model behaviors it had independent reason to distrust. The models are students; the assessment refuses to grade the students by asking the students.

Distillation pipelines rarely have this. The teacher defines the target, the benchmark is drawn from the teacher's world, and there is often no outside referent at all. Climate can discover its teacher was wrong, because the atmosphere gets a vote. Your compression pipeline usually cannot, and that asymmetry should make you humbler about what “97% of teacher performance” means.

Honesty requires the counterweights. The seven-day crash is a discovered principle, not the state of the art: stable neural parameterizations now run across resolutions (Yuval and O'Gorman, 2020, among others), and NeuralGCM (Kochkov and colleagues, Nature, 2024) tracks climate statistics for decades at 140-kilometer resolution, with realistic tropical-cyclone frequencies emerging on their own, at savings the authors describe as orders of magnitude. The field now has a five-billion-example benchmark, ClimSim, for exactly this hybrid work. And the compression story is not uniformly grim: Rasp's 2018 network conserved column moist static energy to a remarkable degree without ever being told to, a physical invariant emerging from imitation alone. Students sometimes learn structure nobody put in the curriculum. The point is not that compression fails. It is that fifty years of compressing the atmosphere taught one field exactly where the costs hide, and the map transfers.

What to do with the map

Four habits, each one a climate lesson wearing AI clothes.

Evaluate the closed loop, not just the open one. The offline benchmark cannot see feedback failures, and agentic deployment (where a model's outputs become its own next inputs through tool calls, memory, and multi-step plans) is exactly the coupled simulation. Before trusting a distilled model in a loop, test it in the loop: long rollouts, self-conditioning, the model marinating in its own outputs. The better offline model can be the one that diverges.

Audit the two tails separately. Ask of any compressed model: how does it do on rare but well-sampled outputs (probably fine), and how does it do on rare, undersampled inputs (this is where it quietly died). Build the second test set deliberately, because no aggregate metric will build it for you.

Ask the tuning question of your own evals. How hard has this system been optimized against this metric, and if someone genuinely improved a component tomorrow, would the metric reward them or punish them for breaking the compensations? If you cannot answer, your benchmark score is measuring your tuning history, not your model.

Keep an outside referent. The IPCC narrows the model range using evidence the models never saw. Find your equivalent: human evals off-distribution, live user outcomes, adversarial probes drawn from outside the teacher's world. A student graded only on the teacher's syllabus will look exactly as good as the syllabus, and no better, forever.

Climate scientists did not choose to compress; the atmosphere forced them, and the forcing made them serious about the costs decades before AI made compression an elective convenience. That is the real gift here. When a field has no choice but to live inside its approximations, it learns precisely how approximations betray you. The parameterization literature is a fifty-year field manual for the thing your team started doing three years ago. Read it before your seven days are up.

Sources

Brenowitz, N. D., Henn, B., McGibbon, J., Clark, S. K., Kwa, A., Perkins, W. A., Watt-Meyer, O., & Bretherton, C. S. (2020). “Machine Learning Climate Model Dynamics: Offline versus Online Performance.” arXiv:2011.03081 (Climate Change AI, NeurIPS 2020). The offline-winning neural network crashed the coupled 200-km run within 7 days while the random forest stayed stable; instability traced to linearized network behavior under wave dynamics and feedback-generated out-of-sample inputs.
Rasp, S., Pritchard, M. S., & Gentine, P. (2018). “Deep learning to represent subgrid processes in climate models.” PNAS, 115(39), 9684–9689. “Closely matches that of SPCAM, including the tail”; conventional schemes' “too much drizzle and a lack of extremes”; ~20×/~8× speedups; “unable to generalize to much warmer climates” (+4 K); emergent conservation of column moist static energy.
Hooker, S., Courville, A., Clark, G., Dauphin, Y., & Frome, A. (2019). “What Do Compressed Deep Neural Networks Forget?” arXiv:1911.05248. Long-tail concentration of compression damage; Pruning Identified Exemplars.
Khairoutdinov, M. F., & Randall, D. A. (2001). Superparameterization. GRL, 28 (doi:10.1029/2001GL013552).
Grabowski, W. W. (2001). Superparameterization. J. Atmos. Sci., 58.
Hinton, G., Vinyals, O., & Dean, J. (2015). “Distilling the Knowledge in a Neural Network.” arXiv:1503.02531; lineage: Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006). Dark knowledge and soft targets. Subgrid-PDF analog: Bogenschutz, P., & Krueger, S. (2013), SHOC, JAMES; CLUBB.
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). “DistilBERT.” arXiv:1910.01108. 40% fewer parameters, 60% faster, ~97% of GLUE.
Suzuki, K., et al. (2013). “Evaluating cloud tuning in a climate model with satellite observations.” GRL (doi:10.1002/grl.50874).
Hourdin, F., et al. (2017). “The Art and Science of Climate Model Tuning.” BAMS, 98(3) (paraphrased; the compounding-tuning trap).
Zelinka, M., et al. (2020). CMIP6 ECS spread. GRL (doi:10.1029/2019GL085782): 1.8–5.6 K vs. 2.1–4.7 K previously.
Volodin, E. (2021): 1.8–4.1 K within one model.
IPCC AR6 WG1 Ch. 7: assessed ECS likely 2.5–4.0 °C.
Kochkov, D., Yuval, J., Langmore, I., et al. (2024). “Neural general circulation models for weather and climate.” Nature, 632, 1060–1066. Yuval, J., & O'Gorman, P. A. (2020). Nature Communications: stable ML parameterizations. ClimSim (NeurIPS 2023 D&B): 5.7 billion training pairs.

A student graded only on the teacher's syllabus will look exactly as good as the syllabus, and no better, forever.

MD Anderson Spent at Least $62 Million on an AI It Never Tested Outside the Building

Tue, 28 Jul 2026 00:31:19 +0000

The improvement you measured is a covariance, not a cause

In 2013, MD Anderson Cancer Center began building a clinical decision support tool with IBM, using Watson technology. It was called the Oncology Expert Advisor. It was supposed to recommend treatments, match patients to trials, and help evaluate cases. Over roughly four years the project consumed at least $62 million.

By September 2016 it was not in clinical use. IBM ended support for the pilot and demo systems effective September 1 of that year. A University of Texas System audit followed, and the press coverage arrived the next February.

The number everyone quotes is the $62 million. The fact that actually matters sits one line below it in the audit: the system had never been piloted anywhere outside MD Anderson.

That is not a statement about whether the software was any good. It is a statement about what could possibly have been known. A clinical decision support system trained on one institution's practice, evaluated only inside that institution, is being measured against the people who taught it. Agreement is the expected result. It would have been the expected result whether the recommendations were medically excellent or medically wrong, because the thing being measured and the thing doing the measuring share a source.

There was no out-of-sample test to fail. That is what the $62 million bought.

Keep two findings apart

The audit is easy to misread, and misreading it is the fastest way to get this story wrong.

What the auditors found was a procurement problem. Contracts, approvals, compliance. Nearly all of the spending went ahead without board approval. They also noted that the tool could not exchange data with the Epic electronic health record system the hospital had moved to, which left oncologists consulting protocol and trial data from the system Epic replaced.

What the auditors explicitly did not do was evaluate the model. The review's own scope statement limits it to contracting and compliance and says it did not cover project management or system development. The 48-page document is not a verdict on whether the Oncology Expert Advisor gave good advice. Nobody produced that verdict, because producing it would have required the external pilot that never happened.

So there are two failures here and they are not the same failure. One is that money moved without the approvals it needed. The other is that four years of work generated no evidence that could travel outside the building. The first is a governance problem. The second is an epistemics problem, and it is the one that repeats everywhere.

What external validation looks like when someone actually runs it

Watson for Oncology, a related but distinct product, was trained with Memorial Sloan Kettering and did get deployed to other institutions. So we can see what happens when the test finally runs.

Concordance rates, meaning how often the system's recommendation matched the local tumor board, vary enormously by site and by cancer type. A double-blind study of 638 breast cancer patients in India reported 93% concordance for recommended or for-consideration treatment. Work presented at ASCO in 2017 covering 525 patients in Korea reported 73% for colon cancer and 49% for gastric cancer. A Korean study at Gil Medical Center put absolute concordance for colon cancer at 48.9%, rising to 65.8% when cases judged acceptable were included. For gastric cancer the same kind of analysis reported 41.5% at the recommended level and 87.7% at the for-consideration level.

Read those numbers as a group rather than individually. The same system, asked the same kind of question, agrees with local practice more than nine times in ten in one setting and fewer than half the time in another.

The interesting reading is not that the system was bad. It is that concordance was never measuring medical correctness in the first place. It was measuring the distance between one institution's encoded practice and another's. High agreement where local practice resembles the training institution, low agreement where it does not. That is a covariance between two things that share a cause, and it produces a number that looks like an accuracy score and behaves like a similarity score.

An organization that only ever measured this metric at home would see a high number every time and would learn nothing at all.

Why the home number is so convincing

It would be comforting to file this under carelessness, but the internal number is persuasive for reasons that survive being smart and careful.

The first is that it is not imprecise. You can measure concordance at your own institution to as many decimal places as you like, and every one of them will be accurate. The precision is real. It is attached to the wrong quantity. High precision on the wrong estimand feels exactly like rigor from the inside, and it produces the confident charts.

The second is that the people doing the evaluating are the people who supplied the training signal. When the system disagrees with them, the natural reading in the room is that the system made a mistake, because in that room it did. Disagreement gets logged as error and corrected away. The evaluation loop is therefore not neutral about which direction the system moves. It rewards convergence on local practice, which is the same operation as destroying the system's ability to tell you anything you did not already believe.

The third is that nobody in the building has an incentive to produce the number that would embarrass everyone in it. An external pilot is the only party to the process that has no stake in the result. That is not a cynical point about human character. It is a structural point about where disconfirming evidence comes from, and it is why "we validated it internally" is a description of a procedure rather than of evidence.

The same bug, in your own field

If oncology feels far away, here is the version with a coefficient attached.

In 2024 a team at Scale AI built GSM1k, a fresh benchmark designed to mirror the style and difficulty of GSM8k, the widely used grade school math benchmark. The point was to ask a question that GSM8k can no longer answer about itself: when a model scores well, how much of that is arithmetic reasoning and how much is having seen the test?

Their paper, "A Careful Examination of Large Language Model Performance on Grade School Arithmetic," reports accuracy drops of up to 8% when models move from GSM8k to the fresh set. Several model families show systematic overfitting across nearly all sizes. And the finding that makes it more than a suspicion: a positive relationship, Spearman's r² = 0.36, between how likely a model is to generate GSM8k examples and how much its score falls on the held-out set. Models that can recite the test do worse when the test changes.

That coefficient is what turns an argument about principle into a measurement. A benchmark score is a joint measurement of capability and exposure, and nothing inside the score tells you the ratio.

Three numbers that drifted on the way here

While assembling this piece we ran into the thesis three separate times, in our own sources.

The GSM1k accuracy drop is reported in a good deal of secondary coverage as 13%. The paper's abstract says up to 8%. We took the primary.

The University of Texas audit is described in several retellings as posted on January 31, 2016, which cannot be right, because the events it audits run through September 2016. The posting date is January 31, 2017. A year fell off in transit.

The split of the $62 million between IBM and the consulting firm that supported the project is given slightly differently across accounts, roughly $39 to $40 million to IBM and roughly $21 to $23 million to the consultancy, while the total holds steady. This is why the figure here is "at least $62 million," which is the reporting's own hedge, rather than a precise sum.

None of these drifts is scandalous. Each write-up was doing its honest best to report what a study or an audit found. That is the point. A number does not need anyone to lie about it in order to arrive somewhere false. It only needs to be copied a few times by people who did not go back to the source, and each copy is individually reasonable.

The counterweight, because the lazy version of this argument is wrong

The GSM1k paper also found that frontier models showed minimal signs of overfitting. They generalized to problems guaranteed to be absent from their training data.

That matters, and leaving it out would be the same sin this essay is about. The honest claim is not that benchmarks are theater or that measured progress is fake. Real capability gains exist and the same study that quantifies contamination also demonstrates them.

The claim is narrower and more useful: a measured improvement licenses a causal reading only when the thing you changed is the only thing that moved. When your evaluation shares a source with your training, whether that source is an institution, a market regime, or a scraped corpus, some fraction of the agreement you observe is an echo, and the score alone will not tell you how much.

The zero that was a base rate

Here is the version of this that nearly got past us this morning.

Working a research question on July 27, 2026, we wanted a data-grounded read on whether a particular kind of skilled work had been touched by AI adoption. The Anthropic Economic Index publishes a labor market impacts release with task penetration and job exposure data, so we pulled the task-level file and looked at every calibration and metrology task in it.

Every one read 0.0.

That is a striking result. Measured AI penetration into this work, zero. It would have made an excellent line, and we were most of the way to writing it.

Then we counted the rest of the file. By our count of that file, of 17,998 task statements only 1,354 carry a nonzero value. About 92.5% of the file reads 0.0. Our finding was the modal value of the dataset. We had measured the sparsity of a file and were about to report it as a fact about an occupation.

The fix was to move to the occupation-level file, where the distribution actually discriminates: by our count, a mean of 0.0770 across 756 occupations with 54.4% sitting at exactly zero. There the relevant proxies read 0.0324 and 0.0. Still low, but now meaningfully low, because there was a spread to be low against.

The zero did not change. What changed was knowing how ordinary a zero was in the population it came from.

What to compute before you believe a delta

The check that catches all four of these cases is short, and it is worth running before the number goes in a slide.

Ask what else moved during the window you measured. If the answer is anything other than "only the thing I changed," the improvement is a covariance and you should say so out loud rather than let the reader assume otherwise.

Ask what the base rate of your value is in the population it came from. An extreme reading means nothing until you know how common that reading is. A zero drawn from a file that is mostly zeros is a description of the file.

Ask what a fresh sample would say. Not a held-out split of the same collection, which shares every bias the training data has, but a matched sample sourced independently, the way GSM1k was built to mirror GSM8k without inheriting it. If getting one is impractical, that is a real constraint and worth stating. What is not acceptable is spending four years and $62 million without ever noticing that you never had one.

Three questions. None of them require new tooling, and any of them would have raised a hand somewhere in Houston in 2014.

Sources

University of Texas System Administration, special review of MD Anderson's Oncology Expert Advisor procurement (48 pages; report November 2016, results posted January 31, 2017). Scope limited to contracting, procurement and compliance; explicitly excludes project management and system development.
Matthew Herper, "MD Anderson Benches IBM Watson In Setback For Artificial Intelligence In Medicine," Forbes, February 19, 2017.
The Register, coverage of the MD Anderson audit, February 20, 2017.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati and Summer Yue (Scale AI), "A Careful Examination of Large Language Model Performance on Grade School Arithmetic," arXiv:2405.00332, submitted May 1, 2024, revised November 22, 2024. Figures quoted from the abstract: accuracy drops of up to 8%, Spearman's r² = 0.36, frontier models showing minimal signs of overfitting.
S. P. Somashekhar et al., "Watson for Oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board," Annals of Oncology, 2018 (PMID 29324970). 638 breast cancer cases, Manipal Comprehensive Cancer Center, 2014–2016. Concordance 93%, where a recommendation counts as concordant if the tumor board's choice was designated "recommended" or "for consideration" by the system. The study included a blinded second review by the tumor board in 2016 of the cases where the two disagreed; earlier conference reporting of the same work quotes a lower pre-review figure.
ASCO 2017, Gachon University Gil Medical Center, Incheon: 525 patients treated 2012–2016 (340 colon, stage II–IV; 185 chemotherapy-naïve advanced gastric). Concordance 73% colon, 49% gastric.
Won-Suk Lee et al., "Assessing Concordance With Watson for Oncology, a Cognitive Computing Decision Support System for Colon Cancer Treatment in Korea," JCO Clinical Cancer Informatics, 2018 (doi:10.1200/CCI.17.00109, PMID 30652564). 656 patients, stage II–IV colon cancer, 2009–2016. Absolute concordance 48.9%; 65.8% (432 of 656) when cases judged acceptable are included.
Concordance Rate between Clinicians and Watson for Oncology among Patients with Advanced Gastric Cancer: Early, Real-World Experience in Korea (PMC6377977). 65 advanced gastric cancer patients, Gachon Gil Medical Center, 2016–2017. Concordance 41.5% (27 of 65) at the recommended level, 87.7% (57 of 65) at the for-consideration level.
Anthropic Economic Index, labor market impacts release: task_penetration.csv (task level) and job_exposure.csv (occupation level). The task-level and occupation-level counts given above are our own counts of those two files as accessed on July 27, 2026, not figures published by the index.

Related reading from us

Your Agent Eval Is One-Factor-at-a-Time, and Fisher Proved That's Blind. That piece is about experimental design, changing one factor at a time versus factorial designs. This one is the observational counterpart: what you can and cannot infer when you did not run an experiment at all and only have a measured delta.
The Answer Key Was in the Training Data. Direct contamination, which is the sharpest special case of the mechanism described here.
Zillow Disabled Its Human Pricing Override. Then It Wrote Down $407.9 Million. A governance failure rather than a validation failure. Different bug, comparable bill, and worth reading alongside this one precisely because the two are so often confused.

How to batch 5 coding-agent turns into 1 macro workflow

Tue, 21 Jul 2026 02:57:59 +0000