A new leaderboard update from benchmarking firm Vals AI is scrambling the pecking order among frontier AI coding models. Terminal-Bench 4.0, a test that measures how well AI agents handle real terminal-based software engineering work, posted fresh results on October 18, 2026, and the numbers tell a messier story than “one model wins everything.” Claude Opus 5.5 topped the terminal-coding test at 65.15%, edging out Claude Sonnet 5.5 at 64.14% and GPT-6 Astra at 59.60%. But on Vals AI’s broader Vals Index, a separate leaderboard tracking general model capability, Google’s Gemini 4 Argon took the top spot at 68.90%, narrowly ahead of Sonnet 5.5 and Opus 5.5.
That split result is the real headline. No single model now wins across every major benchmark, and the gap between first and third place on Terminal-Bench 4.0 has narrowed to roughly five and a half points among the top three finishers. For engineering teams picking an AI coding assistant, and for the vendors selling them, this is exactly the kind of leaderboard churn that reshapes ChatGPT vs Claude vs Gemini comparisons almost monthly now.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What Terminal-Bench 4.0 Actually Measures
Terminal-Bench is published and maintained by Vals AI, a benchmarking outfit that has become one of the more closely watched independent scorers in the AI industry over the past year. Unlike benchmarks that simply ask a model to write a function and grade the output, Terminal-Bench puts the model inside a real shell environment. It has to inspect files, read error messages, run commands, edit code or configuration, execute tests, and keep iterating until the task is actually finished, not just plausibly answered.
That distinction matters more than it sounds. A model can write elegant code in a chat window and still fail badly once it has to operate a real development environment, where a single wrong flag, a missing dependency, or a misread stack trace can derail the whole task. Terminal-Bench 4.0 is built to catch exactly that gap. Scores on the leaderboard represent the percentage of benchmark tasks a model’s agent setup completed successfully, which means the result reflects a full system, not a bare model.
The October 18 run used an agent harness called Mini-SWE-agent, described by benchmarking site Klu.ai as a lightweight, Bash-only setup that gives the model a simple loop: issue a command, read the output, decide the next move, repeat. According to Klu.ai’s methodology notes, the published scores are averaged across three attempts per task, denoted avg@3 in benchmarking shorthand. That detail is worth remembering before anyone treats a single percentage point as gospel. A model is being scored as “model plus agent loop plus tools plus execution environment,” and swapping any one of those pieces can move the number.
The October 18 Leaderboard, Row by Row
Vals AI’s October 18 update covers 50 tested models in total, according to the company’s benchmarks directory, which makes Terminal-Bench 4.0 one of the broader agentic coding leaderboards currently running. A separate snapshot from tracking site BenchLeader, dated October 19, counted 46 model configurations with published results and rounded Opus 5.5’s score to 65.2%, a difference small enough to fall within normal rounding and sampling noise between snapshots.
| Rank | Model | Agent Harness | Score | Cost per Task | Duration |
|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 | Mini-SWE-agent | 65.15% | $13.20 | 1h 04m |
| 2 | Claude Sonnet 5.5 | Mini-SWE-agent | 64.14% | $16.51 | 1h 22m |
| 3 | GPT-6 Astra | Mini-SWE-agent | 59.60% | $9.58 | 35m 42s |
| 8 | Claude Opus 5 | Mini-SWE-agent | 53.53% | $18.60 | 1h 07m |
Several things jump out of that table. First, the price-performance picture is not straightforward. GPT-6 Astra scored lowest among the top three but was also the cheapest per task and by far the fastest, finishing in just over half an hour against more than an hour for both Claude models. Second, Claude Sonnet 5.5 actually cost more per task than the higher-scoring Opus 5.5, which undercuts the usual assumption that a smaller, cheaper-tier model should always be the budget option. Third, generational improvement inside Anthropic’s own lineup is stark: Opus 5.5 at 65.15% versus Opus 5 at 53.53% is an 11.6-point jump within the same benchmark and harness, which is a far bigger single-generation leap than most model families manage.
Benchmark Reliability: The Fallback-Serving Problem
There is a wrinkle worth flagging for anyone who treats these scores as absolute. Klu.ai’s methodology page reports that 22 of 198 total attempts attributed to Opus 5.5 were actually served by older Opus 5 or Opus 4.8 models, likely due to backend routing or fallback behavior during high load. Adjusted for that fallback, Klu.ai calculates Opus 5.5’s “pure” score at 58.08%, well below the 65.15% headline figure on the main Vals leaderboard. Sonnet 5.5 had a smaller version of the same issue, with four attempts served by the older Sonnet 5, bringing its fallback-adjusted score to 62.63%.
Neither of those adjusted numbers is the officially published Vals leaderboard score, but they are a useful reminder that benchmark infrastructure itself can introduce noise that has nothing to do with raw model capability. An 11% fallback rate on a flagship model is not trivial. It means roughly one in nine benchmark attempts credited to Opus 5.5 may have actually been handled by a different, older model behind the scenes, likely due to capacity constraints during a high-traffic testing window. For a company running its own internal comparison before a procurement decision, that is a strong argument for repeating any public benchmark locally, under controlled conditions, rather than trusting a single headline number.
Why Gemini 4 Argon Wins a Different Leaderboard
While Claude Opus 5.5 leads the terminal-coding test, the picture flips on Vals AI’s broader Vals Index, which the company also runs and which aims to capture general-purpose model capability rather than a single workflow like shell-based coding. On that index, dated October 16 and repeated in Vals AI’s October 18 update, Gemini 4 Argon ranks first at 68.90%, with Claude Sonnet 5.5 close behind at 67.04% and Claude Opus 5.5 at 66.97%.
An independent comparison published on developer community site Dev.to by writer Gabriel Anhaia put the contrast bluntly, framing the practical takeaway as something close to “race them on your own repository” rather than trusting either leaderboard blindly, since Gemini 4 Argon’s Vals Index win did not carry over to a win on Terminal-Bench. A separate Dev.to piece by writer Hassan pulled apart Google’s own benchmark comparison figures, which reportedly showed Opus 5.5 at 66.4% against Argon at 57.4% on a Terminal-Bench style comparison run by Google itself, a different number again from Vals AI’s own 57.58% figure for Argon on the same benchmark. Multiple organizations running the same nominal test and landing on different scores is common in this industry, and it is a big part of why serious buyers increasingly run their own acceptance tests rather than shopping purely off a public leaderboard screenshot.
Put simply: there is no longer one benchmark that crowns a single “best” model. Terminal-Bench rewards models that are disciplined and efficient inside a shell loop. The Vals Index rewards something closer to overall reasoning and task breadth. A model provider can legitimately claim the top spot on one chart while trailing on another, and increasingly, that is exactly what is happening.
Terminal-Bench vs. SWE-Bench: Two Different Yardsticks
Terminal-Bench is not the only agentic coding benchmark drawing attention this year, and buyers researching this space will likely run into SWE-Bench scores from the same model releases, which measure something related but not identical. SWE-Bench typically grades a model on resolving real pull-request-style issues pulled from open-source repositories, scoring whether the generated patch passes the project’s existing test suite. Terminal-Bench, by contrast, drops the model into a live shell session and grades full task completion across a broader range of operations, including file navigation, environment setup, and multi-step debugging, not just a single patch.
The two tests tend to produce different rankings because they stress different skills. A model tuned heavily for patch-style code generation can still stumble in a raw terminal session if it struggles to interpret ambiguous shell output or recover from a failed command. That is part of why industry figures following both benchmarks, including the 78% SWE-Bench figure reported for Codex CLI’s agents earlier this year, should be read as a measure of a specific skill rather than a universal coding IQ score. Buyers evaluating tools for autonomous terminal-based agents should weight Terminal-Bench more heavily; buyers evaluating tools mainly for reviewing and generating pull requests should weight SWE-Bench-style results more heavily.
Market Impact: What the Split Verdict Means for AI Vendors
For the three companies involved, a divided leaderboard carries real commercial weight even without a dramatic single-day stock reaction. Anthropic’s Claude Opus 5.5 win on Terminal-Bench gives its sales teams a credible, independently-run data point to use with enterprise customers specifically evaluating autonomous coding agents, a fast-growing segment of the market. Google’s Gemini 4 Argon win on the broader Vals Index supports a different pitch, aimed at buyers who want one model to handle a wider spread of tasks beyond coding. OpenAI’s GPT-6 Astra, despite placing third on Terminal-Bench, has a strong efficiency argument: the lowest cost per completed task and the fastest completion time of the three, which matters enormously for any company running these agents at scale across thousands of daily tasks.
None of the three outcomes is a knockout blow, and that is itself the market signal. The AI coding assistant category, which already includes aggressive competition from Mistral’s Large 4 and open coding models like JetBrains’ Mellum line, looks set to stay a multi-vendor market through 2027 rather than consolidate behind one dominant model, simply because different benchmarks keep validating different leaders.
Pricing: Token Cost vs. Task Cost
One of the more useful, and more confusing, aspects of the October update is that Vals AI publishes two different kinds of cost figures: raw per-token API pricing, and the total cost it actually took to run each model through a benchmark task, including every retry, tool call, and context token consumed along the way. Those numbers tell different stories.
| Model | Input Price / 1M Tokens | Output Price / 1M Tokens | Benchmark Cost per Task/Test |
|---|---|---|---|
| Claude Opus 5.5 | $4 | $20 | $13.20 (Terminal-Bench) |
| Claude Sonnet 5.5 | $2 | $10 | $16.51 (Terminal-Bench) |
| GPT-6 Astra | $10 | $50 | $9.58 (Terminal-Bench) |
| Gemini 4 Argon | $4 | $20 | $15.68 (Vals Index) |
Notice that GPT-6 Astra carries the highest per-token price of the four, at $10 input and $50 output per million tokens, yet it posted the lowest cost per completed benchmark task. That is only possible because it also finished tasks in roughly a third of the time its Claude rivals needed, burning far fewer tokens and tool calls along the way even at a higher sticker price. Meanwhile Claude Sonnet 5.5, priced at the cheapest token rate on this table, actually cost more per Terminal-Bench task than the pricier Opus 5.5, because it took longer and needed more turns to reach a working solution. The lesson for anyone budgeting an AI coding deployment is that token price alone is a poor proxy for real cost. Total cost to complete a task, including retries and tool overhead, is the number that actually hits a cloud bill.
Historical Context: How Fast the Frontier Is Moving
The available Vals AI data does not include a full archived snapshot of where Terminal-Bench scores stood in the middle of 2026, so a precise month-by-month trend line is not something that can be stated as verified fact. But the numbers that are public tell a clear generational story on their own. Claude Opus 5 sits at 53.53% on the same October 18 leaderboard, run under the identical Mini-SWE-agent harness, while its successor Opus 5.5 reaches 65.15%. An 11.6-point jump within one model generation, on the same test and same agent setup, is a sizable leap and suggests Anthropic made real architectural or training gains between the two releases rather than just a marketing refresh.
That kind of jump also puts this round of releases in a broader context. Coding-specific scores across the industry have moved quickly in 2026: Codex CLI’s agents were reported hitting 78% on SWE-Bench earlier this year, JetBrains pushed its open Mellum coding model to a 47% SWE-Bench score in a separate release, and Cursor’s AI agents now reportedly merge the majority of pull requests on its platform. Terminal-Bench measures something adjacent but distinct from SWE-Bench, agentic shell competence rather than pull-request-style code generation, but both sets of numbers point the same direction: coding agents are closing the gap between “helpful assistant” and “autonomous operator” faster than most enterprise procurement cycles can track.
Competitive Landscape: No Clean Winner
Anthropic can point to Terminal-Bench and claim the top agentic coding spot with Opus 5.5. Google can point to the Vals Index and claim the top general capability spot with Gemini 4 Argon. OpenAI’s GPT-6 Astra lands third on Terminal-Bench but wins decisively on both speed and cost efficiency per completed task. None of the three companies has issued a public statement specifically reacting to the October 18 leaderboard update, and no attributable quote from a named analyst at any of the major labs appears in the public coverage of this release, so any claim of an official “response” from Anthropic, Google, OpenAI, or xAI should be treated as absent rather than implied.
What is observable is the pattern itself. This is the same dynamic that showed up when Grok 4.7 was compared against Claude Sonnet 5.5 and DeepSeek’s V4.1-Flash on price and capability earlier this cycle: different benchmarks, different pricing tiers, and different strengths depending on the task, with no single model dominating every column. The AI coding assistant market has moved from “pick the best model” to “pick the best model for this specific workflow and this specific budget,” which is a much harder purchasing decision to make from a leaderboard screenshot alone.
What This Means for Enterprise AI Buyers
For engineering leaders evaluating AI coding tools heading into 2027 budget cycles, the October 18 Terminal-Bench update offers a few concrete lessons rather than a simple shopping list.
- Match the benchmark to the actual workflow. Terminal-Bench specifically measures shell-based, agentic software engineering tasks. A team evaluating a chat-style coding assistant for quick snippets is testing a different capability than a team deploying an autonomous agent that opens pull requests unsupervised.
- Price per completed task matters more than price per token. GPT-6 Astra’s higher token price did not translate into a higher total cost, because it needed fewer turns and less time to finish. Sonnet 5.5’s lower token price did not save money on Terminal-Bench, because it needed more of them.
- Check for routing and fallback behavior. Klu.ai’s finding that roughly 11% of Opus 5.5’s benchmark attempts were actually served by older model versions is a reminder to ask any vendor directly how consistently a given model version is served in production, not just in a one-off benchmark run.
- Run a private acceptance suite. Public leaderboards use generic tasks. Enterprise codebases differ wildly in language mix, monorepo scale, legacy dependencies, and internal tooling, and no public benchmark can substitute for testing an agent against a real, representative slice of your own repository.
- Treat benchmark specialization as the norm, not the exception. With Opus 5.5 leading one leaderboard and Argon leading another, buyers should expect this pattern to continue rather than wait for a single model to pull decisively ahead on every metric.
Predictions: Where the Benchmark Race Goes Next
Based on the pace and pattern of releases through 2026, several trends look likely to continue into early 2027.
- Benchmark fragmentation will deepen. As more specialized leaderboards emerge alongside Terminal-Bench and the Vals Index, expect more “split decisions” where no single model tops every chart, making vendor marketing claims harder to compare at a glance.
- Cost-per-task reporting will become standard. The gap between token pricing and actual completed-task cost exposed by this leaderboard is significant enough that more benchmarking firms are likely to publish total-cost figures alongside raw scores going forward.
- Routing transparency will become a competitive issue. As fallback-serving discrepancies like the one Klu.ai flagged for Opus 5.5 get more scrutiny, expect pressure on providers to disclose when a benchmark run was served by a different model version than advertised.
- Agentic coding benchmarks will keep gaining weight relative to static code-generation tests. The industry’s shift toward autonomous, multi-step coding agents means tests like Terminal-Bench, which evaluate full task completion in a live environment, are likely to carry more influence on purchasing decisions than single-function code tests.
- Expect rapid within-generation swings. The 11.6-point jump between Opus 5 and Opus 5.5 on the same benchmark suggests that whichever lab ships its next major update first could reclaim the top Terminal-Bench spot before competitors respond, keeping the leaderboard unusually volatile through the next two or three quarters.
Frequently Asked Questions
What is Terminal-Bench 4.0?
Terminal-Bench 4.0 is a benchmark published by Vals AI that evaluates how well AI models, operating as agents inside a terminal environment, can complete real software engineering tasks: inspecting files, running commands, editing code, and verifying that the result actually works, rather than simply suggesting code in a chat window.
Which AI model currently tops Terminal-Bench 4.0?
As of the October 18, 2026 update, Claude Opus 5.5 leads with a score of 65.15%, ahead of Claude Sonnet 5.5 at 64.14% and GPT-6 Astra at 59.60%, based on data published by Vals AI.
Why does Gemini 4 Argon lead a different leaderboard than Claude Opus 5.5?
Gemini 4 Argon leads the separate Vals Index, which measures broader general-purpose model capability rather than the narrower shell-based coding tasks that Terminal-Bench focuses on. A model can rank differently on each test depending on which specific skill is being measured.
Is Claude Opus 5.5’s 65.15% score fully accurate?
It is the official headline score published on the main Vals AI leaderboard. However, benchmarking site Klu.ai reported that roughly 11% of Opus 5.5’s test attempts were actually served by older Opus 5 or Opus 4.8 models due to backend routing, producing a fallback-adjusted score of 58.08% in its own analysis. Both figures come from real published sources but measure slightly different things.
What is Mini-SWE-agent?
Mini-SWE-agent is the lightweight, Bash-only agent harness used to run models through the Terminal-Bench 4.0 tasks. It gives a model a simple loop of issuing shell commands, reading the results, and iterating until the task is solved, and the benchmark score reflects the combined performance of the model and this harness together.
Does the cheapest AI model also complete tasks the cheapest?
Not necessarily. GPT-6 Astra carries the highest per-token API price among the top three Terminal-Bench finishers but recorded the lowest total cost per completed task, because it finished tasks faster and with fewer tool calls. Claude Sonnet 5.5, despite a lower token price than Opus 5.5, cost more per completed task on this benchmark.
How many models were tested in the October 2026 Terminal-Bench update?
Vals AI’s benchmarks directory listed 50 tested models as of the October 18, 2026 update, while a separate BenchLeader snapshot dated October 19 counted 46 model configurations with published results, a small discrepancy typical of snapshots taken on slightly different dates.
Should enterprises choose an AI coding model based on this leaderboard alone?
Benchmark results are a useful starting signal but not a substitute for direct testing. Enterprise codebases vary widely in language, scale, and tooling, and the split results between Terminal-Bench and the Vals Index show that a model’s rank can shift significantly depending on which capability is being measured.
