Quick Answer
LLM inference cost is what you pay every time a model generates output. Five things set the number: token volume, model choice, context length, calls per task, and hardware utilization. Rates run from cents to dollars per million tokens, and heavy agentic users burn $100 to $250 a month each. Prices keep falling. Bills keep rising.
In April 2026, the engineering team at ProjectDiscovery, the company behind the open source Nuclei security scanner, published the numbers from their own inference bill. Their cache hit rate was sitting at 7%, because dynamic working memory lived inside the system prompt and invalidated the cacheable prefix on almost every step. Moving it out got them to 74% in a single deployment. Explicit cache breakpoints and deliberate TTLs took them to 84%, and cut total LLM spend by 59% to 70% across 9.8 billion cached tokens.
Same models. Same features. Same traffic. The only thing that changed was how they packaged what they sent.
That one engineering post is the entire inference cost problem in miniature. The unit price of intelligence is in freefall, yet most bills never feel it, because the bill was never really about the price. It is about volume, structure, and the architectural defaults nobody audited.
So this is not another glossary entry. It is the math of the meter, the documented savings behind each lever, and the discipline that separates teams whose AI features have margins from teams whose AI features have vibes.
What is LLM inference, and why does it cost money?
LLM inference is a trained model doing its job at runtime: turning your prompt into a response, token by token, on expensive accelerator hardware. Training builds the model once. Inference runs it forever, on every request, for every user.
Forever is the operative word for a budget. Training is a one-time capital event you read about in headlines. Inference cost is an operating expense that scales with success: more users, more features, more agents, more spend. A successful product’s cumulative inference bill laps its training bill and never looks back.
The meter is the token, roughly three quarters of an English word. All inference pricing reduces to per-million-token rates, quoted separately for input (your prompt and context) and output (what the model generates). The cost per token is tiny. The volume never is. Every dollar you will ever spend on inference is arithmetic on that meter.
Report
Finance needs to prove AI’s return: CloudZero report
260 senior finance leaders (more than half CFOs) told us why the speed of seeing AI spend, not the size of it, separates who pulls ahead on AI from who gets burned.
How expensive is LLM inference?
Anywhere from under $0.20 to over $25 per million tokens depending on the model, a spread wider than 100x sitting behind one innocent-looking dropdown. Current flagship anchors: GPT-5.4 at $2.50 input and $15 output per million, Claude Sonnet 4.6 at $3 and $15, Claude Opus 4.8 at $5 and $25, with small models like Haiku 4.5 at $1 and $5.
Those anchors also reveal the first rule of LLM pricing: output costs roughly 5x input across major providers, because generating tokens one at a time is harder work than reading them in parallel. A model that pads its answers is not being thorough. It is being expensive on the costly side of the meter.
Per task, the same model produces wildly different bills depending on architecture. Here is illustrative math at a workhorse blended rate of about $1 per million input tokens and $4 per million output:
| Workload, illustrative math | Input tokens | Output tokens | Cost per task | At 100K tasks/month |
|---|---|---|---|---|
| Simple chat reply | 500 | 300 | ~$0.002 | ~$170 |
| RAG query with retrieved context | 6,000 | 500 | ~$0.008 | ~$800 |
| Agentic task, 12 chained calls | 60,000 | 15,000 | ~$0.12 | ~$12,000 |
One model, one rate card, 70x apart per task. Architecture, not LLM cost, is the real variable, which is why conversations about LLM costs that begin at the rate card usually optimize the smaller lever.
There’s no provider-by-provider table here on purpose. Rates change monthly, and stale price tables are how good articles become bad advice. We keep the live comparison in one place instead: our LLM API pricing comparison covers OpenAI, Anthropic, Google, and the challengers, with vendor mechanics in the OpenAI, Claude, and Azure OpenAI pricing guides.
Why is your bill rising while cost per token collapses?
Because consumption is growing faster than prices are falling, and it is not close.
The price collapse is real and historic. GPT-4 class performance that sold for about $30 per million tokens in early 2023 goes for under a dollar in 2026. Epoch AI measured the price of hitting fixed capability milestones and found declines of 9x to 900x per year, with a median around 50x, rising to roughly 200x when you look only at models released since 2024. Their summary is that prices have fallen rapidly but unequally across tasks.
Consumption is winning anyway. The Price of Progress, a 2026 study of benchmark-level pricing, found both halves of the paradox at once: the price of reaching a given benchmark score fell 5x to 10x per year, while the cost of running the frontier itself rose 3x to 18x, because each marginal capability gain demands disproportionately more inference.
Economists call cheaper inputs driving higher total spend Jevons paradox. Cheaper intelligence gets used for more things, and usage compounds faster than discounts. The 19th century version ran on coal. Yours runs on inference costs, and it will not resolve itself.
The CloudZero read on the paradox is blunt: falling prices are not a cost strategy, they are a tailwind you only capture if you can see where consumption grows. The metric that survives the paradox is unit cost, spend per feature, per customer, per task, because unit cost tells you whether growing consumption is growing value or just growing.
What does LLM inference cost per token actually mean?
LLM inference cost per token is not one number. For any model it is at least four numbers, and the spread between them is where production bills are decided:
Standard input versus output rates
As above, output runs about 5x input. Cap max tokens per use case and instruct models toward brevity. Small percentage, every call, forever.
Cached input rates
This is the big one. Anthropic bills cache reads at 10% of the input rate, a 90% discount: Sonnet 4.6 input drops from $3 to $0.30 per million, and Opus 4.8 from $5 to $0.50. Cache writes cost 1.25x for a five-minute window or 2x for an hour, so a single hit already breaks even.
OpenAI caches automatically above 1,024 tokens, historically at 50% off, and its newest flagships now match the 90% read discount. Google Gemini discounts cached context 75% with a storage fee. The race to discount repeated context is over, and repeated context won.
Reasoning tokens
Models that think before answering bill the thinking as output, the expensive kind, and the user never sees it. If your bill jumped without a usage jump, check whether a default quietly flipped to a reasoning mode. Sometimes the depth earns its cost. Often nobody checked.
Batch rates
Both Anthropic and OpenAI cut prices 50% for asynchronous batch processing. “Does this actually need an instant answer?” is the cheapest question in AI architecture, and most pipelines answer it wrong by default. Stack batch on cached input and the combined discount approaches 95% off list.
Your effective LLM token cost is the blend of all four, weighted by architecture. Two teams on the identical model can pay effective rates 5x apart. Full vendor mechanics live in our OpenAI API cost per token guide.
What drives LLM inference cost up?
Five drivers, ordered by how often they explain a shocking invoice:
- Call multiplication from agents. A chatbot spends one call per question. An agent completing a task reasons, calls tools, verifies, and self-corrects: 10 to 20 model calls per user action. Teams that piloted with chat and shipped with agents watch consumption grow 10x with zero change in pricing or headcount. Our agentic AI cost analysis and Claude Code pricing guide both show this multiplier in the wild, and vendor benchmarks put heavy agentic users at $100 to $250 per person per month.
- Context length. Every token in the window is processed on every call. Retrieval pipelines that stuff documents, histories that never truncate, and agents that hoard state inflate input 3x to 5x without anyone deciding to spend more. Context is a budget. Most applications spend it like someone else’s money.
- Model selection. Routing a task to a frontier model instead of a capable workhorse raises its cost 10x to 20x, often for quality the use case cannot even display. Defaulting to the biggest model is the most expensive habit in production AI precisely because it never looks like a decision.
- Unbounded output. Verbose defaults, unset max token limits, and reasoning modes left on for easy tasks all inflate the expensive side of the meter. Nobody approves this spend because it arrives one wordy answer at a time, which is exactly why it compounds unwatched.
- Utilization, if you self-host. Own the GPUs and the meter flips: you pay for hardware time whether it generates or idles. A cluster at 30% utilization triples your effective GPU inference cost per token versus the same cluster kept busy.
Notice what is missing from the list: vendor rates. Rates set the floor. These five, all architectural and all yours, set the multiple above it. That is the core of how CloudZero thinks about AI inference cost: it behaves like cloud spend, not like a SaaS subscription, so it deserves cloud-grade engineering attention rather than procurement-grade annual reviews.
How do you reduce LLM inference cost?
LLM cost optimization comes down to eight levers. Ranked by effort against savings:
- Right-size the model per task. Route classification, extraction, and summarization to small fast models; reserve frontier models for work that measurably needs them. The highest-yield form of inference optimization there is: 60% to 90% savings on routed traffic, no hardware changes, no user-visible difference on fitted tasks.
- Route on cost, latency, and quality together. Good routers score all three; quality-only routing is how budgets die. Even a two-tier router, cheap model first with escalation on low confidence, captures most of the value and is a weekend of engineering.
- Cache aggressively, and structure prompts to be cacheable. Stable content (system prompts, tool definitions, reference documents) goes at the front where prefix matching works; variable content goes at the end. With 90% read discounts, prompt order alone can halve input spend. ProjectDiscovery’s 7% to 84% cache hit turnaround from the top of this article is exactly this lever, executed. The other published caching receipts tell the same story: one production agent went from $720 to $72 per month by adding three cache_control markers, and a team running 50,000 document analyses monthly reported $8,000 with caching against $45,000 without, an 82% cut on identical work. The gap is the norm, not the exception. Onboarding new customers, we routinely find organizations spending six figures monthly on APIs at 5% to 15% cache hit rates, when 80% or better is achievable. Free money, ignored at scale.
- Impose output discipline. Max token caps per use case, concise-answer instructions, and no paying output rates for pleasantries.
- Batch everything that can wait. Nightly summarization, bulk classification, evaluations: all of it belongs at 50% off. The only cost is admitting the work was never urgent.
- Trim context relentlessly. Truncate histories, retrieve fewer and better documents, stop resending state the model ignores. Teams auditing context typically find 30% to 50% of input tokens doing nothing for output quality.
- Quantize, distill, and serve smart if you self-host. Compressed models cover most workloads at a fraction of the compute, and LLM inference optimization techniques like continuous batching and paged attention push more tokens through the same silicon. NVIDIA’s engineering guides cover the serving stack; the budget owner’s job is asking whether anyone has applied it.
- Measure per feature before optimizing anything. The uncomfortable one. None of the seven levers above can be aimed without knowing which product, feature, or customer generates the spend. LLM inference optimization without allocation is guessing with confidence, and it reliably tunes the visible chatbot while the background pipeline burns 80% of the budget.
Run seriously, this list will reduce LLM costs 40% to 70% inside a quarter, and the first three levers usually fund the rest. Then hold the gains: a router tuned in March drifts by August as models and prices move underneath it. Efficiency here is an operating practice with an owner, not a project with an end date.
Should you self-host to cut inference costs?
Only if you can keep the hardware busy. Self-hosting trades a per-token meter for a per-hour one, and the entire business case lives in utilization. Sustained, predictable, high-volume traffic on a busy cluster beats API rates decisively. Spiky or modest traffic on idle GPUs loses to the API every time, and you inherit an ops team as a consolation prize.
Run the math before the migration, not after. Accelerator rates are the starting point: our H100 GPU cost breakdown and cloud GPU pricing comparison cover per-hour pricing across providers, and our SageMaker pricing guide covers the managed middle path, including serverless inference for spiky loads.
Then benchmark honestly: published LLM inference benchmark data measures tokens per second per GPU under your actual batch sizes and context lengths, and it is the difference between a plan and a hope. The AI inference economics here reward the boring answer. If volume is uncertain or engineering appetite is thin, stay on APIs and spend the energy on the eight levers above.
What changed in LLM inference cost in 2026?
If you follow LLM inference cost news, four developments this year actually matter for a budget:
The 90% cache discount became a standard, not a differentiator. What started as Anthropic’s signature pricing move is now matched by OpenAI’s newest flagships and approached by Google’s 75%. Translation for architects: prompt structure now converts directly into money on every major platform, no vendor loyalty required.
Agentic consumption became the dominant spend driver. The center of the cost conversation moved from “which model is cheapest” to “how many calls does your workflow chain,” because call multiplication swamps rate differences for most production teams. Whether all that autonomous work needs a human checkpoint is its own budget line; our human-in-the-loop cost analysis runs that math.
Reasoning-token billing went mainstream. Thinking models moved from research curiosity to default option, and their hidden output-rate billing is now a routine explanation for “our bill jumped and usage didn’t.”
Prices kept collapsing, unevenly. Per Epoch’s ongoing tracking, the steepest declines cluster at last year’s frontier as it commoditizes, while the newest capabilities hold premium rates. Budgeting lesson: yesterday’s expensive tier is today’s bargain, if your routing notices.
We refresh this section as the market moves. Profound’s citation research finds 40% to 60% of AI-cited domains change month over month, and cost guidance goes stale even faster than citations do.
How do you track LLM inference cost to the product level?
Start from the question no provider dashboard answers: which feature, team, or customer generated this spend, and was it worth it? API consoles report totals by key. Totals do not survive contact with a CFO who wants the margin impact of the new AI feature, in numbers, by Thursday.
Getting to answers takes three layers. Tag usage by feature and environment so consumption has an owner. Normalize across providers into one view, because multi-model stacks are the norm and every vendor reports differently. Then allocate to unit economics: inference spend per feature, per customer, per transaction, the denominators the business actually runs on.
This is CloudZero’s home turf. The platform ingests usage from OpenAI, Anthropic, and cloud AI services alongside AWS, Azure, and GCP, and allocates all of it to the products and teams that drive it. Inference-heavy businesses like Duolingo, Grammarly, and Upstart operate on this per-unit view, where “our AI bill grew 40%” resolves into “the new feature costs $0.11 per active user and retains them; ship more of it.” Same bill, opposite meeting.
Instrument before you scale, not after the invoice that finally gets a meeting. Our AI cost monitoring guide covers the setup, understanding AI costs covers the framework, and the broader AI cost optimization guide connects visibility to the savings levers above.
See your inference spend the way you ship: per feature
Token prices will keep falling, and your bill will keep not caring, because consumption is the variable now. CloudZero turns raw inference spend into cost per product, per feature, and per customer, so the price collapse everyone writes about actually shows up in your margins instead of vanishing into growth.
Request a demo to see inference cost in the units your business runs on: per feature, per customer, per task.
Frequently asked questions about LLM inference cost
