Repository navigation
Description
Check for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
The tiered-pricing parser in _get_token_base_cost (litellm/litellm_core_utils/llm_cost_calc/utils.py) is written to handle an arbitrary threshold: it scans every key starting with input_cost_per_token_above_, parses the number out of the middle of the key name (key.split("_above_")[1].split("_tokens")[0], with optional k meaning times 1000), and applies that tier when prompt_tokens exceeds the parsed threshold. The docstring states this works for input_cost_per_token_above_[x]k_tokens or input_cost_per_token_above_[x]_tokens, implying any threshold is supported.
In practice only three thresholds ever reach that parser: 128k, 200k, 272k, and 512k. The reason is get_model_info in litellm/utils.py, which rebuilds the ModelInfo by copying a fixed allow-list of keys field by field. For input-token tiers it copies only input_cost_per_token_above_128k_tokens, input_cost_per_token_above_200k_tokens, and input_cost_per_token_above_272k_tokens (see litellm/utils.py:5908-5916; output tiers have the same fixed set at 5960-5971). Any other threshold you configure, for example input_cost_per_token_above_500k_tokens, is not in that list, so it never makes it onto the ModelInfo returned to the cost calculator. The tier is dropped without any warning and billing silently falls back to the base input_cost_per_token.
The effect is that the general-purpose parser is effectively dead code for any provider or custom model whose context-size tier boundary is not one of those three hard-coded values, even though that boundary is exactly the kind of thing the parser was meant to make dynamic. A user who registers a model with a 500k or 90k boundary gets undercharged (or overcharged) with no indication that the configured tier was ignored.
Expected: a configured input_cost_per_token_above_<N>_tokens value is honored for any N that the parser can read, so registering a model with a non-standard tier boundary applies that tier once prompt_tokens crosses it.
Actual: only 128k/200k/272k/512k survive get_model_info; every other threshold is dropped and the base per-token cost is used instead.
Steps to Reproduce
No real model call is needed; this exercises only the public cost-tracking API.
- Save the following as
repro.py:
import litellm
from litellm.types.utils import Usage
from litellm.utils import get_model_info
litellm.register_model({
"custom-tiered-model": {
"litellm_provider": "openai",
"mode": "chat",
"input_cost_per_token": 1e-6,
"output_cost_per_token": 2e-6,
# Non-standard threshold (anything other than 128k/200k/272k):
"input_cost_per_token_above_500k_tokens": 9e-6,
}
})
usage = Usage(prompt_tokens=600_000, completion_tokens=10, total_tokens=600_010)
prompt_cost, _ = litellm.cost_calculator.cost_per_token(
model="custom-tiered-model", custom_llm_provider="openai", usage_object=usage
)
per_token = prompt_cost / usage.prompt_tokens
info = get_model_info(model="custom-tiered-model", custom_llm_provider="openai")
print("configured key kept by get_model_info:",
"input_cost_per_token_above_500k_tokens" in info
and info.get("input_cost_per_token_above_500k_tokens") is not None)
print("expected per-token cost (500k tier): 9e-06")
print("actual per-token cost:", per_token)- Run
python repro.py. - Observe that
get_model_infodid not keep the configured 500k tier, and the per-token cost is the base1e-06instead of the configured9e-06.
Relevant log output
What part of LiteLLM is this about?
SDK (litellm Python package)
What LiteLLM version are you on ?
1.89.0 rc2
Twitter / LinkedIn details
No response
Activity
ylcnymn commentedon Jun 14, 2026
Hi! I'd like to work on this issue. I've studied the root cause:
get_model_infoinlitellm/utils.pyhas a fixed allow-list that only copiesinput_cost_per_token_above_128k_tokens,200k, and272k(lines ~5908-5916), while the downstream parser in_get_token_base_costalready supports arbitrary thresholds. The fix is to forward anyinput_cost_per_token_above_<N>_tokenskey from the model config instead of enforcing a hard-coded allow-list.I just submitted a fix for #30358 (PR #30406) in the same codebase, so I'm familiar with the repo conventions. Could you assign this issue to me?
fix(utils): preserve arbitrary above-threshold tiered pricing keys in…
chore: litellm oss staging (#31185)