Skip to content

[Bug]: Custom input_cost_per_token_above_<N>_tokens tiers are silently dropped unless N is 128k/200k/272k #30344

Closed

Description

@ZeroClover

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

The tiered-pricing parser in _get_token_base_cost (litellm/litellm_core_utils/llm_cost_calc/utils.py) is written to handle an arbitrary threshold: it scans every key starting with input_cost_per_token_above_, parses the number out of the middle of the key name (key.split("_above_")[1].split("_tokens")[0], with optional k meaning times 1000), and applies that tier when prompt_tokens exceeds the parsed threshold. The docstring states this works for input_cost_per_token_above_[x]k_tokens or input_cost_per_token_above_[x]_tokens, implying any threshold is supported.

In practice only three thresholds ever reach that parser: 128k, 200k, 272k, and 512k. The reason is get_model_info in litellm/utils.py, which rebuilds the ModelInfo by copying a fixed allow-list of keys field by field. For input-token tiers it copies only input_cost_per_token_above_128k_tokens, input_cost_per_token_above_200k_tokens, and input_cost_per_token_above_272k_tokens (see litellm/utils.py:5908-5916; output tiers have the same fixed set at 5960-5971). Any other threshold you configure, for example input_cost_per_token_above_500k_tokens, is not in that list, so it never makes it onto the ModelInfo returned to the cost calculator. The tier is dropped without any warning and billing silently falls back to the base input_cost_per_token.

The effect is that the general-purpose parser is effectively dead code for any provider or custom model whose context-size tier boundary is not one of those three hard-coded values, even though that boundary is exactly the kind of thing the parser was meant to make dynamic. A user who registers a model with a 500k or 90k boundary gets undercharged (or overcharged) with no indication that the configured tier was ignored.

Expected: a configured input_cost_per_token_above_<N>_tokens value is honored for any N that the parser can read, so registering a model with a non-standard tier boundary applies that tier once prompt_tokens crosses it.

Actual: only 128k/200k/272k/512k survive get_model_info; every other threshold is dropped and the base per-token cost is used instead.

Steps to Reproduce

No real model call is needed; this exercises only the public cost-tracking API.

  1. Save the following as repro.py:
import litellm
from litellm.types.utils import Usage
from litellm.utils import get_model_info

litellm.register_model({
    "custom-tiered-model": {
        "litellm_provider": "openai",
        "mode": "chat",
        "input_cost_per_token": 1e-6,
        "output_cost_per_token": 2e-6,
        # Non-standard threshold (anything other than 128k/200k/272k):
        "input_cost_per_token_above_500k_tokens": 9e-6,
    }
})

usage = Usage(prompt_tokens=600_000, completion_tokens=10, total_tokens=600_010)
prompt_cost, _ = litellm.cost_calculator.cost_per_token(
    model="custom-tiered-model", custom_llm_provider="openai", usage_object=usage
)
per_token = prompt_cost / usage.prompt_tokens

info = get_model_info(model="custom-tiered-model", custom_llm_provider="openai")
print("configured key kept by get_model_info:",
      "input_cost_per_token_above_500k_tokens" in info
      and info.get("input_cost_per_token_above_500k_tokens") is not None)
print("expected per-token cost (500k tier): 9e-06")
print("actual   per-token cost:", per_token)
  1. Run python repro.py.
  2. Observe that get_model_info did not keep the configured 500k tier, and the per-token cost is the base 1e-06 instead of the configured 9e-06.

Relevant log output

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

1.89.0 rc2

Twitter / LinkedIn details

No response

Activity

  1. ylcnymn commented on Jun 14, 2026

    @ylcnymn

    Hi! I'd like to work on this issue. I've studied the root cause: get_model_info in litellm/utils.py has a fixed allow-list that only copies input_cost_per_token_above_128k_tokens, 200k, and 272k (lines ~5908-5916), while the downstream parser in _get_token_base_cost already supports arbitrary thresholds. The fix is to forward any input_cost_per_token_above_<N>_tokens key from the model config instead of enforcing a hard-coded allow-list.

    I just submitted a fix for #30358 (PR #30406) in the same codebase, so I'm familiar with the repo conventions. Could you assign this issue to me?

  2. added a commit that references this issue on Jun 24, 2026
    6bf3b9b
  3. added a commit that references this issue on Jun 26, 2026
    133da06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions