Repository navigation
chore(release): backport #29493, #29983, #29984, #29986, #30160, #30202, #30327, #30220 to stable/1.87.x and cut 1.87.3 - #30405
Conversation
âĶ29493) * feat(proxy): add disable_budget_reservation general setting (#27639) * feat(proxy): register disable_budget_reservation in ConfigGeneralSettings (#27639) * docs(proxy): document disable_budget_reservation concurrency tradeoff (#27639) * ci: re-trigger flaky docker build (prisma generate ECONNRESET) * fix(proxy): warn and document budget enforcement tradeoff when disable_budget_reservation is set (#27639) (cherry picked from commit 1032dd7)
âĶthropic streaming logging Targeted subset of staging commit cfcdf87 (#30202): only the anthropic_passthrough_logging_handler.py hardening hunks and their four tests are taken; the rest of that staging batch is intentionally excluded. (cherry picked from commit cfcdf87) (cherry picked from commit 973c7eb)
âĶcombined view (#30327) The grace-period branch assigned the recursive get_data result (a finished LiteLLM_VerificationTokenView) back into the variable that the combined-view dict normalization then subscripts, raising TypeError on every request made with a rotated key inside its grace window; auth surfaced that as a 401. Return the recursive result directly instead. Regression test drives the full get_data flow: old hash misses the view, deprecated table resolves to the active token, and the call must return the view object (cherry picked from commit 5047eaf)
âĶstable/1.87.x Backport of #30220 dependency bumps adapted to this line: pypdf pinned to 6.13.1 (this line exact-pins these dependencies), tornado and aiohttp held via [tool.uv] constraint-dependencies (tornado>=6.5.6; aiohttp>=3.13.5,<3.14 so a lock regeneration cannot move onto 3.14 while vcrpy is incompatible), and the dashboard vitest devDependencies raised 3.2.4 -> 3.2.6. Locks refreshed in a follow-up commit.
uv.lock: litellm 1.87.3, pypdf 6.13.1, tornado 6.5.7 (aiohttp held at 3.13.5). package-lock.json: vitest family 3.2.6 and brace-expansion 5.0.6.
|
|
Codecov Reportâ Patch coverage is ðĒ Thoughts on this report? Let us know! |
Greptile SummaryThis is a targeted backport onto
Confidence Score: 4/5The core production fixes are correct and well-tested; two minor issues in the new exception-classification helpers do not affect the happy path. Both findings are in exception_handler.py: the broad OSError catch could silently convert non-DB auth failures into 503s, and the traceback-frame walk in is_prisma_engine_internal_error is coupled to prisma-client-py internal module layout. Neither blocks the critical fixes and both are limited to the already-failing auth-error path. litellm/proxy/db/exception_handler.py â the two newly added static methods are the main area to revisit if exception classification behaves unexpectedly after a prisma version bump.
|
| Filename | Overview |
|---|---|
| litellm/proxy/auth/auth_exception_handler.py | Adds 503 DB-unavailable branch after ProxyException re-raise; ordering is correct so genuine 401s are not reclassified, but the OSError check in the backing method is broader than necessary. |
| litellm/proxy/db/exception_handler.py | Adds is_database_service_unavailable_error and is_prisma_engine_internal_error; is_prisma_engine_internal_error relies on traceback frame inspection against prisma.engine.* module names, which is brittle against prisma internal restructuring. |
| litellm/proxy/utils.py | Fixes deprecated-key grace-period path (direct return bypasses the dict-subscript crash on LiteLLM_VerificationTokenView); improves cached-plan recovery to reconnect via attempt_db_reconnect instead of mutating the query string. |
| litellm/proxy/auth/user_api_key_auth.py | Threads general_settings through to _reserve_budget_after_common_checks; only one call site exists so no risk of missed callers; disable_budget_reservation early-return is well-guarded. |
| litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py | Resolves unknown costing model from litellm_params/model_group/SSE message_start; skips [DONE] sentinels and catches json.JSONDecodeError to harden streaming logging pipeline. |
| litellm/proxy/proxy_cli.py | Adds disable_prepared_statements parameter to _build_db_connection_url_params; reads database_disable_prepared_statements from general_settings with proper string-to-bool coercion; extra_params still wins over the flag. |
| litellm/proxy/_types.py | Adds database_disable_prepared_statements and disable_budget_reservation Optional[bool] fields to ConfigGeneralSettings with detailed docstrings. |
| tests/test_litellm/proxy/db/test_exception_handler.py | Comprehensive regression suite for the new is_database_service_unavailable_error; covers prisma P1001-masquerade, engine teardown AttributeError, cached-plan path, asyncpg injection, and application-AttributeError exclusion. |
| tests/test_litellm/proxy/auth/test_auth_exception_handler.py | Adds three parametrized test blocks verifying 503 for infra errors, 503 for prisma engine teardown AttributeError, and 401 preserved for genuine auth failures. |
| tests/test_litellm/proxy/auth/test_user_api_key_auth.py | Adds disable_budget_reservation tests; updates existing test to pass new general_settings={}; end-to-end builder tests for 503/401 paths; no existing assertions are weakened. |
| pyproject.toml | Version bump 1.87.2 to 1.87.3; pypdf pinned to 6.13.1; tornado and aiohttp added as uv constraint-dependencies. |
Reviews (1): Last reviewed commit: "chore: refresh uv.lock and dashboard pac..." | Re-trigger Greptile
| return True | ||
|
|
There was a problem hiding this comment.
Overly broad OSError catch may misclassify non-DB auth failures as 503
OSError is the parent of ConnectionError and TimeoutError (the intended targets) but also of FileNotFoundError, PermissionError, IsADirectoryError, and others. If one of those surfaces during auth â for example, a JWKS/cert file that can't be opened â this check returns True and the caller emits a 503 "authentication database temporarily unreachable" instead of a more meaningful error. The comment in the method ("OSError already covers ConnectionError and TimeoutError") is accurate but doesn't acknowledge that the match is wider than those two. Narrowing to (ConnectionError, TimeoutError, asyncio.TimeoutError) would preserve the intent without the false-positive surface.
| tb = getattr(e, "__traceback__", None) | ||
| while tb is not None: | ||
| if tb.tb_frame.f_globals.get("__name__", "").startswith("prisma.engine"): | ||
| return True | ||
| tb = tb.tb_next | ||
| return False | ||
|
|
There was a problem hiding this comment.
Traceback-frame inspection couples detection to prisma-client-py internal module layout
The method walks the live traceback and accepts any frame whose __name__ starts with "prisma.engine". This works for the current prisma-client-py release, but a minor refactor in the library (module rename, split into sub-packages, etc.) would silently break detection â the AttributeError would fall through to 401 again during DB teardown, reproducing the original outage edge case. A note in the docstring naming the prisma-client-py version range this was verified against, or a version pin in the test, would make the brittleness explicit and easier to revisit when upgrading prisma.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Greptile SummaryThis is a targeted backport to
Confidence Score: 4/5The backport is safe to merge; every fix targets a confirmed production regression with tests, proof-of-fix output, and adversarial review backing. Every fix targets a confirmed production regression with tests and proof-of-fix output. The auth exception ordering (503 check after ProxyException re-raise) is correct. Two minor observations: the disable_budget_reservation warning fires on every request (potentially noisy at scale) and the OSError catch in is_database_service_unavailable_error is broad, though the context makes misclassification unlikely in practice. litellm/proxy/db/exception_handler.py and litellm/proxy/auth/user_api_key_auth.py contain the new classification and budget-reservation logic worth a second look.
|
| Filename | Overview |
|---|---|
| litellm/proxy/auth/auth_exception_handler.py | Adds a 503 branch after the ProxyException re-raise so DB infrastructure errors surface correctly instead of falling through to 401; placement and ordering are correct. |
| litellm/proxy/auth/user_api_key_auth.py | Adds required general_settings param to _reserve_budget_after_common_checks and wires up disable_budget_reservation; the only production call site is updated and tests cover the flag. Warning is logged per-request, which can be noisy at scale. |
| litellm/proxy/db/exception_handler.py | Adds is_prisma_engine_internal_error (traceback-walk) and is_database_service_unavailable_error (composite classifier); OSError catch is broad but reasonable in context. |
| litellm/proxy/utils.py | Fixes the deprecated-key grace-period crash and replaces the naive SQL-comment cache-bust with a singleflight Prisma client reconnect. |
| litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py | Adds costing model resolution for unknown-model passthrough; [DONE] and non-JSON SSE frames are now skipped safely. |
| litellm/proxy/proxy_cli.py | database_disable_prepared_statements wiring: bool/str coercion, pgbouncer=true URL param, extra_params override precedence all correct. |
| litellm/proxy/_types.py | Adds two Optional[bool] fields to ConfigGeneralSettings with descriptive docs; no schema regressions. |
| pyproject.toml | Version bump 1.87.2 to 1.87.3; pypdf pin updated; tornado and aiohttp lower-bounds added as constraint-dependencies. |
Comments Outside Diff (1)
-
litellm/proxy/auth/user_api_key_auth.py, line 92-101 (link)Warning logged on every request when flag is active
verbose_proxy_logger.warning(...)emits a line on every single request whiledisable_budget_reservation=True. At high throughput (e.g., 1 000 req/s) this generates millions of warning-level log lines per hour, potentially crowding out signals from real issues and filling log buffers. The PR description acknowledges this is intentional as a reminder, but a one-shot warning at startup (or a per-minute rate-limited log) would achieve the same goal without the per-request cost.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Reviews (2): Last reviewed commit: "chore: refresh uv.lock and dashboard pac..." | Re-trigger Greptile
| if isinstance(e, (OSError, asyncio.TimeoutError)): | ||
| return True |
There was a problem hiding this comment.
isinstance(e, OSError) covers the intended cases (ConnectionError, TimeoutError which are OSError subclasses in Python 3.3+) but also catches FileNotFoundError, PermissionError, BlockingIOError, and every other OSError subclass. In the auth handler this is very unlikely to matter â OSError subclasses from non-network sources rarely reach here â but if any auth-path middleware (logging, telemetry, config file reads) raises a FileNotFoundError, it would be silently upgraded to a 503 instead of exposing the real bug. A more targeted check would limit the net to the network-error subclasses explicitly.
Relevant issues
Backports the patch set that went into 1.84.8 onto stable/1.87.x, plus two additions specific to this line, and cuts 1.87.3. stable/1.87.x branched from staging on 2026-05-23 and already carried the two prerequisites these build on (#27921 and #28395), so the database-resilience picks land on the same foundation 1.84.8 used. Each change was checked against this line first: none were already present, and the buggy or missing code each one targets is present here, so every pick is needed.
The cluster: recover from PostgreSQL cached-plan errors by reconnecting the Prisma client (#29983); add an option to disable server-side prepared statements for DB lookups (#29984); return 5xx on DB infrastructure errors during auth and reserve 401 for genuine auth failures (#29986); resolve the passthrough costing model when the body model is "unknown" (#30160); skip [DONE] sentinels and non-JSON SSE frames in Anthropic streaming logging (the targeted hardening subset of #30202); return the deprecated-key lookup result directly in the get_data combined view so grace-period rotated keys stop 401-ing (#30327); add the disable_budget_reservation general setting (#29493); and bump the dashboard vitest toolchain plus pypdf, tornado and the aiohttp pin (#30220).
Linear ticket
Pre-Submission checklist
make test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewWhat is included
Cherry-picks in staging merge order, each carrying its
-xprovenance footer:Then the dependency bump reproduced from #30220, the 1.87.2 -> 1.87.3 version bump, and a uv.lock / package-lock.json refresh.
Adaptation notes
stable/1.87.x has diverged from staging since late May, so several picks needed line-specific handling. None changes the fix's own production logic.
seed_request_identityhelper that newer staging tests@patch, so those patch entries were dropped from the picked tests (there is nothing to neutralize here); a cosmetic reformat hunk that targeted a staging-only test method absent on this line was also dropped. The fix's two source files and its three DB-error regression tests are intact.test_prisma_client_get_data.pyand its conftest) that was introduced on staging after this line branched and is absent here; bringing it would pull a large unrelated test-infra module. The source fixes apply cleanly and are proven below by the live grace-period reproducer and by the gauntlet.ui/litellm-dashboard/src/lib/http/schema.d.tshunk (that generated file is not present on this line). The Python option, CLI wiring and CLI tests are intact.[tool.uv] constraint-dependenciesblock (tornado>=6.5.6; aiohttp>=3.13.5,<3.14 so a lock regeneration cannot move onto 3.14 while vcrpy is incompatible); the dashboard vitest family moves 3.2.4 -> 3.2.6 and brace-expansion 5.0.5 -> 5.0.6. uv.lock and package-lock.json were regenerated on this line; uv.lock churn is limited to litellm, pypdf and tornado, and package-lock churn to the vitest family and brace-expansion.Known noise on this line
== Truestyle) that are present on the 1.87.x baseline as well; this PR introduces no new ones (the count is equal or lower per file).redis_cache.py, the proxy_server spend counters), are non-blocking, and appear on the clean baseline too. No traceback references any file this PR touches.Screenshots / Proof of Fix
Run against a live proxy on the picked 1.87.3 code (
claude-proxyon port 4010), hitting real provider APIs.#30327 grace-period rotated key, the customer-reported symptom (old key 401-ing during the grace window). Before the fix this returns HTTP 401 from a
TypeError: 'LiteLLM_VerificationTokenView' object is not subscriptable; here it returns 200:The proxy log shows zero
not subscriptableoccurrences on that path.Passthrough logging fixes (#30160, SSE hardening), real Anthropic streaming through the proxy, including a non-JSON
pingframe that the hardening must tolerate:The stream completes and the logging handler raises no JSONDecode error on the
ping/[DONE]frames. A baseline real completion (claude-haiku-4-5) also returns cleanly.Targeted test delta on this line: 188 passing at baseline, 240 passing after the picks, 0 new failures. The increase is the regression tests each pick brought (the #29986 503/401 builder tests, the #30160 cost-resolution tests, the SSE hardening tests, the #29984 CLI tests, the #29493 budget tests).
An adversarial multi-agent review (deep, two rounds) returned SURVIVED, confirming among other things that the #29986 503 branch is ordered after the 401 branches so a genuine authentication failure is never reclassified as 5xx, that #29983's
attempt_db_reconnectpre-exists on this line, and that the #30327 direct-return short-circuits the crash (reverting it in place reproduces the raise).Type
ð§đ Refactoring
Changes
Targeted backport across the database-resilience, passthrough-logging, budget and dependency areas, cutting 1.87.3.