Skip to content

chore(release): backport #29493, #29983, #29984, #29986, #30160, #30202, #30327, #30220 to stable/1.87.x and cut 1.87.3 - #30405

Merged
yuneng-berri merged 10 commits into
stable/1.87.xfrom
litellm_backport_1_87_x_0613
Jun 14, 2026
Merged

yuneng-berri merged 10 commits into
stable/1.87.xfrom
litellm_backport_1_87_x_0613

Conversation

@yuneng-berri

Copy link
Copy Markdown
Contributor

Relevant issues

Backports the patch set that went into 1.84.8 onto stable/1.87.x, plus two additions specific to this line, and cuts 1.87.3. stable/1.87.x branched from staging on 2026-05-23 and already carried the two prerequisites these build on (#27921 and #28395), so the database-resilience picks land on the same foundation 1.84.8 used. Each change was checked against this line first: none were already present, and the buggy or missing code each one targets is present here, so every pick is needed.

The cluster: recover from PostgreSQL cached-plan errors by reconnecting the Prisma client (#29983); add an option to disable server-side prepared statements for DB lookups (#29984); return 5xx on DB infrastructure errors during auth and reserve 401 for genuine auth failures (#29986); resolve the passthrough costing model when the body model is "unknown" (#30160); skip [DONE] sentinels and non-JSON SSE frames in Anthropic streaming logging (the targeted hardening subset of #30202); return the deprecated-key lookup result directly in the get_data combined view so grace-period rotated keys stop 401-ing (#30327); add the disable_budget_reservation general setting (#29493); and bump the dashboard vitest toolchain plus pypdf, tornado and the aiohttp pin (#30220).

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

What is included

Cherry-picks in staging merge order, each carrying its -x provenance footer:

Then the dependency bump reproduced from #30220, the 1.87.2 -> 1.87.3 version bump, and a uv.lock / package-lock.json refresh.

Adaptation notes

stable/1.87.x has diverged from staging since late May, so several picks needed line-specific handling. None changes the fix's own production logic.

Known noise on this line

  • The changed test files carry pre-existing ruff findings (unused imports, == True style) that are present on the 1.87.x baseline as well; this PR introduces no new ones (the count is equal or lower per file).
  • The local proxy log shows Redis connection tracebacks from the spend-counter cache; that cache has no Redis configured in local dev. Those tracebacks are in untouched code (redis_cache.py, the proxy_server spend counters), are non-blocking, and appear on the clean baseline too. No traceback references any file this PR touches.

Screenshots / Proof of Fix

Run against a live proxy on the picked 1.87.3 code (claude-proxy on port 4010), hitting real provider APIs.

#30327 grace-period rotated key, the customer-reported symptom (old key 401-ing during the grace window). Before the fix this returns HTTP 401 from a TypeError: 'LiteLLM_VerificationTokenView' object is not subscriptable; here it returns 200:

# generate a key, then regenerate it with a 1h grace window
OLDKEY=$(curl -s "http://localhost:4010/key/generate" -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" -d '{"models":["claude-haiku-4-5"],"max_budget":1.0}' | jq -r .key)
curl -s -X POST "http://localhost:4010/key/regenerate" -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" -d "{\"key\":\"$OLDKEY\",\"grace_period\":\"1h\"}" >/dev/null

# use the OLD key during the grace window
curl -s -o /dev/null -w "HTTP %{http_code}\n" "http://localhost:4010/v1/models" -H "Authorization: Bearer $OLDKEY"
HTTP 200
HTTP 200   # used twice, per the original report

The proxy log shows zero not subscriptable occurrences on that path.

Passthrough logging fixes (#30160, SSE hardening), real Anthropic streaming through the proxy, including a non-JSON ping frame that the hardening must tolerate:

curl -s "http://localhost:4010/anthropic/v1/messages" -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" -H "anthropic-version: 2023-06-01" \
  -d '{"model":"claude-haiku-4-5","max_tokens":20,"stream":true,"messages":[{"role":"user","content":"say hi in one word"}]}'
event: message_start
event: content_block_start
event: ping
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hi"}}
event: content_block_stop
event: message_delta

The stream completes and the logging handler raises no JSONDecode error on the ping / [DONE] frames. A baseline real completion (claude-haiku-4-5) also returns cleanly.

Targeted test delta on this line: 188 passing at baseline, 240 passing after the picks, 0 new failures. The increase is the regression tests each pick brought (the #29986 503/401 builder tests, the #30160 cost-resolution tests, the SSE hardening tests, the #29984 CLI tests, the #29493 budget tests).

An adversarial multi-agent review (deep, two rounds) returned SURVIVED, confirming among other things that the #29986 503 branch is ordered after the 401 branches so a genuine authentication failure is never reclassified as 5xx, that #29983's attempt_db_reconnect pre-exists on this line, and that the #30327 direct-return short-circuits the crash (reverting it in place reproduces the raise).

Type

ðŸ§đ Refactoring

Changes

Targeted backport across the database-resilience, passthrough-logging, budget and dependency areas, cutting 1.87.3.

Ar-maan05 and others added 10 commits June 13, 2026 16:07
â€Ķ29493)

* feat(proxy): add disable_budget_reservation general setting (#27639)

* feat(proxy): register disable_budget_reservation in ConfigGeneralSettings (#27639)

* docs(proxy): document disable_budget_reservation concurrency tradeoff (#27639)

* ci: re-trigger flaky docker build (prisma generate ECONNRESET)

* fix(proxy): warn and document budget enforcement tradeoff when disable_budget_reservation is set (#27639)

(cherry picked from commit 1032dd7)
â€Ķr genuine auth failures (#29986)

(cherry picked from commit da9d64b)
â€Ķthropic streaming logging

Targeted subset of staging commit cfcdf87 (#30202): only the
anthropic_passthrough_logging_handler.py hardening hunks and their four
tests are taken; the rest of that staging batch is intentionally excluded.

(cherry picked from commit cfcdf87)
(cherry picked from commit 973c7eb)
â€Ķcombined view (#30327)

The grace-period branch assigned the recursive get_data result (a
finished LiteLLM_VerificationTokenView) back into the variable that the
combined-view dict normalization then subscripts, raising TypeError on
every request made with a rotated key inside its grace window; auth
surfaced that as a 401. Return the recursive result directly instead.

Regression test drives the full get_data flow: old hash misses the view,
deprecated table resolves to the active token, and the call must return
the view object

(cherry picked from commit 5047eaf)
â€Ķstable/1.87.x

Backport of #30220 dependency bumps adapted to this line: pypdf pinned to
6.13.1 (this line exact-pins these dependencies), tornado and aiohttp held via
[tool.uv] constraint-dependencies (tornado>=6.5.6; aiohttp>=3.13.5,<3.14 so a
lock regeneration cannot move onto 3.14 while vcrpy is incompatible), and the
dashboard vitest devDependencies raised 3.2.4 -> 3.2.6. Locks refreshed in a
follow-up commit.
uv.lock: litellm 1.87.3, pypdf 6.13.1, tornado 6.5.7 (aiohttp held at 3.13.5).
package-lock.json: vitest family 3.2.6 and brace-expansion 5.0.6.
@yuneng-berri
yuneng-berri requested a review from a team June 14, 2026 00:07
@CLAassistant

CLAassistant commented Jun 14, 2026 •

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 4 committers have signed the CLA.

✅ Ar-maan05
✅ yuneng-berri
✅ Sameerlite
❌ yassin-berriai
You have signed the CLA already but the status is still pending? Let us recheck it.

@yuneng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

@codecov

codecov Bot commented Jun 14, 2026 •

Copy link
Copy Markdown

@greptile-apps

greptile-apps Bot commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This is a targeted backport onto stable/1.87.x cutting version 1.87.3. It ports a cluster of database-resilience, passthrough-logging, and budget fixes from recent staging work, along with a dependency refresh.

Confidence Score: 4/5

The core production fixes are correct and well-tested; two minor issues in the new exception-classification helpers do not affect the happy path.

Both findings are in exception_handler.py: the broad OSError catch could silently convert non-DB auth failures into 503s, and the traceback-frame walk in is_prisma_engine_internal_error is coupled to prisma-client-py internal module layout. Neither blocks the critical fixes and both are limited to the already-failing auth-error path.

litellm/proxy/db/exception_handler.py — the two newly added static methods are the main area to revisit if exception classification behaves unexpectedly after a prisma version bump.

Important Files Changed

Filename Overview
litellm/proxy/auth/auth_exception_handler.py Adds 503 DB-unavailable branch after ProxyException re-raise; ordering is correct so genuine 401s are not reclassified, but the OSError check in the backing method is broader than necessary.
litellm/proxy/db/exception_handler.py Adds is_database_service_unavailable_error and is_prisma_engine_internal_error; is_prisma_engine_internal_error relies on traceback frame inspection against prisma.engine.* module names, which is brittle against prisma internal restructuring.
litellm/proxy/utils.py Fixes deprecated-key grace-period path (direct return bypasses the dict-subscript crash on LiteLLM_VerificationTokenView); improves cached-plan recovery to reconnect via attempt_db_reconnect instead of mutating the query string.
litellm/proxy/auth/user_api_key_auth.py Threads general_settings through to _reserve_budget_after_common_checks; only one call site exists so no risk of missed callers; disable_budget_reservation early-return is well-guarded.
litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py Resolves unknown costing model from litellm_params/model_group/SSE message_start; skips [DONE] sentinels and catches json.JSONDecodeError to harden streaming logging pipeline.
litellm/proxy/proxy_cli.py Adds disable_prepared_statements parameter to _build_db_connection_url_params; reads database_disable_prepared_statements from general_settings with proper string-to-bool coercion; extra_params still wins over the flag.
litellm/proxy/_types.py Adds database_disable_prepared_statements and disable_budget_reservation Optional[bool] fields to ConfigGeneralSettings with detailed docstrings.
tests/test_litellm/proxy/db/test_exception_handler.py Comprehensive regression suite for the new is_database_service_unavailable_error; covers prisma P1001-masquerade, engine teardown AttributeError, cached-plan path, asyncpg injection, and application-AttributeError exclusion.
tests/test_litellm/proxy/auth/test_auth_exception_handler.py Adds three parametrized test blocks verifying 503 for infra errors, 503 for prisma engine teardown AttributeError, and 401 preserved for genuine auth failures.
tests/test_litellm/proxy/auth/test_user_api_key_auth.py Adds disable_budget_reservation tests; updates existing test to pass new general_settings={}; end-to-end builder tests for 503/401 paths; no existing assertions are weakened.
pyproject.toml Version bump 1.87.2 to 1.87.3; pypdf pinned to 6.13.1; tornado and aiohttp added as uv constraint-dependencies.

Reviews (1): Last reviewed commit: "chore: refresh uv.lock and dashboard pac..." | Re-trigger Greptile

Comment on lines +183 to +184
return True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Overly broad OSError catch may misclassify non-DB auth failures as 503

OSError is the parent of ConnectionError and TimeoutError (the intended targets) but also of FileNotFoundError, PermissionError, IsADirectoryError, and others. If one of those surfaces during auth — for example, a JWKS/cert file that can't be opened — this check returns True and the caller emits a 503 "authentication database temporarily unreachable" instead of a more meaningful error. The comment in the method ("OSError already covers ConnectionError and TimeoutError") is accurate but doesn't acknowledge that the match is wider than those two. Narrowing to (ConnectionError, TimeoutError, asyncio.TimeoutError) would preserve the intent without the false-positive surface.

Comment on lines +134 to +140
tb = getattr(e, "__traceback__", None)
while tb is not None:
if tb.tb_frame.f_globals.get("__name__", "").startswith("prisma.engine"):
return True
tb = tb.tb_next
return False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Traceback-frame inspection couples detection to prisma-client-py internal module layout

The method walks the live traceback and accepts any frame whose __name__ starts with "prisma.engine". This works for the current prisma-client-py release, but a minor refactor in the library (module rename, split into sub-packages, etc.) would silently break detection — the AttributeError would fall through to 401 again during DB teardown, reproducing the original outage edge case. A note in the docstring naming the prisma-client-py version range this was verified against, or a version pin in the test, would make the brittleness explicit and easier to revisit when upgrading prisma.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@greptile-apps

greptile-apps Bot commented Jun 14, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This is a targeted backport to stable/1.87.x cutting version 1.87.3. It cherry-picks a set of database-resilience, passthrough-logging, budget, and dependency fixes from staging, with line-specific adaptation where test infrastructure or generated files diverged.

  • DB resilience (auth path): get_data now returns the deprecated-key LiteLLM_VerificationTokenView directly instead of letting it fall through to dict normalization (fixing the grace-period TypeError); cached-plan errors reconnect the Prisma client via attempt_db_reconnect (singleflight) instead of injecting a SQL comment; a new is_database_service_unavailable_error classifier upgrades all infrastructure-level DB failures during auth from 401 to 503.
  • Passthrough logging / costing: _resolve_costing_model resolves the \"unknown\" sentinel from litellm_params or model_group; _extract_model_from_anthropic_chunks recovers the model from the message_start SSE event; [DONE] sentinels and non-JSON SSE frames are now skipped without crashing the logging pipeline.
  • Operator escape hatches: database_disable_prepared_statements adds pgbouncer=true to the Prisma URL for pgBouncer transaction-pooling deployments; disable_budget_reservation lets operators opt out of optimistic per-request budget reservation if they encounter phantom BudgetExceededErrors.

Confidence Score: 4/5

The backport is safe to merge; every fix targets a confirmed production regression with tests, proof-of-fix output, and adversarial review backing.

Every fix targets a confirmed production regression with tests and proof-of-fix output. The auth exception ordering (503 check after ProxyException re-raise) is correct. Two minor observations: the disable_budget_reservation warning fires on every request (potentially noisy at scale) and the OSError catch in is_database_service_unavailable_error is broad, though the context makes misclassification unlikely in practice.

litellm/proxy/db/exception_handler.py and litellm/proxy/auth/user_api_key_auth.py contain the new classification and budget-reservation logic worth a second look.

Important Files Changed

Filename Overview
litellm/proxy/auth/auth_exception_handler.py Adds a 503 branch after the ProxyException re-raise so DB infrastructure errors surface correctly instead of falling through to 401; placement and ordering are correct.
litellm/proxy/auth/user_api_key_auth.py Adds required general_settings param to _reserve_budget_after_common_checks and wires up disable_budget_reservation; the only production call site is updated and tests cover the flag. Warning is logged per-request, which can be noisy at scale.
litellm/proxy/db/exception_handler.py Adds is_prisma_engine_internal_error (traceback-walk) and is_database_service_unavailable_error (composite classifier); OSError catch is broad but reasonable in context.
litellm/proxy/utils.py Fixes the deprecated-key grace-period crash and replaces the naive SQL-comment cache-bust with a singleflight Prisma client reconnect.
litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py Adds costing model resolution for unknown-model passthrough; [DONE] and non-JSON SSE frames are now skipped safely.
litellm/proxy/proxy_cli.py database_disable_prepared_statements wiring: bool/str coercion, pgbouncer=true URL param, extra_params override precedence all correct.
litellm/proxy/_types.py Adds two Optional[bool] fields to ConfigGeneralSettings with descriptive docs; no schema regressions.
pyproject.toml Version bump 1.87.2 to 1.87.3; pypdf pin updated; tornado and aiohttp lower-bounds added as constraint-dependencies.

Comments Outside Diff (1)

  1. litellm/proxy/auth/user_api_key_auth.py, line 92-101 (link)

    P2 Warning logged on every request when flag is active

    verbose_proxy_logger.warning(...) emits a line on every single request while disable_budget_reservation=True. At high throughput (e.g., 1 000 req/s) this generates millions of warning-level log lines per hour, potentially crowding out signals from real issues and filling log buffers. The PR description acknowledges this is intentional as a reminder, but a one-shot warning at startup (or a per-minute rate-limited log) would achieve the same goal without the per-request cost.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Reviews (2): Last reviewed commit: "chore: refresh uv.lock and dashboard pac..." | Re-trigger Greptile

Comment on lines +182 to +183
if isinstance(e, (OSError, asyncio.TimeoutError)):
return True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 OSError catch is broad

isinstance(e, OSError) covers the intended cases (ConnectionError, TimeoutError which are OSError subclasses in Python 3.3+) but also catches FileNotFoundError, PermissionError, BlockingIOError, and every other OSError subclass. In the auth handler this is very unlikely to matter — OSError subclasses from non-network sources rarely reach here — but if any auth-path middleware (logging, telemetry, config file reads) raises a FileNotFoundError, it would be silently upgraded to a 503 instead of exposing the real bug. A more targeted check would limit the net to the network-error subclasses explicitly.

@yuneng-berri
yuneng-berri merged commit 5c8b847 into stable/1.87.x Jun 14, 2026
72 of 76 checks passed
@yuneng-berri
yuneng-berri deleted the litellm_backport_1_87_x_0613 branch June 14, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants