Skip to content

feat: package Rust OCR bridge in LiteLLM wheel - #31267

Merged
6 commits merged into
litellm_internal_stagingfrom
litellm_hotfix_rust_pip_binary
Jun 25, 2026
Merged

6 commits merged into
litellm_internal_stagingfrom
litellm_hotfix_rust_pip_binary

Conversation

@ghost

@ghost ghost commented Jun 25, 2026 •

Copy link
Copy Markdown

What changed

Packages the existing Rust OCR bridge inside the default litellm pip wheel instead of creating a separate Rust wheel package.

  • Build backend changes from uv_build to maturin so the PyO3 crate can be built as litellm.rust_bridge._native.
  • uv build still works as the build frontend; the previous uv_build source excludes were ported to [tool.maturin].exclude.
  • The public wheel/sdist exclude litellm/proxy/enterprise with both the bare directory and recursive /** pattern.
  • Docker images install Rust only in builder stages; runtime images do not need Rust/Cargo.
  • OCR Rust path remains opt-in and falls back to Python when the native bridge is unavailable.

Related install-report issue: #31261

Why this shape

We looked at the Pydantic/Polars packaging pattern: ship the native Rust extension inside the default Python package, with maturin/PyO3 building platform wheels. That avoids a second LiteLLM package for users while still letting us split the bridge later if needed because Python calls it through litellm.rust_bridge.

Validation

  • uv lock
  • uv build --sdist --wheel
  • Artifact inspection confirmed:
    • wheel includes litellm/rust_bridge/_native...so
    • wheel/sdist include litellm/rust_bridge/loader.py
    • wheel/sdist do not include litellm/proxy/enterprise
  • uv run black litellm/rust_bridge litellm/ocr/rust_bridge.py tests/test_litellm/ocr/test_rust_bridge.py
  • uv run ruff check litellm/rust_bridge litellm/ocr/rust_bridge.py tests/test_litellm/ocr/test_rust_bridge.py
  • uv run pytest tests/test_litellm/ocr/test_rust_bridge.py -q (19 passed)
  • (cd litellm-rust && cargo fmt --check && cargo clippy --workspace --all-targets --locked -- -D warnings && cargo test --workspace --locked)
  • Main Docker build + import smoke passed
  • Non-root Docker build + import smoke passed
  • CircleCI using_litellm_on_windows passed after adding Rust to that Windows build job

Review status

  • Greptile flagged the missing litellm/proxy/enterprise/** exclude and missing negative-cache in the native loader; both are fixed in the latest commit and covered by artifact inspection/tests.
  • Veria AI passed with No security issues found.
  • CI is green on the latest PR head.

@ghost
ghost self-requested a review June 25, 2026 03:04
@CLAassistant

CLAassistant commented Jun 25, 2026 •

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Ishaan Jaff seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codecov

codecov Bot commented Jun 25, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.90909% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
litellm/rust_bridge/loader.py 87.50% 2 Missing ⚠ïļ

ðŸ“Ē Thoughts on this report? Let us know!

@greptile-apps

This comment has been minimized.

Comment thread pyproject.toml Resolved
Comment thread litellm/rust_bridge/loader.py Resolved
@ghost

ghost commented Jun 25, 2026

Copy link
Copy Markdown
Author

Requesting fresh bot review on latest head 3ac48285ee40d567c046f9a32321d50c8d14e510.

Greptile findings have been addressed:

  • Added litellm/proxy/enterprise/** to [tool.maturin].exclude and artifact-checked that wheel/sdist do not include litellm/proxy/enterprise.
  • Added native-bridge negative-cache and a regression test for the missing-extension path.

Veria status on latest head: passing, No security issues found, zero annotations.

@greptile-apps please re-review the latest head.

Comment thread pyproject.toml Resolved
@ghost

ghost commented Jun 25, 2026

Copy link
Copy Markdown
Author

@greptile review

Comment thread .circleci/config.yml Resolved
yuneng-berri
yuneng-berri previously approved these changes Jun 25, 2026
@yuneng-berri
yuneng-berri dismissed their stale review June 25, 2026 18:07

approved by mistake

@ghost
ghost force-pushed the litellm_hotfix_rust_pip_binary branch 2 times, most recently from c67e1d0 to 2ae93b1 Compare June 25, 2026 18:23
@ghost
ghost requested a review from yuneng-berri June 25, 2026 18:29
@ghost
ghost enabled auto-merge (squash) June 25, 2026 18:58
@ghost
ghost force-pushed the litellm_hotfix_rust_pip_binary branch from 70842d8 to ad4dff1 Compare June 25, 2026 19:08
@ghost

ghost commented Jun 25, 2026

Copy link
Copy Markdown
Author

CI is green on final head ad4dff15f0 after rebasing onto current litellm_internal_staging. The previous flaky llm_responses_api_testing job passed, along with Windows install, pip build, Docker DB image, lint, OSV, and Rust checks.\n\nMerge is currently blocked only by the required BerriAI/litellm-engineering approval rule after the rebase/lockfile fix refreshed the head commit.

@ghost
ghost self-requested a review June 25, 2026 19:27
@ghost
ghost merged commit d8ef1da into litellm_internal_staging Jun 25, 2026
124 checks passed
@ghost
ghost deleted the litellm_hotfix_rust_pip_binary branch June 25, 2026 19:32
google-tim pushed a commit to google-tim/litellm that referenced this pull request Jun 26, 2026
BerriAI#31267 switched the build backend to maturin to bundle the Rust OCR
bridge into the litellm wheel. On the release runner, maturin builds
with `--compatibility off` and emits a `cp312-cp312-linux_x86_64`
wheel. PyPI only accepts `manylinux*` Linux wheels, so the upload is
rejected with `400 Bad Request - unsupported platform tag
'linux_x86_64'`, which blocked the 1.91.0.dev2 publish.

Revert `[build-system]` to the pure-Python `uv_build` backend and
restore `[tool.uv.build-backend]` so the release once again produces a
universal `litellm-<ver>-py3-none-any.whl` that installs on every
OS / arch / Python.

The Rust bridge stays optional: `litellm/rust_bridge/loader.py` already
falls back gracefully when `_native` is absent (`try/except
ImportError`), so runtime behavior is unchanged - the native module is
simply not bundled. Rust acceleration can ship as a separate
platform-specific package later.
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants