# LiteLLM

> LiteLLM is an open-source (MIT) Python SDK and AI Gateway that give 100+ LLM providers one OpenAI-compatible API. The SDK is a library for Python apps; the gateway is a server you run in your own environment, with virtual keys, budgets, spend tracking, routing and fallbacks, guardrails, and an MCP gateway, that apps in any language call by changing their base URL.

- Which one: use the AI Gateway when several apps, teams, users, or coding agents (Claude Code, Codex, Cursor) share models and need keys, budgets, spend tracking, or a client that is not Python. Use the Python SDK when one Python app calls models directly.
- Start the gateway: `curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/quickstart.sh | sh` (macOS, Linux) or `irm https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/quickstart.ps1 | iex` (Windows PowerShell). See "Start the AI Gateway" below.
- Python SDK: `uv add litellm` (or `pip install litellm`), then `litellm.completion(model="<provider>/<model>", messages=[...])`. Responses always use the OpenAI Chat Completions shape.
- Enterprise: a license key (`LITELLM_LICENSE`) on the same gateway image adds SSO, audit logs, and admin roles. Contact: https://www.litellm.ai/enterprise#talk-to-sales
- Every docs page is available as markdown by appending `.md` to its URL, for example https://docs.litellm.ai/docs/proxy/docker_quick_start.md. The full docs are in https://docs.litellm.ai/llms-full.txt.

## Start the AI Gateway

One command installs and starts the gateway, a Postgres database, and the Admin UI with Docker Compose. It needs Docker, Podman, or Rancher Desktop with Compose v2.

macOS and Linux:

```bash
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/quickstart.sh | sh
```

Windows (PowerShell):

```powershell
irm https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/quickstart.ps1 | iex
```

- Result: the gateway and Admin UI on http://localhost:4000 (the next free port if 4000 is taken), listening on 127.0.0.1 only. Files go to `~/litellm-gateway` by default.
- Credentials: the script generates `LITELLM_MASTER_KEY`, `LITELLM_SALT_KEY`, and the database password into `~/litellm-gateway/.env` (mode 600) and never prints them. Keep `.env`: a new salt key makes stored provider keys unreadable.
- Admin UI: http://localhost:4000/ui, username `admin`, password is the `LITELLM_MASTER_KEY` value in `.env`. Add a model under Models + Endpoints > Add Model, then create a virtual key under Virtual Keys.
- Agents and CI: the script asks nothing when there is no terminal, under CI or Claude Code, with `--yes` (`curl -fsSL <url> | sh -s -- --yes`), or on Windows with `$env:LITELLM_YES=1`. `LITELLM_DIR` and `LITELLM_PORT` override the folder and port.
- Manage it: `curl -fsSL <url> | sh -s -- <action>` or the `litellm-gateway` command it installs, with `status`, `start`, `stop`, `restart`, `logs`, `open`, `update`, or `uninstall`.
- Read the script first: https://github.com/BerriAI/litellm/blob/main/scripts/quickstart.sh and https://github.com/BerriAI/litellm/blob/main/scripts/quickstart.ps1

Other ways to run it:

- The same stack step by step with Docker Compose, then the Admin UI walkthrough: https://docs.litellm.ai/docs/proxy/docker_quick_start.md
- One container with no database, from a `config.yaml`: https://docs.litellm.ai/docs/proxy/docker_quick_start.md#running-without-a-database. Only the master key works on this path: no virtual keys, Admin UI model management, or spend tracking, and budgets are not enforced.
- Production on Kubernetes with Helm, or with the official Terraform modules for AWS and GCP: https://docs.litellm.ai/docs/proxy/deploy.md. AWS and GCP also have 1-click Cloud Shell deploys into your own account: https://docs.litellm.ai/docs/proxy/deploy.md#1-click-deploy-from-cloud-shell
- Hosted by LiteLLM, with nothing to operate: talk to sales (https://www.litellm.ai/enterprise#talk-to-sales) or book an onboarding call (https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version).
- Image: `docker.litellm.ai/berriai/litellm`, also on `ghcr.io/berriai/litellm`, signed with cosign (https://docs.litellm.ai/docs/proxy/docker_image_security.md).

## Call the gateway from any language

The gateway is OpenAI-compatible and Anthropic-compatible, so an app keeps its existing SDK and changes the base URL. Authenticate with a virtual key (or the master key) and use a model name added in the Admin UI.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000", api_key="sk-<your-virtual-key>")
response = client.chat.completions.create(
    model="gpt-6.1-sol",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)
```

```typescript
import OpenAI from "openai";

const client = new OpenAI({ baseURL: "http://localhost:4000", apiKey: "sk-<your-virtual-key>" });
const response = await client.chat.completions.create({
  model: "gpt-6.1-sol",
  messages: [{ role: "user", content: "Hello" }],
});
console.log(response.choices[0].message.content);
```

```bash
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-<your-virtual-key>" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-6.1-sol", "messages": [{"role": "user", "content": "Hello"}]}'
```

- Claude Code: `export ANTHROPIC_BASE_URL=http://localhost:4000` and `export ANTHROPIC_AUTH_TOKEN=sk-<your-virtual-key>` (https://docs.litellm.ai/docs/proxy/client_setup/claude_code.md).
- Codex, Cursor, OpenCode, and other clients: https://docs.litellm.ai/docs/proxy/client_setup/overview.md

## Use the Python SDK

```bash
uv add litellm
```

```python
from litellm import completion

# Provider keys come from environment variables, for example OPENAI_API_KEY or ANTHROPIC_API_KEY
response = completion(
    model="openai/gpt-6.1-sol",  # always "<provider>/<model>", for example "anthropic/claude-sonnet-5"
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)
```

- Every provider returns the OpenAI Chat Completions shape. Provider list and model names: https://docs.litellm.ai/docs/providers.md
- Streaming and async: `stream=True` and `await litellm.acompletion(...)` (https://docs.litellm.ai/docs/completion/stream.md).
- Retries, fallbacks, and load balancing across deployments inside one process: the Router (https://docs.litellm.ai/docs/routing.md).
- Fewer dependencies for SDK-only apps: `litellm-core` (https://docs.litellm.ai/docs/litellm_core.md).

## When to choose LiteLLM

- Data and keys stay with you: the gateway runs in your own machine, cluster, or cloud account, so prompts, responses, and provider keys never pass through a third party (https://docs.litellm.ai/docs/data_security.md).
- One endpoint for 100+ providers, including OpenAI, Anthropic, Azure, Bedrock, Gemini Enterprise Agent Platform (formerly Vertex AI), and self-hosted models, with fallbacks between them.
- A platform team gives each team, app, user, or coding agent its own virtual key, budget, rate limit, and model access, with spend tracked per key.
- Open source under MIT with no markup on model spend. Enterprise adds SSO, audit logs, admin roles, and support SLAs on the same image (https://docs.litellm.ai/docs/enterprise.md).

## LiteLLM at a glance

LiteLLM is the most widely used and most secure open-source AI gateway, trusted by teams at Netflix, Okta, Ramp, NASA, Zurich, Cloudera, AT&T, and Lemonade (https://www.litellm.ai/enterprise). Figures as of 2026-10-28; each links to its source for current values.

- Adoption: 62.2k GitHub stars and 12.8k forks (https://github.com/BerriAI/litellm), with 1,800+ contributors.
- Usage: 92.2 million downloads of the `litellm` Python package in the last month (https://pypistats.org/packages/litellm).
- Container image: 518M pulls of the LiteLLM Gateway image on GitHub Container Registry and Docker Hub (https://github.com/BerriAI/litellm/pkgs/container/litellm).
- Security: SOC 2 Type II audited (https://trust.litellm.ai/). Every Docker image on GHCR is signed with cosign and can be verified before it runs (https://docs.litellm.ai/docs/proxy/docker_image_security.md). Enterprise support includes a 72-hour security patch SLA (https://docs.litellm.ai/docs/enterprise.md).
- Deployment: self-hosted in your own cloud, so prompts, responses, and provider keys stay in your infrastructure (https://docs.litellm.ai/docs/data_security.md).
- License: open source under MIT; Enterprise features need a license key (https://github.com/BerriAI/litellm/blob/main/LICENSE).

## Key pages by task

### Run and deploy the gateway

- [AI Gateway quickstart](https://docs.litellm.ai/docs/proxy/docker_quick_start.md): Start the gateway with one command, log in to the Admin UI, add a model, create a virtual key, send a request
- [Production Deployment](https://docs.litellm.ai/docs/proxy/deploy.md): Production on Kubernetes with Helm, or Terraform on AWS and GCP, including 1-click Cloud Shell deploys
- [Production Best Practices](https://docs.litellm.ai/docs/proxy/prod.md): Production checklist: workers, Redis, database, and migrations
- [config.yaml](https://docs.litellm.ai/docs/proxy/configs.md): Define models, settings, and routing in config.yaml
- [All config settings](https://docs.litellm.ai/docs/proxy/config_settings.md): Every config.yaml setting and environment variable

### Keys, budgets, and spend

- [Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys.md): Issue virtual keys with model access, expiry, and metadata
- [Budgets, Rate Limits](https://docs.litellm.ai/docs/proxy/users.md): Budgets and rate limits per key, user, team, and customer
- [Spend Tracking](https://docs.litellm.ai/docs/proxy/cost_tracking.md): Spend per key, user, team, and model, and how to query it
- [Admin UI](https://docs.litellm.ai/docs/proxy/ui.md): The Admin UI: models, keys, teams, and usage in the browser
- [SSO for Admin UI](https://docs.litellm.ai/docs/proxy/admin_ui_sso.md): Single sign-on for the Admin UI

### Routing, fallbacks, and caching

- [Router: load balancing and fallbacks](https://docs.litellm.ai/docs/routing.md): Load balancing, retries, cooldowns, and fallbacks across deployments (SDK Router and gateway)
- [Fallbacks (Provider Failover)](https://docs.litellm.ai/docs/proxy/reliability.md): Fail over to another model or provider when one fails
- [Proxy - Load Balancing](https://docs.litellm.ai/docs/proxy/load_balancing.md): Spread traffic across several deployments of one model on the gateway
- [Auto Router [Add-on]](https://docs.litellm.ai/docs/auto_router/index.md): Auto Router: send each request to the cheapest model that answers it well
- [Caching](https://docs.litellm.ai/docs/proxy/caching.md): Cache responses to cut spend and latency

### Coding agents and clients

- [Connect clients to the gateway](https://docs.litellm.ai/docs/proxy/client_setup/overview.md): Point Claude Code, Codex, Cursor, OpenCode, or any SDK at the gateway
- [Claude Code (CLI)](https://docs.litellm.ai/docs/proxy/client_setup/claude_code.md): Claude Code through the gateway
- [Claude Code with a Claude subscription](https://docs.litellm.ai/docs/tutorials/claude_code_max_subscription.md): Claude Code with a Claude Pro, Max, Team, or Enterprise subscription through the gateway
- [Use Claude Code with Non-Anthropic Models](https://docs.litellm.ai/docs/tutorials/claude_non_anthropic_models.md): Claude Code with models from other providers
- [Codex (CLI)](https://docs.litellm.ai/docs/proxy/client_setup/codex_cli.md): Codex CLI through the gateway

### MCP and agents

- [MCP Overview](https://docs.litellm.ai/docs/mcp.md): MCP gateway: one endpoint for MCP servers, with per-key access control
- [Agent Gateway (A2A Protocol) - Overview](https://docs.litellm.ai/docs/a2a.md): Agent gateway for A2A agents
- [Agent resources](https://docs.litellm.ai/docs/agent_resources.md): Copy-paste prompts, MCP, and skills for coding agents that set up LiteLLM

### Observability and safety

- [Logging](https://docs.litellm.ai/docs/proxy/logging.md): Send request logs to Langfuse, Datadog, S3, OpenTelemetry, and more
- [Prometheus metrics](https://docs.litellm.ai/docs/proxy/prometheus.md): Prometheus metrics for the gateway
- [Guardrails - Quick Start](https://docs.litellm.ai/docs/proxy/guardrails/quick_start.md): Guardrails: PII masking, prompt injection checks, and content filters
- [Data Privacy and Security](https://docs.litellm.ai/docs/data_security.md): What data LiteLLM stores and where

### Python SDK

- [Python SDK quickstart](https://docs.litellm.ai/docs/index.md): Install the SDK and make a first call to any provider
- [LiteLLM Python SDK](https://docs.litellm.ai/docs/python_sdk.md): What the SDK covers: completion, embeddings, images, audio, and more
- [Streaming + Async](https://docs.litellm.ai/docs/completion/stream.md): Streaming and async calls
- [Prompt Caching](https://docs.litellm.ai/docs/completion/prompt_caching.md): Prompt caching across providers
- [OpenAI-Compatible Endpoints](https://docs.litellm.ai/docs/providers/openai_compatible.md): Call any OpenAI-compatible endpoint

### Enterprise

- [Enterprise](https://docs.litellm.ai/docs/enterprise.md): What Enterprise adds: SSO, audit logs, admin roles, multi-region, support SLAs

## LiteLLM AI Gateway

- [LiteLLM AI Gateway](https://docs.litellm.ai/docs/simple_proxy.md): The LiteLLM AI Gateway is a self-hosted server that gives your applications one OpenAI-compatible endpoint for 100+...

## AI Gateway

- [CLI - Quick Start](https://docs.litellm.ai/docs/proxy/quick_start.md): Setup LiteLLM Proxy quickly via CLI.
- [Gateway Quickstart](https://docs.litellm.ai/docs/learn/gateway_quickstart.md): Start LiteLLM Gateway, add models and keys, then connect applications and SDKs to one shared endpoint.
- [Benchmarks](https://docs.litellm.ai/docs/benchmarks.md): Performance test results for LiteLLM Gateway.
- [Upgrade Guide](https://docs.litellm.ai/docs/proxy/upgrade_guide.md): Upgrade LiteLLM with Helm or Terraform.
- [Health Checks](https://docs.litellm.ai/docs/proxy/health.md): Two things consume the proxy's health state.
- [What Needs Redis](https://docs.litellm.ai/docs/proxy/redis_requirements.md): Features that do not work, or do not work correctly, when the LiteLLM proxy runs without Redis.
- [Database Sizing](https://docs.litellm.ai/docs/proxy/db_sizing.md): How to size Postgres for LiteLLM Proxy, with instance recommendations for AWS, Azure, and GCP.
- [Redis Sizing](https://docs.litellm.ai/docs/proxy/redis_sizing.md): How to size Redis for LiteLLM Proxy, with instance recommendations for AWS, Azure, and GCP.
- [Server Tuning](https://docs.litellm.ai/docs/proxy/server_tuning.md): Optional deep tuning for the LiteLLM Proxy container; alternative ASGI servers, worker recycling, hitless restarts,...
- [Scale for 100T+ tokens per month](https://docs.litellm.ai/docs/proxy/high_throughput.md): Benchmark throughput: 13.5 billion tokens per minute (TPM).
- [Database Read Replica](https://docs.litellm.ai/docs/proxy/db_read_replica.md): LiteLLM Proxy can route read-only queries to a separate database endpoint while
- [Multi-Region Deployment](https://docs.litellm.ai/docs/proxy/multi_region.md): Run LiteLLM proxy instances in multiple regions of the same cloud provider, all connected to one shared PostgreSQL...
- [Global Control Plane](https://docs.litellm.ai/docs/proxy/global_control_plane.md): Deploy a single LiteLLM UI that manages multiple independent LiteLLM proxy instances, each with its own database,...
- [Migrate from Kubernetes Manifests to the Microservices Chart](https://docs.litellm.ai/docs/proxy/manifests_to_microservices.md): Move a LiteLLM proxy deployed with raw Kubernetes manifests onto the componentized litellm Helm chart so gateway,...
- [Secret Managers Overview](https://docs.litellm.ai/docs/secret_managers/overview.md): LiteLLM supports reading secrets (eg.
- [AWS Secret Manager](https://docs.litellm.ai/docs/secret_managers/aws_secret_manager.md): Store your proxy keys in AWS Secret Manager.
- [AWS Key Management V1](https://docs.litellm.ai/docs/secret_managers/aws_kms.md): BETA] AWS Key Management v2 is on the enterprise tier.
- [Azure Key Vault](https://docs.litellm.ai/docs/secret_managers/azure_key_vault.md): Usage with LiteLLM Proxy Server
- [CyberArk Conjur](https://docs.litellm.ai/docs/secret_managers/cyberark.md): | Feature | Support | Description |
- [Google Secret Manager](https://docs.litellm.ai/docs/secret_managers/google_secret_manager.md): Support for Google Secret Manager
- [Google Key Management Service](https://docs.litellm.ai/docs/secret_managers/google_kms.md): Use encrypted keys from Google KMS on the proxy
- [Hashicorp Vault](https://docs.litellm.ai/docs/secret_managers/hashicorp_vault.md): | Feature | Support | Description |
- [Custom Secret Manager](https://docs.litellm.ai/docs/secret_managers/custom_secret_manager.md): Integrate your custom secret management system with LiteLLM.
- [Security Best Practices](https://docs.litellm.ai/docs/proxy/security_best_practices.md): Security is a top priority at LiteLLM.
- [Outbound Requests and Offline Mode](https://docs.litellm.ai/docs/proxy/outbound_requests.md): LiteLLM sends requests to the providers, callbacks, guardrails, secret managers and SSO servers you configure.
- [OWASP LLM Top 10 (2026) mapping](https://docs.litellm.ai/docs/proxy/security_owasp_llm_top10.md): This page maps every item of the OWASP Top 10 for LLM Applications 2026 to the LiteLLM gateway controls that address...
- [[Beta] Rust AI Gateway](https://docs.litellm.ai/docs/proxy/rust_gateway.md): Run LiteLLM request translation on the Rust core.
- [File Management](https://docs.litellm.ai/docs/proxy/config_management.md): include external YAML files in a config.yaml
- [CLI Arguments](https://docs.litellm.ai/docs/proxy/cli.md): This page documents all command-line interface (CLI) arguments available for the LiteLLM proxy server.
- [SSL, HTTP Proxy Security Settings](https://docs.litellm.ai/docs/guides/security_settings.md): If you're in an environment using an older TTS bundle, with an older encryption, follow this guide.
- [Model Management](https://docs.litellm.ai/docs/proxy/model_management.md): When STOREMODELIN_DB is on, your models live in the database rather than in a static config.yaml.
- [How Model Access Works](https://docs.litellm.ai/docs/proxy/model_access_guide.md)
- [Restrict Model Access](https://docs.litellm.ai/docs/proxy/model_access.md): Restrict models by Virtual Key
- [Model Access Groups](https://docs.litellm.ai/docs/proxy/model_access_groups.md)
- [Access Groups](https://docs.litellm.ai/docs/proxy/access_groups.md): Access Groups simplify how you define and manage resource access across your organization.
- [Allow Teams to Add Models](https://docs.litellm.ai/docs/proxy/team_model_add.md): Allow team to add a their own models/key for that project - so any OpenAI call they make uses their OpenAI key.
- [Per-Team/Project Credential Routing](https://docs.litellm.ai/docs/proxy/credential_routing.md): Route the same model to different LLM provider endpoints (e.g.
- [Claude Desktop & Cowork SSO](https://docs.litellm.ai/docs/tutorials/claude_desktop_cowork.md): Configure Claude Desktop and Cowork to use LiteLLM through third-party inference with SSO.
- [Claude Desktop & Cowork with virtual keys](https://docs.litellm.ai/docs/proxy/client_setup/claude_desktop.md): Claude Desktop and Cowork use the same gateway settings.
- [Claude Code SSO](https://docs.litellm.ai/docs/tutorials/claude_code_gateway.md): Set up Claude Code with Claude apps gateway sign-in, Lite CLI SSO, or a Claude subscription through LiteLLM.
- [Codex Desktop with SSO](https://docs.litellm.ai/docs/proxy/client_setup/codex_chatgpt_desktop_sso.md): Codex in ChatGPT Desktop reads the same ~/.codex/config.toml as the Codex CLI and runs the same authentication helper.
- [Codex (ChatGPT Desktop)](https://docs.litellm.ai/docs/proxy/client_setup/codex_chatgpt_desktop.md): Select Codex in the ChatGPT desktop app.
- [Codex CLI with SSO](https://docs.litellm.ai/docs/proxy/client_setup/codex_cli_sso.md): Use your organization's LiteLLM SSO login to connect Codex CLI to the gateway.
- [Use your ChatGPT subscription with Codex](https://docs.litellm.ai/docs/tutorials/openai_codex_byok.md): Connect Codex to LiteLLM with your ChatGPT subscription.
- [Hermes CLI with SSO](https://docs.litellm.ai/docs/proxy/client_setup/hermes_sso.md): Hermes can call lite auth print-token through its keycmd provider setting.
- [Hermes CLI with virtual keys](https://docs.litellm.ai/docs/proxy/client_setup/hermes.md): Hermes Agent can use LiteLLM as a named custom provider.
- [Cursor Integration](https://docs.litellm.ai/docs/tutorials/cursor_integration.md): Route Cursor IDE requests through LiteLLM for unified logging, budget controls, and access to any model.
- [Gemini CLI](https://docs.litellm.ai/docs/tutorials/litellm_gemini_cli.md): This tutorial shows you how to integrate the Gemini CLI with LiteLLM Proxy, allowing you to route requests through...
- [GitHub Copilot / VS Code](https://docs.litellm.ai/docs/tutorials/github_copilot_integration.md): This tutorial shows two ways to use LiteLLM Proxy from VS Code's chat.
- [OpenClaw + LiteLLM Integration](https://docs.litellm.ai/docs/tutorials/openclaw_integration.md): OpenClaw is a self-hosted AI assistant that connects chat apps (WhatsApp, Telegram, Discord, and more) to LLM providers.
- [OpenCode Quickstart](https://docs.litellm.ai/docs/tutorials/opencode_integration.md): This tutorial shows how to connect OpenCode to your existing LiteLLM instance and switch between models.
- [Qwen Code CLI](https://docs.litellm.ai/docs/tutorials/litellm_qwen_code_cli.md): This tutorial shows you how to integrate the Qwen Code CLI with LiteLLM Proxy, allowing you to route requests through...
- [Langchain, OpenAI SDK, LlamaIndex, Instructor, Curl examples](https://docs.litellm.ai/docs/proxy/user_keys.md): LiteLLM Proxy is OpenAI-Compatible, and supports:
- [Request Headers](https://docs.litellm.ai/docs/proxy/request_headers.md): Special headers that are supported by LiteLLM.
- [Response Headers](https://docs.litellm.ai/docs/proxy/response_headers.md): When you make a request to the proxy, the proxy will return the following headers:
- [Forward Client Headers to LLM API](https://docs.litellm.ai/docs/proxy/forward_client_headers.md): Control which model groups can forward client headers to the underlying LLM provider APIs.
- [Clientside LLM Credentials](https://docs.litellm.ai/docs/proxy/clientside_auth.md): Pass User LLM API Keys, Fallbacks
- [Model Discovery](https://docs.litellm.ai/docs/proxy/model_discovery.md): Use this to give users an accurate list of models available behind provider endpoint, when calling /v1/models for...
- [CLI Authentication](https://docs.litellm.ai/docs/proxy/cli_sso.md): Use the litellm cli to authenticate to the LiteLLM Gateway.
- [Claude Code with Okta SSO (JWT Auth)](https://docs.litellm.ai/docs/tutorials/claude_code_okta_sso.md): Route Claude Code through LiteLLM using each developer's Okta identity.
- [Claude Code with Bring Your Own Key (BYOK)](https://docs.litellm.ai/docs/tutorials/claude_code_byok.md): Use Claude Code with your own Anthropic credentials through the LiteLLM proxy.
- [Role-based Access Controls (RBAC)](https://docs.litellm.ai/docs/proxy/access_control.md): Role-based access control (RBAC) is based on Organizations, Teams and Internal User Roles
- [SAML 2.0 SSO](https://docs.litellm.ai/docs/proxy/saml_sso.md): SSO is free for up to 5 users.
- [Event Hooks for SSO Login](https://docs.litellm.ai/docs/proxy/custom_sso.md): SSO is free for up to 5 users.
- [SCIM with LiteLLM](https://docs.litellm.ai/docs/tutorials/scim_litellm.md): Enterprise: SCIM support requires a premium license.
- [OIDC - JWT-based Auth](https://docs.litellm.ai/docs/proxy/token_auth.md): Use JWT's to auth admins / users / projects into the proxy.
- [JWT → Virtual Key Mapping](https://docs.litellm.ai/docs/proxy/jwt_key_mapping.md): Map JWT tokens to LiteLLM virtual keys, so every JWT client gets the same granular controls as a virtual key: model...
- [Oauth 2.0 Authentication](https://docs.litellm.ai/docs/proxy/oauth2.md): Use this if you want to use an Oauth2.0 token to make /chat, /embeddings requests to the LiteLLM Proxy
- [Control Model Access with OIDC (Azure AD/Keycloak/etc.)](https://docs.litellm.ai/docs/proxy/jwt_auth_arch.md)
- [[BETA] OpenID Connect (OIDC)](https://docs.litellm.ai/docs/oidc.md): LiteLLM supports using OpenID Connect (OIDC) for authentication to upstream services .
- [ID-JAG](https://docs.litellm.ai/docs/proxy/id_jag/overview.md): ID-JAG (Identity Assertion JWT Authorization Grant) lets an application access another service on a signed-in user's...
- [LiteLLM as resource app](https://docs.litellm.ai/docs/proxy/id_jag/resource_app.md): Let an application exchange an ID-JAG for a LiteLLM access token, then use it to run inference or call MCP tools on...
- [LiteLLM as requesting app](https://docs.litellm.ai/docs/proxy/id_jag/requesting_app.md): Let LiteLLM call MCP tools on behalf of a signed-in user.
- [Provisioning identities and issuing keys](https://docs.litellm.ai/docs/proxy/identity_provisioning.md): A question that comes up with SSO deployments is whether a user provisioned through your identity provider (via SCIM)...
- [Service Accounts](https://docs.litellm.ai/docs/proxy/service_accounts.md): Use this if you want to create Virtual Keys that are not owned by a specific user but instead created for production...
- [Custom Auth](https://docs.litellm.ai/docs/proxy/custom_auth.md): You can now override the default api key auth.
- [IP Address Filtering](https://docs.litellm.ai/docs/proxy/ip_address.md): Restrict which IP's can call the proxy endpoints.
- [Control Public & Private Routes](https://docs.litellm.ai/docs/proxy/public_routes.md): Control which routes require authentication and which routes are publicly accessible.
- [Rotating the Master Key](https://docs.litellm.ai/docs/proxy/master_key_rotations.md): The master key is the proxy's admin credential; it authenticates admin API calls and logs you into the Admin UI.
- [Audit Logs](https://docs.litellm.ai/docs/proxy/audit_logs.md)
- [Model Cost Map](https://docs.litellm.ai/docs/proxy/model_cost_map.md): LiteLLM prices every request from its model cost map, modelpricesandcontextwindow.json.
- [Request Tags for Spend Tracking](https://docs.litellm.ai/docs/proxy/request_tags.md): Add tags to model deployments to track spend by environment, AWS account, or any custom label.
- [Billing](https://docs.litellm.ai/docs/proxy/billing.md): Bill internal teams, external customers for their usage
- [Custom LLM Pricing](https://docs.litellm.ai/docs/proxy/custom_pricing.md)
- [Spend Capture Rate](https://docs.litellm.ai/docs/proxy/spend_capture_rate.md): Compare the spend LiteLLM tracked for OpenAI against what OpenAI billed for the same days, on a schedule with Slack...
- [Billable Request Metering](https://docs.litellm.ai/docs/proxy/billing_metrics.md): LiteLLM Enterprise pricing is usage-based.
- [Custom Model Cost Map](https://docs.litellm.ai/docs/proxy/custom_model_cost_map.md): LiteLLM prices every request from its model cost map, a JSON file mapping each model to per-token rates, context...
- [Azure PTU Flat Cost Attribution](https://docs.litellm.ai/docs/proxy/ptu_flat_cost.md): Azure provisioned throughput is billed by the hour for reserved capacity, not per token.
- [Pricing Calculator (Cost Estimation)](https://docs.litellm.ai/docs/proxy/pricing_calculator.md): Estimate LLM costs based on expected token usage and request volume.
- [Fee/Price Margin on LLM Costs](https://docs.litellm.ai/docs/proxy/provider_margins.md): Apply percentage-based or fixed-amount margins to specific providers or globally.
- [Provider Discounts](https://docs.litellm.ai/docs/proxy/provider_discounts.md): Apply percentage-based discounts to specific providers.
- [Off-Peak Pricing](https://docs.litellm.ai/docs/proxy/off_peak_pricing.md): Some providers charge less during fixed clock windows; DeepSeek, for example, halves its rates outside its peak hours.
- [Auto Sync New Models (Day-0 Launches)](https://docs.litellm.ai/docs/proxy/sync_models_github.md): Automatically keep your model pricing and context window data up to date without restarting your service.
- [Vertex AI PayGo and Priority](https://docs.litellm.ai/docs/tutorials/vertex_ai_pay_go.md)
- [Maximum Retention Period for Spend Logs](https://docs.litellm.ai/docs/proxy/spend_logs_deletion.md): This walks through how to set the maximum retention period for spend logs.
- [Setting Team Budgets](https://docs.litellm.ai/docs/proxy/team_budgets.md)
- [[Beta] Project Management](https://docs.litellm.ai/docs/proxy/project_management.md): Projects in LiteLLM sit between teams and keys in the organizational hierarchy, enabling fine-grained access control...
- [Customers / End-Users](https://docs.litellm.ai/docs/proxy/customers.md): Track spend, set budgets and permissions for your customers.
- [Team Soft Budget Alerts](https://docs.litellm.ai/docs/proxy/ui_team_soft_budget_alerts.md): Email budget alerts require an enterprise license.
- [Setting Tag Budgets](https://docs.litellm.ai/docs/proxy/tag_budgets.md): Track spend and set budgets for your API requests using tags.
- [Model Access Group Budgets](https://docs.litellm.ai/docs/proxy/model_access_group_budgets.md): Give a model access group one shared budget.
- [Dynamic TPM/RPM Allocation](https://docs.litellm.ai/docs/proxy/dynamic_rate_limit.md): Prevent projects from gobbling too much tpm/rpm.
- [[Beta] Separate ITPM / OTPM Rate Limits](https://docs.litellm.ai/docs/proxy/io_token_rate_limits.md): Enforce input tokens per minute (ITPM) and output tokens per minute (OTPM) separately on router deployments.
- [Budget / Rate Limit Tiers](https://docs.litellm.ai/docs/proxy/rate_limit_tiers.md): Define tiers with rate limits.
- [Temporary Budget Increase](https://docs.litellm.ai/docs/proxy/temporary_budget_increase.md): Set a temporary budget increase for a LiteLLM Virtual Key or a team member.
- [Budget Reset Times and Timezones](https://docs.litellm.ai/docs/proxy/budget_reset_and_tz.md): LiteLLM supports predictable budget reset times that align with natural calendar boundaries.
- [Budget Fallbacks](https://docs.litellm.ai/docs/proxy/budget_fallbacks.md): Available on v1.92.x and later.
- [Redis and Valkey Cache](https://docs.litellm.ai/docs/proxy/caching_redis.md): Credentials, namespaces, ACL users, cluster and sentinel topologies, and TLS for the LiteLLM proxy's Redis cache.
- [AWS ElastiCache IAM Authentication](https://docs.litellm.ai/docs/proxy/elasticache_iam.md): Connect LiteLLM's Redis cache to AWS ElastiCache (Redis OSS or Valkey) using IAM authentication, so
- [GCP Memorystore IAM Authentication](https://docs.litellm.ai/docs/proxy/gcp_memorystore_iam.md): Authenticate the LiteLLM proxy to GCP Memorystore for Redis with a service account instead of a password.
- [Azure Redis Entra ID Authentication](https://docs.litellm.ai/docs/proxy/azure_redis_ad.md): Authenticate the LiteLLM proxy to Azure Cache for Redis or Azure Managed Redis with Microsoft Entra ID instead of an...
- [Semantic Caching](https://docs.litellm.ai/docs/proxy/caching_semantic.md): Serve cached LLM responses for prompts that are similar but not identical, and the traffic shapes where that goes wrong.
- [S3 and GCS Cache](https://docs.litellm.ai/docs/proxy/caching_object_storage.md): Back the LiteLLM proxy's response cache with an S3 or GCS bucket.
- [Cache Controls](https://docs.litellm.ai/docs/proxy/caching_controls.md): Control what the LiteLLM proxy caches and for how long, per request, per virtual key, and proxy-wide.
- [Cache Settings](https://docs.litellm.ai/docs/proxy/caching_settings.md): Every cache_params setting the LiteLLM proxy accepts in config.yaml.
- [Prompt Compression (Headroom)](https://docs.litellm.ai/docs/proxy/headroom.md): Headroom is the context optimization layer for LLM applications.
- [A/B Testing - Traffic Mirroring](https://docs.litellm.ai/docs/traffic_mirroring.md): Traffic mirroring allows you to "mimic" production traffic to a secondary (silent) model for evaluation purposes.
- [Team Bring-Your-Own Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/team_based_guardrails.md): Team-based guardrails let developers register a guardrail for their team via the API; an admin then reviews and...
- [Guardrail Load Balancing](https://docs.litellm.ai/docs/proxy/guardrails/guardrail_load_balancing.md): Load balance guardrail requests across multiple guardrail deployments.
- [Guardrail Testing Playground](https://docs.litellm.ai/docs/proxy/guardrails/test_playground.md): Test and compare multiple guardrails in real-time with an interactive playground interface.
- [LiteLLM Content Filter (Built-in Guardrails)](https://docs.litellm.ai/docs/proxy/guardrails/litellm_content_filter.md): Built-in guardrail for detecting and filtering sensitive information using regex patterns and keyword matching.
- [Realtime API Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/realtime_guardrails.md): Guard voice conversations in the Realtime API by intercepting speech transcriptions before the LLM responds.
- [Batch API Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/batch_guardrails.md): Guard the records inside a batch input file, so a batch job cannot move content past a guardrail that would have...
- [[BETA] Generic Guardrail API - Integrate Without a PR](https://docs.litellm.ai/docs/adding_provider/generic_guardrail_api.md)
- [Adding a New Guardrail Integration](https://docs.litellm.ai/docs/adding_provider/simple_guardrail_tutorial.md): You're going to create a class that checks text before it goes to the LLM or after it comes back.
- [Adding Guardrail Support to Endpoints](https://docs.litellm.ai/docs/adding_provider/adding_guardrail_support.md): This guide explains how to add guardrail translation support to new LiteLLM endpoints (e.g., Chat Completions,...
- [[Beta] Guardrail Policies](https://docs.litellm.ai/docs/proxy/guardrails/guardrail_policies.md): Use policies to group guardrails and control which ones run for specific teams, keys, or models.
- [Policy Flow Builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder.md): The Policy Flow Builder lets you design guardrail pipelines with conditional execution.
- [Policy Templates](https://docs.litellm.ai/docs/proxy/guardrails/policy_templates.md): Policy templates provide pre-configured guardrail policies that you can use as a starting point for your organization.
- [Tag-Based Policy Attachments](https://docs.litellm.ai/docs/proxy/guardrails/policy_tags.md): Apply guardrail policies automatically to any key or team that has a specific tag.
- [Dynamic Callback Management](https://docs.litellm.ai/docs/proxy/dynamic_logging.md): LiteLLM's dynamic callback management enables teams to control logging behavior on a per-request basis without...
- [StandardLoggingPayload Specification](https://docs.litellm.ai/docs/proxy/logging_spec.md): Terminal success and failure callback events include kwargs"standardloggingobject"] when LiteLLM finishes building the...
- [Team/Key Based Logging](https://docs.litellm.ai/docs/proxy/team_logging.md)
- [Email Notifications](https://docs.litellm.ai/docs/proxy/email.md)
- [Customize UI Logo](https://docs.litellm.ai/docs/proxy/ui/ui_edit_logo.md): Personalize your LiteLLM dashboard by replacing the default logo with your own company branding.
- [UI - Custom Root Path](https://docs.litellm.ai/docs/proxy/custom_root_ui.md): Use this when you want to serve LiteLLM on a custom base url path like https4000/api/v1
- [Plugins](https://docs.litellm.ai/docs/proxy/plugins.md): Plugins let external services appear as selectable modes in the LiteLLM UI sidebar alongside the AI Gateway.
- [AI Hub](https://docs.litellm.ai/docs/proxy/ai_hub.md): Share models and agents with your organization.
- [Model Compare Playground UI](https://docs.litellm.ai/docs/proxy/model_compare_ui.md): Compare multiple LLM models side-by-side in an interactive playground interface.
- [Internal User Self-Serve](https://docs.litellm.ai/docs/proxy/self_serve.md): Allow users to create their own keys on Proxy UI.
- [[BETA] Public Teams](https://docs.litellm.ai/docs/proxy/public_teams.md): Expose available teams to your users to join on signup.
- [[Beta] Project Management UI](https://docs.litellm.ai/docs/proxy/ui_project_management.md): Manage projects directly from the LiteLLM Admin UI.
- [Search tools (Admin UI)](https://docs.litellm.ai/docs/proxy/ui_search_tools.md): Control which teams and virtual keys may call each configured web search integration (/v1/search) and see team/key...
- [Bulk Edit Users](https://docs.litellm.ai/docs/proxy/ui/bulk_edit_users.md): Assign existing users to a default team and default model access.
- [Control Page Visibility for Internal Users](https://docs.litellm.ai/docs/proxy/ui/page_visibility.md): Configure which navigation tabs and pages are visible to internal users (non-admin developers) in the LiteLLM UI.
- [Customer Usage](https://docs.litellm.ai/docs/proxy/customer_usage.md): Track and visualize end-user spend directly in the dashboard.
- [Endpoint Activity](https://docs.litellm.ai/docs/proxy/endpoint_activity.md): Track and visualize API endpoint usage directly in the dashboard.
- [Getting Started with UI Logs](https://docs.litellm.ai/docs/proxy/ui_logs.md): View Spend, Token Usage, Key, Team Name for Each Request to LiteLLM
- [UI Spend Log Settings](https://docs.litellm.ai/docs/proxy/ui_spend_log_settings.md): Configure spend log behavior directly from the Admin UI, with no config file edits or proxy restart required.
- [Session Logs](https://docs.litellm.ai/docs/proxy/ui_logs_sessions.md): Group requests into sessions.
- [Deleted Keys & Teams Audit Logs](https://docs.litellm.ai/docs/proxy/deleted_keys_teams.md): View deleted API keys and teams along with their spend and budget information at the time of deletion for auditing and...
- [Chat UI](https://docs.litellm.ai/docs/proxy/ui/chat_ui.md): Chat with the models on your LiteLLM gateway, use MCP tools, and see your own keys, logs, and usage on one page in the...
- [LiteAdmin MCP](https://docs.litellm.ai/docs/proxy/liteadmin_mcp.md): Connect Claude Desktop, Claude Code, or Codex to your LiteLLM gateway with LiteAdmin MCP.
- [Deploy LiteAdmin MCP on Enterprise](https://docs.litellm.ai/docs/proxy/liteadmin_mcp_enterprise.md): Enable LiteAdmin MCP in unified or componentized LiteLLM deployments and connect Claude Code or Codex.
- [Set up the LiteAdmin Slack app](https://docs.litellm.ai/docs/proxy/liteadmin_slack.md): Install LiteAdmin in Slack, deploy the Enterprise worker or a standalone agent, and connect your own LiteLLM admin...
- [Modify / Reject Incoming Requests](https://docs.litellm.ai/docs/proxy/call_hooks.md): - Modify data before making llm api calls on proxy
- [Agentic Loop Hook](https://docs.litellm.ai/docs/proxy/agentic_loop_hook.md): Build a CustomLogger callback that intercepts a model response, fulfills tool calls server-side, and reruns the model,...
- [Post-Call Rules](https://docs.litellm.ai/docs/proxy/rules.md): Use this to fail a request based on the output of an llm api call.
- [Worker Startup Hooks](https://docs.litellm.ai/docs/proxy/worker_startup_hooks.md): Use LITELLMWORKERSTARTUPHOOKS to run custom initialization functions in **each worker process** during proxy startup.
- [Call any LiteLLM model in your custom format](https://docs.litellm.ai/docs/extras/creating_adapters.md): Use this to call any LiteLLM supported .completion() model, in your custom format.
- [Memory Management](https://docs.litellm.ai/docs/proxy/memory.md): Store user preferences and feedback so your LLM remembers them across sessions.
- [Life of a Request](https://docs.litellm.ai/docs/proxy/architecture.md)
- [Multi-Tenant Architecture with LiteLLM](https://docs.litellm.ai/docs/proxy/multi_tenant_architecture.md)
- [User Management Hierarchy](https://docs.litellm.ai/docs/proxy/user_management_heirarchy.md): LiteLLM supports a hierarchy of users, teams, organizations, and budgets.
- [How Key-Based Auth Works](https://docs.litellm.ai/docs/proxy/key_auth_arch.md): Explanation of how the proxy resolves what a virtual key can call.
- [What is stored in the DB](https://docs.litellm.ai/docs/proxy/db_info.md): The LiteLLM Proxy uses a PostgreSQL database to store various information.
- [Debugging](https://docs.litellm.ai/docs/proxy/debugging.md): 2 levels of debugging supported.
- [Error Reference](https://docs.litellm.ai/docs/proxy/error_reference.md): Every failed request through the LiteLLM AI Gateway returns an OpenAI-compatible JSON error body and an HTTP status...
- [Diagnosing Errors - Provider vs Gateway](https://docs.litellm.ai/docs/proxy/error_diagnosis.md): Having trouble diagnosing if an error is from the LLM Provider (OpenAI, Anthropic, etc.) or from the LiteLLM AI...

## MCP Gateway

- [Using your MCP](https://docs.litellm.ai/docs/mcp_usage.md): This document covers how to use LiteLLM as an MCP Gateway.
- [MCP Deployment Guide](https://docs.litellm.ai/docs/mcp_deployment.md): How to deploy LiteLLM as a central gateway for LLMs, MCP servers, and agents.
- [MCP Server Usage](https://docs.litellm.ai/docs/mcp_servers/index.md): Setup guides for connecting popular third-party MCP servers through the LiteLLM MCP Gateway.
- [Slack MCP server](https://docs.litellm.ai/docs/mcp_servers/slack.md): Connect Slack's hosted MCP server through the LiteLLM MCP Gateway for workspace search, channel reads, and messaging.
- [Atlassian MCP server](https://docs.litellm.ai/docs/mcp_servers/atlassian.md): Connect Atlassian's hosted remote MCP server through the LiteLLM MCP Gateway for Jira issues, Confluence pages, and...
- [Linear MCP server](https://docs.litellm.ai/docs/mcp_servers/linear.md): Connect Linear's hosted remote MCP server through the LiteLLM MCP Gateway for issues, projects, cycles, and comments.
- [Microsoft 365 MCP server](https://docs.litellm.ai/docs/mcp_servers/microsoft_365.md): Reach Outlook mail and calendar, OneDrive and SharePoint files, and Teams through Microsoft Graph, with every tool...
- [MCP from OpenAPI Specs](https://docs.litellm.ai/docs/mcp_openapi.md): LiteLLM can convert any OpenAPI/Swagger spec into an MCP server, with no custom MCP server code required.
- [MCP Toolsets](https://docs.litellm.ai/docs/mcp_toolsets.md): For direct-client URLs and LLM tool selectors, see the MCP Configuration Reference
- [MCP Server Submissions](https://docs.litellm.ai/docs/mcp_server_submissions.md): LiteLLM supports a submission and approval workflow for MCP servers.
- [MCP Non-OAuth Authentication](https://docs.litellm.ai/docs/mcp_authentication.md): This page covers upstream MCP authentication that does not use an OAuth flow, including no authentication, fixed...
- [MCP OAuth](https://docs.litellm.ai/docs/mcp_oauth.md): LiteLLM supports several OAuth 2.0 patterns for MCP servers.
- [MCP Per-User and Per-Key Upstream Credentials](https://docs.litellm.ai/docs/mcp_per_user_auth.md): Register an MCP server once in LiteLLM, then let users and service accounts call it with their own upstream credentials.
- [MCP OAuth Passthrough](https://docs.litellm.ai/docs/mcp_oauth_passthrough.md): Some MCP servers run their own OAuth issuer and expect the client (Claude Code, Cursor, ChatGPT, etc.) to authenticate...
- [MCP OBO Auth](https://docs.litellm.ai/docs/mcp_obo_auth.md): OAuth 2.0 On-Behalf-Of (OBO) auth lets LiteLLM exchange a user's incoming bearer token for a scoped token that is...
- [MCP ID-JAG Auth (Okta)](https://docs.litellm.ai/docs/mcp_id_jag.md): See the ID-JAG overview and the requesting app setup guide for configuration, SSO options, and troubleshooting.
- [MCP - AWS SigV4 Auth](https://docs.litellm.ai/docs/mcp_aws_sigv4.md): Use AWS SigV4 authentication to connect LiteLLM to MCP servers hosted on AWS Bedrock AgentCore.
- [MCP Zero Trust Auth (JWT Signer)](https://docs.litellm.ai/docs/mcp_zero_trust.md)
- [Gateway Auth Reference](https://docs.litellm.ai/docs/auth_overview.md): LiteLLM exposes two gateway surfaces that share most authentication and authorization primitives but diverge in a few...
- [MCP Permission Management](https://docs.litellm.ai/docs/mcp_control.md): Control which MCP servers and tools can be accessed by specific keys, teams, or organizations in LiteLLM.
- [Grant MCP Server Access to Keys and Teams](https://docs.litellm.ai/docs/mcp_grant_access.md): This guide walks through granting an MCP server to a virtual key and to a team, first in the Admin UI and then through...
- [Allowlisting MCP Client Applications](https://docs.litellm.ai/docs/mcp_client_allowlist.md): Use an allowlist to restrict MCP gateway access to approved client applications, such as Claude Code, Cursor, or an...
- [Exposing MCPs on the Public Internet](https://docs.litellm.ai/docs/mcp_public_internet.md): Control which MCP servers are visible to external callers (e.g., ChatGPT, Claude Desktop) vs.
- [Tool Policies](https://docs.litellm.ai/docs/proxy/tool_policies.md): Tool Policies is a registry of every tool the proxy has seen in traffic (OpenAI and Anthropic request tools, response...
- [MCP Cost Tracking](https://docs.litellm.ai/docs/mcp_cost.md): LiteLLM provides two ways to track costs for MCP tool calls:
- [MCP Guardrails](https://docs.litellm.ai/docs/mcp_guardrail.md): LiteLLM supports applying guardrails to MCP tool calls to ensure security and compliance.
- [MCP Tool Search](https://docs.litellm.ai/docs/mcp_tool_search.md): Swap the full MCP catalog for a fixed set of virtual tools (mcptoolsearch, mcptoolcall, agentsearch, skillsearch) so a...
- [MCP REST API](https://docs.litellm.ai/docs/mcp_rest_api.md): Guide to call MCP tools directly over HTTP.
- [MCP Semantic Tool Filter](https://docs.litellm.ai/docs/mcp_semantic_filter.md): Automatically filter MCP tools by semantic relevance.
- [MCP Configuration Reference](https://docs.litellm.ai/docs/mcp_config_reference.md): This is the canonical decision reference for connecting to LiteLLM's MCP Gateway: which endpoint to use, which...
- [MCP Troubleshooting Guide](https://docs.litellm.ai/docs/mcp_troubleshoot.md): When LiteLLM acts as an MCP gateway, traffic flows Client -> LiteLLM Proxy -> MCP Server, and OAuth-enabled setups add...

## Agent Gateway

- [Invoking A2A Agents](https://docs.litellm.ai/docs/a2a_invoking_agents.md): Learn how to invoke A2A agents through LiteLLM using different methods.
- [A2A Agent Card](https://docs.litellm.ai/docs/a2a_agent_card.md): LiteLLM can proxy A2A-compatible agents, exposing them to your clients through LiteLLM with virtual keys, team...
- [Databricks Agents](https://docs.litellm.ai/docs/a2a_databricks_agent.md): Register a Databricks ResponsesAgent as a LiteLLM agent and call it through /a2a/ or as model: a2a/ on...
- [A2A Agent Authentication Headers](https://docs.litellm.ai/docs/a2a_agent_headers.md): Forward authentication credentials (Bearer tokens, API keys, etc.) from clients to backend A2A agents.
- [Agent Permission Management](https://docs.litellm.ai/docs/a2a_agent_permissions.md): Control which A2A agents can be accessed by specific keys or teams in LiteLLM.
- [A2A Agent Cost Tracking](https://docs.litellm.ai/docs/a2a_cost_tracking.md): LiteLLM supports adding custom cost tracking for A2A agents.
- [Agent Iteration Budgets](https://docs.litellm.ai/docs/a2a_iteration_budgets.md): Control runaway costs from agentic loops with per-session iteration and budget caps.
- [Agent Kill Switch](https://docs.litellm.ai/docs/a2a_kill_switch.md): Store a "stop this agent" webhook on each agent and fire it from LiteLLM when the agent misbehaves.
- [Skills Gateway](https://docs.litellm.ai/docs/skills_gateway.md): LiteLLM acts as a Skills Registry, a central place to register, manage, and discover Claude Code skills across your...

## Auto Router

- [Admin Setup](https://docs.litellm.ai/docs/auto_router/setup.md): Create and configure an Auto Router through the dashboard, an agent skill, config.yaml, the model-management API, or...
- [Customize your classifier](https://docs.litellm.ai/docs/auto_router/optimize_classifier.md): Tune Auto Router classification with heuristic v1 and v2, heuristic-first and hybrid LLM chains, self-hosted...
- [Connect self-hosted classifiers](https://docs.litellm.ai/docs/auto_router/decision_classifiers.md): Use any deployed decision model, such as Strands Decider, vLLM or Cloudflare Clef, or connect a self-hosted Laya or...
- [User Setup](https://docs.litellm.ai/docs/auto_router/user_setup.md): Connect Claude Code or Codex to your team's Auto Router with lite configure and view live session savings in Claude...
- [Recommended Configurations](https://docs.litellm.ai/docs/auto_router/recommended_configurations.md): The bundled 1M Context, Anthropic, OpenAI, Gemini, and Lite presets as config.yaml, the configs the public benchmarks...
- [Public Benchmarks](https://docs.litellm.ai/docs/auto_router/benchmarks.md): Every published measurement of the Auto Router, from Terminal-Bench 2.0 and RouterArena to a four-month production...
- [Prompt Caching](https://docs.litellm.ai/docs/auto_router/prompt_caching.md): Switching models between turns does not throw away your prompt cache.
- [Evaluate on Your Traffic](https://docs.litellm.ai/docs/auto_router/evaluate.md): Run a shadow evaluation to judge the router against your current model on live traffic before switching, then read...
- [Auto Router OTEL Telemetry](https://docs.litellm.ai/docs/auto_router/telemetry.md): Trace the Auto Router configuration, selected model, routing reason, and recovered classifier failures.
- [Auto Router Feature History](https://docs.litellm.ai/docs/auto_router/feature_history.md): Which Auto Router features shipped in which LiteLLM release, so you know what to expect when you upgrade.
- [[Beta] Auto Routing](https://docs.litellm.ai/docs/proxy/auto_routing.md): One router for complexity, semantic, and adaptive routing.
- [Semantic Auto Router (deprecated)](https://docs.litellm.ai/docs/proxy/auto_routing_semantic.md): The semantic Auto Router is superseded by Auto Routing, which folds semantic keyword matching, complexity scoring, and...
- [Auto Routing Benchmark: Cost Ladders](https://docs.litellm.ai/docs/proxy/auto_routing_benchmark.md): This benchmark evaluates how well LiteLLM routing strategies reduce cost while preserving response quality.
- [[BETA] Adaptive Router](https://docs.litellm.ai/docs/adaptive_router.md): Beta feature.

## Python SDK

- [SDK Quickstart](https://docs.litellm.ai/docs/learn/sdk_quickstart.md): Make your first LiteLLM SDK call, then jump to the right docs for the next feature you need.
- [LiteLLM Core](https://docs.litellm.ai/docs/litellm_core.md): Install the LiteLLM SDK with fewer default dependencies and add AWS and Python tokenizer packages when needed
- [Input Params](https://docs.litellm.ai/docs/completion/input.md)
- [/embeddings](https://docs.litellm.ai/docs/embedding/supported_embedding.md)
- [/responses](https://docs.litellm.ai/docs/response_api.md): LiteLLM provides an endpoint in the spec of OpenAI's /responses API
- [/completions](https://docs.litellm.ai/docs/text_completion.md)
- [Image Generations](https://docs.litellm.ai/docs/image_generation.md)
- [Prompt Compression (compress())](https://docs.litellm.ai/docs/completion/prompt_compression.md): This feature is in beta.
- [/audio/transcriptions](https://docs.litellm.ai/docs/audio_transcription.md)
- [/audio/speech](https://docs.litellm.ai/docs/text_to_speech.md)
- [Sandbox / Code Execution](https://docs.litellm.ai/docs/sandbox.md): Run model-generated code inside an isolated sandbox and get its output back.
- [Agents (litellm.agent)](https://docs.litellm.ai/docs/harness.md): Run Claude Code, Codex, OpenCode, Deep Agents and Tool Loop from Python with litellm.agent(), with model calls routed...
- [Using with LiteLLM AI Gateway](https://docs.litellm.ai/docs/harness/gateway.md): Run agents with litellm.agent() against one LiteLLM AI Gateway virtual key, pick the right harness for each job, and...
- [Supported harnesses](https://docs.litellm.ai/docs/harness/supported.md): This release supports five harnesses.
- [Claude Code](https://docs.litellm.ai/docs/harness/claude_code.md): Harness.CLAUDE_CODE runs Anthropic's Claude Code CLI with claude -p --output-format stream-json inside your sandbox...
- [Codex](https://docs.litellm.ai/docs/harness/codex.md): Harness.CODEX runs OpenAI's Codex CLI with codex exec --json inside your sandbox and turns its JSONL events into events.
- [OpenCode](https://docs.litellm.ai/docs/harness/opencode.md): Harness.OPENCODE runs the OpenCode CLI with opencode run --format json inside your sandbox and turns its JSON events...
- [Deep Agents](https://docs.litellm.ai/docs/harness/deepagents.md): Harness.DEEPAGENTS runs LangChain's Deep Agents in your Python process.
- [Tool Loop](https://docs.litellm.ai/docs/harness/tool_loop.md): Harness.TOOL_LOOP runs a minimal tool-calling loop in your Python process.
- [Quickstart](https://docs.litellm.ai/docs/harness/quickstart.md)
- [Models and routing](https://docs.litellm.ai/docs/harness/models.md): model= follows the same convention as litellm.completion.
- [Sessions](https://docs.litellm.ai/docs/harness/sessions.md): A session keeps its working directory and conversation history alive between turns.
- [Events](https://docs.litellm.ai/docs/harness/events.md): litellm.agent(..., stream=True) yields eight kinds of event.
- [Sandboxes](https://docs.litellm.ai/docs/harness/sandboxes.md): A sandbox is required by every call.
- [Tools and skills](https://docs.litellm.ai/docs/harness/tools.md)
- [Permissions](https://docs.litellm.ai/docs/harness/permissions.md): permissions= takes a mode, and each harness sets up its permission behavior to match.
- [Structured output](https://docs.litellm.ai/docs/harness/structured_output.md): Pass a Pydantic model as output=.
- [API reference](https://docs.litellm.ai/docs/harness/api_reference.md): The entry points live on the top-level litellm module, next to litellm.completion.
- [Errors](https://docs.litellm.ai/docs/harness/errors.md): Setup problems raise an exception.
- [Setting API Keys, Base, Version](https://docs.litellm.ai/docs/set_keys.md): LiteLLM allows you to specify the following:
- [SDK Proxy Authentication (OAuth2/JWT Auto-Refresh)](https://docs.litellm.ai/docs/proxy_auth.md): Automatically obtain and refresh OAuth2/JWT tokens when using the LiteLLM Python SDK with a LiteLLM Proxy that...
- [Caching - In-Memory, Redis, s3, gcs, Redis Semantic Cache, Disk](https://docs.litellm.ai/docs/caching/all_caches.md)
- [Completion Token Usage & Cost](https://docs.litellm.ai/docs/completion/token_usage.md): By default LiteLLM returns token usage in all completion requests (See here)
- [Trimming Input Messages](https://docs.litellm.ai/docs/completion/message_trimming.md): Use litellm.trim_messages() to ensure messages does not exceed a model's token limit or specified max_tokens
- [Model Alias](https://docs.litellm.ai/docs/completion/model_alias.md): The model name you show an end-user might be different from the one you pass to LiteLLM - e.g.
- [Prompt Formatting](https://docs.litellm.ai/docs/completion/prompt_formatting.md): LiteLLM automatically translates the OpenAI ChatCompletions prompt format, to other models.
- [Reliability - Retries, Fallbacks](https://docs.litellm.ai/docs/completion/reliable_completions.md): LiteLLM helps prevent failed requests in 2 ways:
- [Mock Completion() Responses - Save Testing Costs](https://docs.litellm.ai/docs/completion/mock_requests.md): For testing purposes, you can use completion() with mock_response to mock calling the completion endpoint.
- [Exception Mapping](https://docs.litellm.ai/docs/exception_mapping.md): LiteLLM maps exceptions across all providers to their OpenAI counterparts.

## LiteLLM Lens

- [LiteLLM Lens](https://docs.litellm.ai/docs/proxy/lens.md): Trace agent runs and investigate recurring failures with LiteLLM Lens.
- [Set up Lens](https://docs.litellm.ai/docs/proxy/lens/deployment.md): Use Lens in your LiteLLM dashboard or deploy it as a separate application.
- [Start standalone Lens](https://docs.litellm.ai/docs/proxy/lens/deployment/local.md): Run Lens and ClickHouse locally, sign in, and send a first trace without a LiteLLM gateway.
- [Add Lens to LiteLLM](https://docs.litellm.ai/docs/proxy/lens/deployment/litellm.md): Connect a running Lens service to your gateway and open it from the LiteLLM sidebar.
- [Kubernetes](https://docs.litellm.ai/docs/proxy/lens/deployment/kubernetes.md): Deploy independent Lens with its own Helm chart and optional LiteLLM connection.
- [Docker Compose on a server](https://docs.litellm.ai/docs/proxy/lens/deployment/server.md): Run standalone Lens and ClickHouse on one server with persistent storage and HTTPS.
- [Connect Lens to Docker Compose](https://docs.litellm.ai/docs/proxy/lens/deployment/docker-compose.md): Add a running Lens service to your existing LiteLLM Compose deployment.
- [Lens containers](https://docs.litellm.ai/docs/proxy/lens/deployment/docker.md): Run the independent Lens container with ClickHouse and optionally connect LiteLLM.
- [Storage and secrets](https://docs.litellm.ai/docs/proxy/lens/deployment/storage.md): Configure supported ClickHouse storage, Lens credentials and GitOps.
- [Configuration and troubleshooting](https://docs.litellm.ai/docs/proxy/lens/deployment/configuration.md): Configure standalone Lens credentials, networking and the optional LiteLLM connection.
- [Releases and images](https://docs.litellm.ai/docs/proxy/lens/deployment/releases.md): Understand paired Lens and LiteLLM releases, standalone deployment, and source builds.
- [Upgrade Lens](https://docs.litellm.ai/docs/proxy/lens/deployment/upgrades.md): Update Lens independently while retaining its records, credentials and recovery point.
- [Build from source](https://docs.litellm.ai/docs/proxy/lens/deployment/development.md): Develop standalone Lens with its Rust backend, ClickHouse and shared UI.
- [Send your first trace](https://docs.litellm.ai/docs/proxy/lens/first-trace.md): Send one model call to Lens, then connect your existing agent.
- [Framework examples through LiteLLM](https://docs.litellm.ai/docs/proxy/lens/framework-examples.md): Complete framework examples for a model connection through LiteLLM.
- [DeepAgents](https://docs.litellm.ai/docs/proxy/lens/integrations/deepagents.md): Run DeepAgents examples and send their agent traces to LiteLLM Lens.
- [LangGraph](https://docs.litellm.ai/docs/proxy/lens/integrations/langgraph.md): Run LangGraph examples and send their agent traces to LiteLLM Lens.
- [LangChain](https://docs.litellm.ai/docs/proxy/lens/integrations/langchain.md): Run LangChain examples and send their agent traces to LiteLLM Lens.
- [OpenAI Agents SDK](https://docs.litellm.ai/docs/proxy/lens/integrations/openai-agents.md): Run OpenAI Agents SDK examples and send their agent traces to LiteLLM Lens.
- [Claude Agent SDK](https://docs.litellm.ai/docs/proxy/lens/integrations/claude-agent-sdk.md): Run Claude Agent SDK examples and send their agent traces to LiteLLM Lens.
- [CrewAI](https://docs.litellm.ai/docs/proxy/lens/integrations/crewai.md): Run CrewAI examples and send their agent traces to LiteLLM Lens.
- [Pydantic AI](https://docs.litellm.ai/docs/proxy/lens/integrations/pydantic-ai.md): Run Pydantic AI examples and send their agent traces to LiteLLM Lens.
- [LlamaIndex](https://docs.litellm.ai/docs/proxy/lens/integrations/llamaindex.md): Run LlamaIndex examples and send their agent traces to LiteLLM Lens.
- [Google ADK](https://docs.litellm.ai/docs/proxy/lens/integrations/google-adk.md): Run Google ADK examples and send their agent traces to LiteLLM Lens.
- [Strands Agents](https://docs.litellm.ai/docs/proxy/lens/integrations/strands.md): Run Strands Agents examples and send their agent traces to LiteLLM Lens.
- [Vercel AI SDK (TypeScript)](https://docs.litellm.ai/docs/proxy/lens/integrations/vercel-ai-sdk-js.md): Run Vercel AI SDK (TypeScript) examples and send their agent traces to LiteLLM Lens.
- [Vercel AI SDK (Python)](https://docs.litellm.ai/docs/proxy/lens/integrations/vercel-ai-sdk-py.md): Run Vercel AI SDK (Python) examples and send their agent traces to LiteLLM Lens.
- [Mastra](https://docs.litellm.ai/docs/proxy/lens/integrations/mastra.md): Run Mastra examples and send their agent traces to LiteLLM Lens.
- [Hermes Agent](https://docs.litellm.ai/docs/proxy/lens/integrations/hermes-agent.md): Run Hermes Agent examples and send their agent traces to LiteLLM Lens.
- [OpenTelemetry](https://docs.litellm.ai/docs/proxy/lens/integrations/opentelemetry.md): Run OpenTelemetry examples and send their agent traces to LiteLLM Lens.
- [OpenClaw](https://docs.litellm.ai/docs/proxy/lens/integrations/openclaw.md): Send OpenClaw agent activity to LiteLLM Lens.
- [Hermes](https://docs.litellm.ai/docs/proxy/lens/integrations/hermes.md): Send Hermes agent activity to LiteLLM Lens.
- [Trace coding agent sessions](https://docs.litellm.ai/docs/proxy/lens/coding-agents.md): Send your personal Claude Code and Codex sessions to LiteLLM Lens.
- [Claude Code](https://docs.litellm.ai/docs/proxy/lens/coding-agents/claude-code.md): Connect personal Claude Code sessions to LiteLLM Lens using the maintained setup guides.
- [Codex](https://docs.litellm.ai/docs/proxy/lens/coding-agents/codex.md): Connect personal Codex sessions to LiteLLM Lens using the maintained setup guides.
- [Investigations](https://docs.litellm.ai/docs/proxy/lens/investigations.md): Review agent activity, investigate failures, and read the supporting evidence.
- [API reference](https://docs.litellm.ai/docs/proxy/lens/api.md): List and read agent runs, read investigation findings, and query traces with SQL through LiteLLM.
- [Historical scalability measurements](https://docs.litellm.ai/docs/proxy/lens/scalability.md): Archived measurements of gateway-hosted Lens, with the current standalone benchmark procedure.

## Features

- [Features](https://docs.litellm.ai/docs/guides/index.md): Features are what LiteLLM does with your requests across providers: load balancing and fallbacks, streaming, tool...

## Claude, Codex, and Copilot subscriptions

- [Claude, Codex, and Copilot subscriptions](https://docs.litellm.ai/docs/subscriptions.md): Use your ChatGPT, Claude, or GitHub Copilot subscription through LiteLLM.

## Routing & Reliability

- [Cache-Aware Routing](https://docs.litellm.ai/docs/proxy/cache_aware_routing.md): Keep requests from the same session on one Kubernetes inference pod so it can reuse its prompt cache, with automatic...
- [Provider specific Wildcard routing](https://docs.litellm.ai/docs/wildcard_routing.md): Proxy all models from a provider
- [Health Check Driven Routing](https://docs.litellm.ai/docs/proxy/health_check_routing.md): Route traffic away from unhealthy deployments before users hit errors.
- [[BETA] Request Prioritization](https://docs.litellm.ai/docs/scheduler.md): Beta feature.
- [Client-Side Fallbacks](https://docs.litellm.ai/docs/proxy/client_side_fallbacks.md): Set fallback models per request from the client or the Python SDK, and control the prompt sent to each fallback model.
- [Timeouts](https://docs.litellm.ai/docs/proxy/timeout.md): The timeout set in router is for the entire length of the call, and is passed down to the completion() call level as...
- [[New] Fallback Management Endpoints](https://docs.litellm.ai/docs/proxy/fallback_management.md): Dedicated endpoints for managing model fallbacks separately from the general configuration.
- [Tag Based Routing](https://docs.litellm.ai/docs/proxy/tag_routing.md)
- [Budget Routing](https://docs.litellm.ai/docs/proxy/provider_budget_routing.md): LiteLLM Supports setting the following budgets:
- [UI - Router Settings for Keys and Teams](https://docs.litellm.ai/docs/proxy/keys_teams_router_settings.md): Configure router settings at the key and team level to achieve granular control over routing behavior, fallbacks,...
- [Manage Routing Groups](https://docs.litellm.ai/docs/proxy/ui/routing_groups.md): Routing groups let you apply different routing strategies to different models in the same router, for example...
- [Routing Plugins](https://docs.litellm.ai/docs/routing_plugins.md): Routing plugins are available from v1.92.x (Thursday's release).
- [Fusion model](https://docs.litellm.ai/docs/fusion.md): Use litellm/fusion-1 for judge-synthesized answers from a parallel model panel.

## Requests & Responses

- [Batching Completion()](https://docs.litellm.ai/docs/completion/batching.md)
- [Structured Outputs (JSON Mode)](https://docs.litellm.ai/docs/completion/json_mode.md)
- ['Thinking' / 'Reasoning Content'](https://docs.litellm.ai/docs/reasoning_content.md)

## Tool Calling

- [Function Calling](https://docs.litellm.ai/docs/completion/function_call.md): Checking if a model supports function calling
- [Web Search](https://docs.litellm.ai/docs/completion/web_search.md): Use web search with litellm
- [Web Search Integration](https://docs.litellm.ai/docs/integrations/websearch_interception.md): Enable transparent server-side web search execution for any LLM provider.
- [Web Fetch](https://docs.litellm.ai/docs/completion/web_fetch.md): The web fetch tool allows LLMs to retrieve full content from specified web pages and PDF documents.
- [Computer Use](https://docs.litellm.ai/docs/completion/computer_use.md): Computer use allows models to interact with computer interfaces by taking screenshots and performing actions like...
- [Code Interpreter](https://docs.litellm.ai/docs/guides/code_interpreter.md): Use OpenAI's Code Interpreter tool to execute Python code in a secure, sandboxed environment.
- [Advisor Tool](https://docs.litellm.ai/docs/completion/anthropic_advisor_tool.md): Pair a faster executor model with a higher-intelligence advisor model that provides strategic guidance mid-generation.
- [Using Vector Stores (Knowledge Bases)](https://docs.litellm.ai/docs/completion/knowledgebase.md)

## Multimodal

- [Using Vision Models](https://docs.litellm.ai/docs/completion/vision.md)
- [Using Audio Models](https://docs.litellm.ai/docs/completion/audio.md): How to send / receive audio to a /chat/completions endpoint
- [Using PDF Input](https://docs.litellm.ai/docs/completion/document_understanding.md): How to send / receive pdf's (other document types) to a /chat/completions endpoint
- [Image Generation in Chat Completions, Responses API](https://docs.litellm.ai/docs/completion/image_generation_chat.md): This guide covers how to generate images when using the chat/completions.

## Prompts & Context

- [Pre-fix Assistant Messages](https://docs.litellm.ai/docs/completion/prefix.md)
- [Predicted Outputs](https://docs.litellm.ai/docs/completion/predict_outputs.md): | Property | Details |

## Compatibility

- [Provider-specific Params](https://docs.litellm.ai/docs/completion/provider_specific_params.md): Providers might offer params not supported by OpenAI (e.g.
- [Drop Unsupported Params](https://docs.litellm.ai/docs/completion/drop_params.md): Drop unsupported OpenAI params by your LLM Provider.
- [Message Sanitization for Tool Calling for anthropic models](https://docs.litellm.ai/docs/completion/message_sanitization.md): Automatically fix common message formatting issues when using tool calling with modify_params=True

## liteagents SDK

- [liteagents SDK](https://docs.litellm.ai/docs/liteagents.md): Switch between DeepAgents, Pydantic AI, Claude Agent SDK, Codex, and OpenCode through one Python SDK, with the same...

## lite CLI

- [LiteLLM Proxy CLI](https://docs.litellm.ai/docs/proxy/management_cli.md): The lite CLI is a command-line tool for managing your LiteLLM proxy
- [lite autoroute](https://docs.litellm.ai/docs/learn/autorouter_cli.md): Try LiteLLM's complexity-based auto-routing locally against your real proxy with the lite CLI, and route Claude Code...

## Self-Hosted Coding Agents

- [Moyai](https://docs.litellm.ai/docs/self_hosted_coding_agents/moyai.md): Run open source coding agents in your cloud, choose models through LiteLLM, and track each teammate's costs.
- [Set up Moyai](https://docs.litellm.ai/docs/self_hosted_coding_agents/moyai/setup.md): Deploy Moyai on Modal, connect your LiteLLM gateway, verify your first cloud task, and connect a GitHub repository.
- [Moyai architecture](https://docs.litellm.ai/docs/self_hosted_coding_agents/moyai/architecture.md): How Moyai separates orchestration, agent sandboxes, and model access, with checkpoints, worker recovery, and handling...

## Cookbook

- [Cookbook](https://docs.litellm.ai/docs/tutorials/index.md): The Cookbook has step-by-step recipes.
- [Set up environment](https://docs.litellm.ai/docs/tutorials/installation.md): Let's get the necessary keys to set up our demo environment.
- [Create your first LLM playground](https://docs.litellm.ai/docs/tutorials/first_playground.md): Create a playground to evaluate multiple LLM Providers in less than 10 minutes.
- [Using Text Completion Format - with Completion()](https://docs.litellm.ai/docs/tutorials/text_completion.md): If your prefer interfacing with the OpenAI Text Completion format this tutorial covers how to use LiteLLM in this format
- [Mock Completion Responses - Save Testing Costs](https://docs.litellm.ai/docs/tutorials/mock_completion.md): Trying to test making LLM Completion calls without calling the LLM APIs ?
- [provider_specific_params](https://docs.litellm.ai/docs/tutorials/provider_specific_params.md)
- [Gradio Chatbot + LiteLLM Tutorial](https://docs.litellm.ai/docs/tutorials/gradio_integration.md): Simple tutorial for integrating LiteLLM completion calls with streaming Gradio chatbot demos
- [Replacing OpenAI ChatCompletion with Completion()](https://docs.litellm.ai/docs/tutorials/azure_openai.md)
- [Llama Together AI Tutorial](https://docs.litellm.ai/docs/tutorials/TogetherAI_liteLLM.md)
- [Llama2 - Huggingface Tutorial](https://docs.litellm.ai/docs/tutorials/huggingface_tutorial.md): Huggingface is an open source platform to deploy machine-learnings models.
- [CodeLlama - Code Infilling](https://docs.litellm.ai/docs/tutorials/huggingface_codellama.md): This tutorial shows how you can call CodeLlama (hosted on Huggingface PRO Inference Endpoints), to fill code.
- [Using Fine-Tuned gpt-3.5-turbo](https://docs.litellm.ai/docs/tutorials/finetuned_chat_gpt.md): LiteLLM allows you to call completion with your fine-tuned gpt-3.5-turbo models
- [Oobabooga Text Web API Tutorial](https://docs.litellm.ai/docs/tutorials/oobabooga.md)
- [Onboard Users for AI Exploration](https://docs.litellm.ai/docs/tutorials/default_team_self_serve.md): v1.73.0 introduces the ability to assign new users to Default Teams.
- [Microsoft SSO: Sync Groups, Members with LiteLLM](https://docs.litellm.ai/docs/tutorials/msft_sso.md): Sync Microsoft SSO Groups, Members with LiteLLM Teams.
- [[Beta] Routing based on request metadata](https://docs.litellm.ai/docs/tutorials/tag_management.md): Create routing rules based on request metadata.
- [Model Fallbacks (Provider Failover) w/ LiteLLM](https://docs.litellm.ai/docs/tutorials/model_fallbacks.md): Implement model fallbacks (provider failover) across OpenAI, Anthropic, and Azure with LiteLLM so a failing provider...
- [Using completion() with Fallbacks (Failover) for Reliability](https://docs.litellm.ai/docs/tutorials/fallbacks.md): Use completion() with model fallbacks (failover) so a failing provider automatically fails over to a backup model for...
- [Auto-Inject Prompt Caching Checkpoints](https://docs.litellm.ai/docs/tutorials/prompt_caching.md): Reduce costs by up to 90% by using LiteLLM to auto-inject prompt caching checkpoints.
- [Aporia Guardrails with LiteLLM Gateway](https://docs.litellm.ai/docs/tutorials/litellm_proxy_aporia.md): In this tutorial we will use LiteLLM AI Gateway with Aporia to detect PII in requests and profanity in responses
- [Presidio PII Masking with LiteLLM - Complete Tutorial](https://docs.litellm.ai/docs/tutorials/presidio_pii_masking.md): This tutorial will guide you through setting up PII (Personally Identifiable Information) masking with Microsoft...
- [File Search in the Responses API](https://docs.litellm.ai/docs/tutorials/file_search_responses_api.md): LiteLLM now supports file_search in the Responses API across both:
- [Chat with sample documents in MongoDB (BETA)](https://docs.litellm.ai/docs/tutorials/mongodb_vector_search.md): Create three fictional policy documents, index them in MongoDB Atlas, test semantic search in the LiteLLM Admin UI,...
- [Using Anthropic File API with LiteLLM Proxy](https://docs.litellm.ai/docs/tutorials/anthropic_file_usage.md)
- [Call Gemini Realtime API with Audio Input/Output](https://docs.litellm.ai/docs/tutorials/gemini_realtime_with_audio.md)
- [Elasticsearch Logging with LiteLLM](https://docs.litellm.ai/docs/tutorials/elasticsearch_logging.md): Send your LLM requests, responses, costs, and performance data to Elasticsearch for analytics and monitoring using...
- [Benchmark LLMs](https://docs.litellm.ai/docs/tutorials/compare_llms.md): Easily benchmark LLMs for a given question by viewing
- [Reliability test Multiple LLM Providers with LiteLLM](https://docs.litellm.ai/docs/tutorials/litellm_Test_Multiple_Providers.md)
- [Evaluate LLMs - MLflow Evals, Auto Eval](https://docs.litellm.ai/docs/tutorials/eval_suites.md)
- [Benchmark LLMs - LM Harness, FastEval, Flask](https://docs.litellm.ai/docs/tutorials/lm_evaluation_harness.md)

## Troubleshooting

- [UI Troubleshooting](https://docs.litellm.ai/docs/troubleshoot/ui_issues.md): If you're experiencing issues with the LiteLLM Admin UI, please include the following information when reporting.
- [Debugging a cost discrepancy](https://docs.litellm.ai/docs/troubleshoot/cost_discrepancy.md): Cost discrepancies between LiteLLM and your provider bill usually come from one of three areas: token ingestion, the...
- [Model missing after Reload Price Data](https://docs.litellm.ai/docs/troubleshoot/missing_model.md): A model that was just added to LiteLLM's pricing map does not show up in the Admin UI model picker, or cost tracking...
- [Latency Overhead Troubleshooting](https://docs.litellm.ai/docs/troubleshoot/latency_overhead.md): Use this guide when you see unexpected latency overhead between LiteLLM proxy and the LLM provider.
- [CPU Issue Classification & Reproduction](https://docs.litellm.ai/docs/troubleshoot/cpu_issues.md): 1.
- [Memory Issue Classification & Reproduction](https://docs.litellm.ai/docs/troubleshoot/memory_issues.md): 1.
- [Spend Update Queue Full Warnings](https://docs.litellm.ai/docs/troubleshoot/spend_queue_warnings.md)
- [MAX_CALLBACKS Limit](https://docs.litellm.ai/docs/troubleshoot/max_callbacks.md)
- [Troubleshooting Prisma Migration Errors](https://docs.litellm.ai/docs/troubleshoot/prisma_migrations.md): Common Prisma migration issues encountered when upgrading or downgrading LiteLLM proxy versions, plus the query engine...
- [Upgrading LiteLLM Proxy (uv/venv)](https://docs.litellm.ai/docs/troubleshoot/pip_venv_upgrade.md): Guide for upgrading LiteLLM Proxy when installed via uv in a virtual environment.
- [Safe Rollback Guide](https://docs.litellm.ai/docs/troubleshoot/rollback.md): This guide outlines the process for safely rolling back a LiteLLM Proxy deployment to a previous version.
- [Outage Troubleshooting](https://docs.litellm.ai/docs/troubleshoot.md): Use this page when LiteLLM nodes go down: pods restart or get OOMKilled, the load balancer marks targets unhealthy, or...

## Production rollout

- [Production rollout](https://docs.litellm.ai/docs/learn/enterprise_quickstart.md): Roll out LiteLLM Enterprise in five steps.

## Activate your license

- [Activate your license](https://docs.litellm.ai/docs/enterprise/activate.md): Set LITELLM_LICENSE on each LiteLLM gateway instance, confirm that the gateway shows Enterprise Edition, and see which...

## Moving from OSS

- [Moving from OSS](https://docs.litellm.ai/docs/enterprise/moving_from_oss.md): Move an open-source LiteLLM gateway to LiteLLM Enterprise.

## Compliance and SOC 2 Type II

- [Compliance and SOC 2 Type II](https://docs.litellm.ai/docs/enterprise/compliance.md): LiteLLM is SOC 2 Type II audited.

## Support and SLA

- [Support and SLA](https://docs.litellm.ai/docs/enterprise/support.md): LiteLLM Enterprise support.

## Version support

- [Version support](https://docs.litellm.ai/docs/enterprise/version_support.md): LiteLLM supports the four most recent stable minor lines with patch releases.

## Optional

- [Release notes](https://docs.litellm.ai/release_notes): Changelog for every LiteLLM release
- [All blog posts](https://docs.litellm.ai/blog.md): every post, newest first
- [Introducing the LiteLLM Private Registry](https://docs.litellm.ai/blog/litellm-private-registry.md) (2026-10-27): LiteLLM is committing to a private registry to provide early access to fixes for the most severe security vulnerabilities, ahead of public release.
- [Morph open-weight models on LiteLLM](https://docs.litellm.ai/blog/morph-provider.md) (2026-10-27): Use Morph's Kimi, GLM, and DeepSeek models through LiteLLM with tool calling, structured output, streaming, and prompt-cache cost tracking.
- [Abliterated models on LiteLLM](https://docs.litellm.ai/blog/abliterated-models.md) (2026-10-26): Call abliterated models from Refuseless and Abliteration through LiteLLM, with gateway access controls, streaming, tool calls, and cached-token pricing.
- [GPT-6.1 Terra API Pricing and Day 0 Support on LiteLLM](https://docs.litellm.ai/blog/gpt-6-1-terra.md) (2026-10-26): GPT-6.1 Terra costs $1 per 1M input tokens and $5 per 1M output tokens, half the price of GPT-6.1 Sol. LiteLLM supports it on day 0.
- [Rust Migration: /v1/messages end-to-end in Rust by Dec 1](https://docs.litellm.ai/blog/rust-migration-rerouted.md) (2026-10-26): By December 1st, the proxy will handle Anthropic messages requests 20x faster
- [OpenCode Go is now a LiteLLM provider](https://docs.litellm.ai/blog/opencode-go.md) (2026-10-24): Use your OpenCode Go subscription through LiteLLM. Connect once, route models to the right API, preserve sessions, and track estimated usage.
- [The attack chain behind the v1.107.0 LiteLLM release](https://docs.litellm.ai/blog/security-incident-oct-21-2026-attack-chain.md) (2026-10-23): How an internal key committed to a public repository in May led to three stolen GitHub tokens and the compromised litellm v1.107.0 release.
- [Tokenize text across vLLM and Watsonx](https://docs.litellm.ai/blog/unified-tokenize.md) (2026-10-23): Choose token counting for prompt budgets or /tokenize to inspect tokens. Use one request format across vLLM and Watsonx through LiteLLM.
- [LiteLLM v1.71.1 Benchmarks](https://docs.litellm.ai/docs/aiohttp_benchmarks.md)
- [Hosted Cache - api.litellm.ai (removed)](https://docs.litellm.ai/docs/caching/caching_api.md): The hosted cache backed by api.litellm.ai has been removed from LiteLLM.
- [LiteLLM - Local Caching](https://docs.litellm.ai/docs/caching/local_caching.md): Caching completion() and embedding() calls when switched on
- [Multiple Deployments](https://docs.litellm.ai/docs/completion/multiple_deployments.md): If you have multiple deployments of the same model, you can pass the list of deployments, and LiteLLM will return the...
- [Shared Session Support](https://docs.litellm.ai/docs/completion/shared_session.md)
- [Contact Us](https://docs.litellm.ai/docs/contact.md): Have a question, need help getting set up, or want to talk about pricing?
- [hosted_debugging](https://docs.litellm.ai/docs/debugging/hosted_debugging.md)
- [Local Debugging](https://docs.litellm.ai/docs/debugging/local_debugging.md): There's 2 ways to do local debugging - litellm.turnondebug() and by passing in a custom function...
- [Get Started](https://docs.litellm.ai/docs/default_code_snippet.md): This section assumes you've already added your API keys in
- [litellm.aembedding()](https://docs.litellm.ai/docs/embedding/async_embedding.md): LiteLLM provides an asynchronous version of the embedding function called aembedding
- [litellm.moderation()](https://docs.litellm.ai/docs/embedding/moderation.md): LiteLLM supports the moderation endpoint for OpenAI
- [Gemini Image Generation Migration Guide](https://docs.litellm.ai/docs/extras/gemini_img_migration.md): Who is impacted by this change?
- [LiteLLM Proxy - Locust Load Test](https://docs.litellm.ai/docs/load_test.md): Locust Load Test LiteLLM Proxy
- [[Beta] Guardrails](https://docs.litellm.ai/docs/old_guardrails.md): Setup Prompt Injection Detection, Secret Detection on LiteLLM Proxy
- [GPT Migrate](https://docs.litellm.ai/docs/projects/GPT Migrate.md): Easily migrate your codebase from one framework or language to another.
- [LiteLLM Proxy](https://docs.litellm.ai/docs/projects/LiteLLM Proxy.md)
- [Anthropic Effort Parameter](https://docs.litellm.ai/docs/providers/anthropic_effort.md): Control how many tokens Claude uses when responding with the effort parameter, trading off between response...
- [Anthropic Programmatic Tool Calling](https://docs.litellm.ai/docs/providers/anthropic_programmatic_tool_calling.md): Programmatic tool calling allows Claude to write code that calls your tools programmatically within a code execution...
- [Anthropic Tool Input Examples](https://docs.litellm.ai/docs/providers/anthropic_tool_input_examples.md): Provide concrete examples of valid tool inputs to help Claude understand how to use your tools more effectively.
- [Azure AI Image Editing](https://docs.litellm.ai/docs/providers/azure_ai_img_edit.md): Azure AI provides powerful image editing capabilities using FLUX models from Black Forest Labs to modify existing...
- [Azure Anthropic (Claude via Azure Foundry)](https://docs.litellm.ai/docs/providers/azure/azure_anthropic.md): LiteLLM supports Claude models deployed via Microsoft Azure Foundry, including Claude Sonnet 4.5, Claude Haiku 4.5,...
- [Empower](https://docs.litellm.ai/docs/providers/empower.md): LiteLLM supports all models on Empower.
- [GigaChat](https://docs.litellm.ai/docs/providers/gigachat.md)
- [RunwayML - Text-to-Speech](https://docs.litellm.ai/docs/providers/runwayml/text-to-speech.md)
- [watsonx.ai Rerank](https://docs.litellm.ai/docs/providers/watsonx/rerank.md)
- [LiteLLM Keys (community key, discontinued)](https://docs.litellm.ai/docs/proxy_api.md): The free community sk-litellm-...
- [Credential Usage Tracking](https://docs.litellm.ai/docs/proxy/credential_usage_tracking.md): When a model is attached to a reusable credential, LiteLLM automatically injects the credential name as a tag on every...
- [Embeddings - /embeddings](https://docs.litellm.ai/docs/proxy/embedding.md): See supported Embedding Providers & Models here
- [Cato Networks](https://docs.litellm.ai/docs/proxy/guardrails/cato_networks.md)
- [Prompt Security](https://docs.litellm.ai/docs/proxy/guardrails/prompt_security.md): Use Prompt Security to protect your LLM applications from prompt injection attacks, jailbreaks, harmful content, PII...
- [GET Daily Spend, Usage Metrics](https://docs.litellm.ai/docs/proxy/metrics.md): The GET /dailymetrics endpoint that this page used to describe has been removed from the proxy.
- [LiteLLM Proxy Performance](https://docs.litellm.ai/docs/proxy/perf.md): The numbers on this page compare the proxy against calling a provider directly.
- [Reject Client-Side Metadata Tags](https://docs.litellm.ai/docs/proxy/reject_clientside_metadata_tags.md)
- [Shared Health Check State Across Pods](https://docs.litellm.ai/docs/proxy/shared_health_check.md): This feature enables coordination of health checks across multiple LiteLLM proxy pods to avoid duplicate health checks...
- [Auto Sync Anthropic Beta Headers](https://docs.litellm.ai/docs/proxy/sync_anthropic_beta_headers.md): Automatically keep your Anthropic beta headers configuration up to date without restarting your service.
- [User Onboarding Guide](https://docs.litellm.ai/docs/proxy/user_onboarding.md): A step-by-step guide to help admins onboard users to your LiteLLM proxy instance and help users get started with their...
- [SearchAPI.io (Google Search)](https://docs.litellm.ai/docs/search/searchapi.md): Get started by creating a free API key via https://www.searchapi.io/.
- [Secret Managers Overview](https://docs.litellm.ai/docs/secret.md): LiteLLM supports reading secrets (eg.
- [Claude Code Gateway reference](https://docs.litellm.ai/docs/tutorials/claude_code_gateway_reference.md): Start with Claude Code SSO setup to choose and configure a sign-in flow.
- [Codex CLI - Granular Cost Tracking](https://docs.litellm.ai/docs/tutorials/codex_customer_tracking.md): Track Codex CLI usage by customer or tags using LiteLLM proxy.
- [Comparing LLMs on a Test Set using LiteLLM](https://docs.litellm.ai/docs/tutorials/compare_llms_2.md): LiteLLM allows you to use any LLM as a drop in replacement for
- [Customize Prompt Templates on OpenAI-Compatible server](https://docs.litellm.ai/docs/tutorials/model_config_proxy.md): You will learn: How to set a custom prompt template on our OpenAI compatible server.
- [/batchPredictionJobs](https://docs.litellm.ai/docs/vertex_batch_passthrough.md): LiteLLM supports Vertex AI batch prediction jobs through passthrough endpoints, allowing you to create and manage...

### Supported Endpoints

[/assistants](https://docs.litellm.ai/docs/assistants.md), [/batches](https://docs.litellm.ai/docs/batches.md), [[BETA] LiteLLM Managed Files with Batches](https://docs.litellm.ai/docs/proxy/managed_batches.md), [Unmanaged Vertex AI Batches](https://docs.litellm.ai/docs/proxy/unmanaged_vertex_batches.md), [/containers](https://docs.litellm.ai/docs/containers.md), [/containers/files](https://docs.litellm.ai/docs/container_files.md), [Output](https://docs.litellm.ai/docs/completion/output.md), [Usage](https://docs.litellm.ai/docs/completion/usage.md), [Custom HTTP Handler](https://docs.litellm.ai/docs/completion/http_handler_config.md), [/converse](https://docs.litellm.ai/docs/bedrock_converse.md), [Provider Files Endpoints](https://docs.litellm.ai/docs/files_endpoints.md), [[BETA] LiteLLM Managed Files](https://docs.litellm.ai/docs/proxy/litellm_managed_files.md), [/fine_tuning](https://docs.litellm.ai/docs/fine_tuning.md), [[BETA] LiteLLM Managed Files with Finetuning](https://docs.litellm.ai/docs/proxy/managed_finetuning.md), [/evals](https://docs.litellm.ai/docs/evals_api.md), [/generateContent](https://docs.litellm.ai/docs/generateContent.md), [/guardrails/apply_guardrail](https://docs.litellm.ai/docs/apply_guardrail.md), [/invoke](https://docs.litellm.ai/docs/bedrock_invoke.md), [/interactions](https://docs.litellm.ai/docs/interactions.md), [Google AI Studio - Managed Agents](https://docs.litellm.ai/docs/managed_agents.md), [/memory](https://docs.litellm.ai/docs/memory_management.md), [/images/edits](https://docs.litellm.ai/docs/image_edits.md), [[BETA] Image Variations](https://docs.litellm.ai/docs/image_variations.md), [/videos](https://docs.litellm.ai/docs/videos.md), [Veo Video Generation with Google AI Studio](https://docs.litellm.ai/docs/proxy/veo_video_generation.md), [Vector Stores - Overview](https://docs.litellm.ai/docs/vector_stores/index.md), [LiteLLM Managed Vector Stores](https://docs.litellm.ai/docs/vector_stores/managed_vector_stores.md), [/vector_stores - Create Vector Store](https://docs.litellm.ai/docs/vector_stores/create.md), [/vector_stores/search - Search Vector Store](https://docs.litellm.ai/docs/vector_stores/search.md), [/vector_stores/\{vector_store_id\}/files](https://docs.litellm.ai/docs/vector_store_files.md), [/rag/ingest](https://docs.litellm.ai/docs/rag_ingest.md), [/rag/query](https://docs.litellm.ai/docs/rag_query.md), [/v1/messages](https://docs.litellm.ai/docs/anthropic_unified/index.md), [Structured Output /v1/messages](https://docs.litellm.ai/docs/anthropic_unified/structured_output.md), [v1/messages → /responses Parameter Mapping](https://docs.litellm.ai/docs/anthropic_unified/messages_to_responses_mapping.md), [Native /v1/messages Passthrough for OpenAI-Compatible Providers](https://docs.litellm.ai/docs/anthropic_unified/native_passthrough.md), [Token Counting](https://docs.litellm.ai/docs/count_tokens.md), [/v1/messages/count_tokens](https://docs.litellm.ai/docs/anthropic_count_tokens.md), [Unified tokenization](https://docs.litellm.ai/docs/tokenize.md), [/v1/decisions and /v1/systemone](https://docs.litellm.ai/docs/decisions.md), [Decision model benchmarks](https://docs.litellm.ai/docs/decisions/benchmark.md), [/v1/extraction](https://docs.litellm.ai/docs/extraction.md), [/moderations](https://docs.litellm.ai/docs/moderation.md), [/ocr](https://docs.litellm.ai/docs/ocr.md), [Why Pass-Through Endpoints?](https://docs.litellm.ai/docs/pass_through/intro.md), [Anthropic Passthrough](https://docs.litellm.ai/docs/pass_through/anthropic_completion.md), [AssemblyAI](https://docs.litellm.ai/docs/pass_through/assembly_ai.md), [Bedrock (boto3) SDK](https://docs.litellm.ai/docs/pass_through/bedrock.md), [Azure Passthrough](https://docs.litellm.ai/docs/pass_through/azure_passthrough.md), [Azure AI Speech](https://docs.litellm.ai/docs/pass_through/azure_speech.md), [Cohere SDK](https://docs.litellm.ai/docs/pass_through/cohere.md), [Amazon Comprehend Medical](https://docs.litellm.ai/docs/pass_through/comprehend_medical.md), [Cursor Cloud Agents](https://docs.litellm.ai/docs/pass_through/cursor.md), [Deepgram Realtime (/listen) WebSocket Passthrough](https://docs.litellm.ai/docs/pass_through/deepgram_listen_websocket.md), [GigaChat](https://docs.litellm.ai/docs/pass_through/gigachat.md), [Google AI Studio SDK](https://docs.litellm.ai/docs/pass_through/google_ai_studio.md), [Langfuse SDK](https://docs.litellm.ai/docs/pass_through/langfuse.md), [Mistral](https://docs.litellm.ai/docs/pass_through/mistral.md), [OpenAI Passthrough](https://docs.litellm.ai/docs/pass_through/openai_passthrough.md), [TinyFish Agent](https://docs.litellm.ai/docs/pass_through/tinyfish.md), [Amazon Transcribe](https://docs.litellm.ai/docs/pass_through/transcribe.md), [TypeSafe AI (Jev)](https://docs.litellm.ai/docs/pass_through/typesafe.md), [Passthrough Managed IDs](https://docs.litellm.ai/docs/proxy/passthrough_managed_ids.md), [Vertex AI SDK](https://docs.litellm.ai/docs/pass_through/vertex_ai.md), [Vertex AI Live API WebSocket Passthrough](https://docs.litellm.ai/docs/pass_through/vertex_ai_live_websocket.md), [Vertex AI Search Datastores (Pass-through)](https://docs.litellm.ai/docs/pass_through/vertex_ai_search_datastores.md), [VLLM](https://docs.litellm.ai/docs/pass_through/vllm.md), [Create Pass Through Endpoints](https://docs.litellm.ai/docs/proxy/pass_through.md), [Pass-Through Cost & Usage Tracking](https://docs.litellm.ai/docs/proxy/pass_through_cost_tracking.md), [Guardrails on Pass-Through Endpoints](https://docs.litellm.ai/docs/proxy/pass_through_guardrails.md), [/realtime](https://docs.litellm.ai/docs/realtime.md), [/realtime - WebRTC Support](https://docs.litellm.ai/docs/proxy/realtime_webrtc.md), [/rerank](https://docs.litellm.ai/docs/rerank.md), [Prompt Management with Responses API](https://docs.litellm.ai/docs/prompt_management.md), [/responses/compact](https://docs.litellm.ai/docs/response_api_compact.md), [Native /v1/responses Passthrough for OpenAI-Compatible Providers](https://docs.litellm.ai/docs/response_api_native_passthrough.md), [Overview](https://docs.litellm.ai/docs/search/index.md), [Perplexity AI Search](https://docs.litellm.ai/docs/search/perplexity.md), [Tavily Search](https://docs.litellm.ai/docs/search/tavily.md), [Exa AI Search](https://docs.litellm.ai/docs/search/exa_ai.md), [Brave Search](https://docs.litellm.ai/docs/search/brave.md), [Parallel AI Search](https://docs.litellm.ai/docs/search/parallel_ai.md), [Google Programmable Search Engine (PSE)](https://docs.litellm.ai/docs/search/google_pse.md), [DataForSEO Search](https://docs.litellm.ai/docs/search/dataforseo.md), [Firecrawl Search](https://docs.litellm.ai/docs/search/firecrawl.md), [SearXNG Search](https://docs.litellm.ai/docs/search/searxng.md), [Linkup Search](https://docs.litellm.ai/docs/search/linkup.md), [Serper Search](https://docs.litellm.ai/docs/search/serper.md), [You.com Search](https://docs.litellm.ai/docs/search/you_com.md), [APISerpent Search](https://docs.litellm.ai/docs/search/apiserpent.md), [Amazon Bedrock AgentCore Web Search](https://docs.litellm.ai/docs/search/agentcore.md), [Nimble Search](https://docs.litellm.ai/docs/search/nimble.md), [Grounding with Bing Search (Microsoft Foundry)](https://docs.litellm.ai/docs/search/bing_grounding.md), [DeepSeek Search](https://docs.litellm.ai/docs/search/deepseek.md), [/skills - Anthropic Skills API](https://docs.litellm.ai/docs/skills.md)

### Resources

[Custom Pricing - SageMaker, Azure, etc](https://docs.litellm.ai/docs/sdk_custom_pricing.md), [Migration Guide - LiteLLM v1.0.0+](https://docs.litellm.ai/docs/migration.md), [Usage telemetry](https://docs.litellm.ai/docs/proxy/telemetry.md), [LiteLLM Self-Hosted Security & Encryption FAQ](https://docs.litellm.ai/docs/proxy/security_encryption_faq.md), [Docker Image Security Guide](https://docs.litellm.ai/docs/proxy/docker_image_security.md), [Migration Policy](https://docs.litellm.ai/docs/migration_policy.md), [API Stability Policy](https://docs.litellm.ai/docs/api_stability_policy.md), [Shared Responsibility Model](https://docs.litellm.ai/docs/shared_responsibility.md), [Release Cycle](https://docs.litellm.ai/docs/proxy/release_cycle.md), [LiteLLM Proxy - 1K RPS Load test on locust](https://docs.litellm.ai/docs/load_test_advanced.md), [LiteLLM SDK vs OpenAI](https://docs.litellm.ai/docs/load_test_sdk.md), [Multi-Instance TPM/RPM (litellm.Router)](https://docs.litellm.ai/docs/load_test_rpm.md), [Smolagents](https://docs.litellm.ai/docs/projects/smolagents.md), [mini-swe-agent](https://docs.litellm.ai/docs/projects/mini-swe-agent.md), [OpenAI Agents SDK](https://docs.litellm.ai/docs/projects/openai-agents.md), [Google ADK (Agent Development Kit)](https://docs.litellm.ai/docs/projects/Google ADK.md), [Agent Lightning](https://docs.litellm.ai/docs/projects/Agent Lightning.md), [Harbor](https://docs.litellm.ai/docs/projects/Harbor.md), [CompatCanary](https://docs.litellm.ai/docs/projects/CompatCanary.md), [Microsoft GraphRAG](https://docs.litellm.ai/docs/projects/GraphRAG.md), [Docq.AI](https://docs.litellm.ai/docs/projects/Docq.AI.md), [PDL](https://docs.litellm.ai/docs/projects/PDL.md), [OpenInterpreter](https://docs.litellm.ai/docs/projects/OpenInterpreter.md), [Elroy](https://docs.litellm.ai/docs/projects/Elroy.md), [dbally](https://docs.litellm.ai/docs/projects/dbally.md), [FastREPL](https://docs.litellm.ai/docs/projects/FastREPL.md), [PROMPTMETHEUS](https://docs.litellm.ai/docs/projects/PROMPTMETHEUS.md), [Codium PR Agent](https://docs.litellm.ai/docs/projects/Codium PR Agent.md), [Prompt2Model](https://docs.litellm.ai/docs/projects/Prompt2Model.md), [SalesGPT](https://docs.litellm.ai/docs/projects/SalesGPT.md), [Softgen](https://docs.litellm.ai/docs/projects/Softgen.md), [Quivr](https://docs.litellm.ai/docs/projects/Quivr.md), [Langstream](https://docs.litellm.ai/docs/projects/Langstream.md), [Otter](https://docs.litellm.ai/docs/projects/Otter.md), [YiVal](https://docs.litellm.ai/docs/projects/YiVal.md), [llmcord.py](https://docs.litellm.ai/docs/projects/llm_cord.md), [pgai](https://docs.litellm.ai/docs/projects/pgai.md), [GPTLocalhost](https://docs.litellm.ai/docs/projects/GPTLocalhost.md), [HolmesGPT](https://docs.litellm.ai/docs/projects/HolmesGPT.md), [Railtracks](https://docs.litellm.ai/docs/projects/Railtracks.md), [Code Quality](https://docs.litellm.ai/docs/extras/code_quality.md), [Rules](https://docs.litellm.ai/docs/rules.md), [[DEPRECATED] Team-based Routing](https://docs.litellm.ai/docs/proxy/team_based_routing.md), [[DEPRECATED] Region-based Routing](https://docs.litellm.ai/docs/proxy/customer_routing.md), [[OLD PROXY [NEW proxy here](./simple_proxy)] Local LiteLLM Proxy Server](https://docs.litellm.ai/docs/proxy_server.md), [Contributing Code](https://docs.litellm.ai/docs/extras/contributing_code.md), [Adding OpenAI-Compatible Providers](https://docs.litellm.ai/docs/contributing/adding_openai_compatible_providers.md), [Directory Structure](https://docs.litellm.ai/docs/adding_provider/directory_structure.md), [Add Rerank Provider](https://docs.litellm.ai/docs/adding_provider/new_rerank_provider.md), [Contributing to Documentation](https://docs.litellm.ai/docs/extras/contributing.md), [Contributing - UI](https://docs.litellm.ai/docs/contributing.md)

### Integrations

[Integrations](https://docs.litellm.ai/docs/integrations/index.md), [Open WebUI](https://docs.litellm.ai/docs/tutorials/openweb_ui.md), [Open WebUI for Organizations (SSO, Teams, Skills, Artifacts)](https://docs.litellm.ai/docs/tutorials/openwebui_org_rollout.md), [Claude Code Compatibility](https://docs.litellm.ai/docs/claude_code_compatibility.md), [Claude Code - Cut Costs](https://docs.litellm.ai/docs/tutorials/claude_code_cut_costs.md), [Auto Router with Claude Code and Claude Desktop](https://docs.litellm.ai/docs/tutorials/claude_code_autorouter.md), [Claude Code Quickstart](https://docs.litellm.ai/docs/tutorials/claude_responses_api.md), [Claude Code - Granular Cost Tracking](https://docs.litellm.ai/docs/tutorials/claude_code_customer_tracking.md), [Claude Code - Budget Status Line](https://docs.litellm.ai/docs/tutorials/claude_code_budget_statusline.md), [Claude Code - Prompt Cache Routing](https://docs.litellm.ai/docs/tutorials/claude_code_prompt_cache_routing.md), [Claude Code - WebSearch Across All Providers](https://docs.litellm.ai/docs/tutorials/claude_code_websearch.md), [Use Claude Code with MCPs](https://docs.litellm.ai/docs/tutorials/claude_mcp.md), [Claude Code Plugin Marketplace (Managed Skills)](https://docs.litellm.ai/docs/tutorials/claude_code_plugin_marketplace.md), [Claude Code - Managing Anthropic Beta Headers](https://docs.litellm.ai/docs/tutorials/claude_code_beta_headers.md), [Claude Code - Context Management](https://docs.litellm.ai/docs/claude_code_context_management.md), [Claude Code - Auto Mode on Non-Claude Models](https://docs.litellm.ai/docs/claude_code_auto_mode.md), [Retool Assist](https://docs.litellm.ai/docs/tutorials/retool_assist.md), [Track Usage for Coding Tools](https://docs.litellm.ai/docs/tutorials/cost_tracking_coding.md), [LiteLLM Skills](https://docs.litellm.ai/docs/tutorials/claude_code_skills.md), [OpenAI Agents SDK with LiteLLM](https://docs.litellm.ai/docs/tutorials/openai_agents_sdk.md), [Claude Agent SDK with LiteLLM](https://docs.litellm.ai/docs/tutorials/claude_agent_sdk.md), [CopilotKit SDK with LiteLLM](https://docs.litellm.ai/docs/tutorials/copilotkit_sdk.md), [Google ADK with LiteLLM](https://docs.litellm.ai/docs/tutorials/google_adk.md), [Google GenAI SDK with LiteLLM](https://docs.litellm.ai/docs/tutorials/google_genai_sdk.md), [LiveKit xAI Realtime Voice Agent](https://docs.litellm.ai/docs/tutorials/livekit_xai_realtime.md), [Letta Integration](https://docs.litellm.ai/docs/integrations/letta.md), [Scalekit with LiteLLM](https://docs.litellm.ai/docs/tutorials/scalekit_agentkit.md), [Using ChatLiteLLM() - Langchain](https://docs.litellm.ai/docs/langchain/index.md), [Instructor](https://docs.litellm.ai/docs/tutorials/instructor.md), [Observability](https://docs.litellm.ai/docs/integrations/observability_integrations.md), [Migrating to OpenTelemetry v2](https://docs.litellm.ai/docs/observability/opentelemetry_v2_migration.md), [OpenTelemetry v2](https://docs.litellm.ai/docs/observability/opentelemetry_v2.md), [OpenTelemetry v1](https://docs.litellm.ai/docs/observability/opentelemetry_integration.md), [SigNoz](https://docs.litellm.ai/docs/observability/signoz.md), [AgentOps](https://docs.litellm.ai/docs/observability/agentops_integration.md), [Argilla](https://docs.litellm.ai/docs/observability/argilla.md), [Arize AX](https://docs.litellm.ai/docs/observability/arize_integration.md), [Arize Phoenix](https://docs.litellm.ai/docs/observability/phoenix_integration.md), [Athina](https://docs.litellm.ai/docs/observability/athina_integration.md), [Braintrust](https://docs.litellm.ai/docs/observability/braintrust.md), [Grafana Cloud](https://docs.litellm.ai/docs/observability/grafana_cloud.md), [Comet Opik](https://docs.litellm.ai/docs/observability/opik_integration.md), [DeepEval](https://docs.litellm.ai/docs/observability/deepeval_integration.md), [Helicone](https://docs.litellm.ai/docs/observability/helicone_integration.md), [Humanloop](https://docs.litellm.ai/docs/observability/humanloop.md), [Langfuse](https://docs.litellm.ai/docs/observability/langfuse_integration.md), [LangSmith](https://docs.litellm.ai/docs/observability/langsmith_integration.md), [Langtrace](https://docs.litellm.ai/docs/observability/langtrace_integration.md), [Levo](https://docs.litellm.ai/docs/observability/levo_integration.md), [Literal AI](https://docs.litellm.ai/docs/observability/literalai_integration.md), [Logfire](https://docs.litellm.ai/docs/observability/logfire_integration.md), [Lunary](https://docs.litellm.ai/docs/observability/lunary_integration.md), [MLflow](https://docs.litellm.ai/docs/observability/mlflow.md), [PromptLayer](https://docs.litellm.ai/docs/observability/promptlayer_integration.md), [Qualifire](https://docs.litellm.ai/docs/observability/qualifire_integration.md), [Weights & Biases](https://docs.litellm.ai/docs/observability/wandb_integration.md), [Azure Sentinel](https://docs.litellm.ai/docs/observability/azure_sentinel.md), [Datadog](https://docs.litellm.ai/docs/observability/datadog.md), [Google Cloud Storage](https://docs.litellm.ai/docs/observability/gcs_bucket_integration.md), [New Relic](https://docs.litellm.ai/docs/observability/newrelic.md), [Parseable](https://docs.litellm.ai/docs/observability/parseable.md), [PostHog](https://docs.litellm.ai/docs/observability/posthog_integration.md), [Sentry](https://docs.litellm.ai/docs/observability/sentry.md), [Slack](https://docs.litellm.ai/docs/observability/slack_integration.md), [Splunk Observability Cloud](https://docs.litellm.ai/docs/observability/splunk_observability_cloud.md), [Sumo Logic](https://docs.litellm.ai/docs/observability/sumologic_integration.md), [Supabase](https://docs.litellm.ai/docs/observability/supabase_integration.md), [Databricks Zerobus](https://docs.litellm.ai/docs/observability/zerobus.md), [Callbacks](https://docs.litellm.ai/docs/observability/callbacks.md), [Custom Callbacks](https://docs.litellm.ai/docs/observability/custom_callback.md), [Generic API](https://docs.litellm.ai/docs/observability/generic_api.md), [Raw Request/Response Logging](https://docs.litellm.ai/docs/observability/raw_request_response.md), [Scrub Logged Data](https://docs.litellm.ai/docs/observability/scrub_data.md), [CloudZero](https://docs.litellm.ai/docs/observability/cloudzero.md), [Focus Export (Experimental)](https://docs.litellm.ai/docs/observability/focus.md), [Greenscale](https://docs.litellm.ai/docs/observability/greenscale_integration.md), [Lago](https://docs.litellm.ai/docs/observability/lago.md), [Mavvrik](https://docs.litellm.ai/docs/observability/mavvrik.md), [OpenMeter](https://docs.litellm.ai/docs/observability/openmeter.md), [PointFive](https://docs.litellm.ai/docs/observability/pointfive.md), [Ramp](https://docs.litellm.ai/docs/observability/ramp_integration.md), [Ternary](https://docs.litellm.ai/docs/observability/ternary.md), [Vantage](https://docs.litellm.ai/docs/observability/vantage.md), [Aim Security](https://docs.litellm.ai/docs/proxy/guardrails/aim_security.md), [Airia](https://docs.litellm.ai/docs/proxy/guardrails/airia.md), [Akto](https://docs.litellm.ai/docs/proxy/guardrails/akto.md), [Alice](https://docs.litellm.ai/docs/proxy/guardrails/alice.md), [Aporia](https://docs.litellm.ai/docs/proxy/guardrails/aporia_api.md), [Azure Content Safety Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/azure_content_guardrail.md), [Bedrock Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/bedrock.md), [Context Compression (Compresr)](https://docs.litellm.ai/docs/proxy/guardrails/compresr.md), [Conduct Guard](https://docs.litellm.ai/docs/proxy/guardrails/conduct.md), [CrowdStrike AIDR](https://docs.litellm.ai/docs/proxy/guardrails/crowdstrike_aidr.md), [Custom Code Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/custom_code_guardrail.md), [Custom Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/custom_guardrail.md), [Decision Model Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/decision_model.md), [DynamoAI Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/dynamoai.md), [EnkryptAI Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/enkryptai.md), [Gray Swan Cygnal Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/grayswan.md), [Guardrails AI](https://docs.litellm.ai/docs/proxy/guardrails/guardrails_ai.md), [HiddenLayer Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/hiddenlayer.md), [IBM Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/ibm_guardrails.md), [Javelin Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/javelin.md), [Lakera AI](https://docs.litellm.ai/docs/proxy/guardrails/lakera_ai.md), [Lasso Security](https://docs.litellm.ai/docs/proxy/guardrails/lasso_security.md), [LLM-as-a-Judge](https://docs.litellm.ai/docs/proxy/guardrails/llm_as_a_judge.md), [LLM Shield Proxy](https://docs.litellm.ai/docs/proxy/guardrails/llm_shield_proxy.md), [Microsoft Agent 365 Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/microsoft_agent_365.md), [Microsoft Purview Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/microsoft_purview.md), [Google Cloud Model Armor](https://docs.litellm.ai/docs/proxy/guardrails/model_armor.md), [Noma Security](https://docs.litellm.ai/docs/proxy/guardrails/noma_security.md), [Onyx Security](https://docs.litellm.ai/docs/proxy/guardrails/onyx_security.md), [OpenAI Moderation](https://docs.litellm.ai/docs/proxy/guardrails/openai_moderation.md), [Pangea](https://docs.litellm.ai/docs/proxy/guardrails/pangea.md), [PANW Prisma AIRS](https://docs.litellm.ai/docs/proxy/guardrails/panw_prisma_airs.md), [PII, PHI Masking - Presidio](https://docs.litellm.ai/docs/proxy/guardrails/pii_masking_v2.md), [Pillar Security](https://docs.litellm.ai/docs/proxy/guardrails/pillar_security.md), [In-memory Prompt Injection Detection](https://docs.litellm.ai/docs/proxy/guardrails/prompt_injection.md), [PromptGuard](https://docs.litellm.ai/docs/proxy/guardrails/promptguard.md), [Qostodian Nexus by Qohash](https://docs.litellm.ai/docs/proxy/guardrails/qohash_qostodian_nexus.md), [Qualifire](https://docs.litellm.ai/docs/proxy/guardrails/qualifire.md), [RepelloAI Argus](https://docs.litellm.ai/docs/proxy/guardrails/repelloai.md), [Rubrik Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/rubrik.md), [Secret Detection/Redaction (Enterprise-only)](https://docs.litellm.ai/docs/proxy/guardrails/secret_detection.md), [Sensitive Data Routing (Built-in Guardrail)](https://docs.litellm.ai/docs/proxy/guardrails/sensitive_data_routing.md), [Straiker](https://docs.litellm.ai/docs/proxy/guardrails/straiker.md), [LiteLLM Tool Permission Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/tool_permission.md), [Relevance-Based Compaction (TypeSafe / Jev)](https://docs.litellm.ai/docs/proxy/guardrails/typesafe.md), [Vigil Guard](https://docs.litellm.ai/docs/proxy/guardrails/vigil_guard.md), [Wingback](https://docs.litellm.ai/docs/proxy/guardrails/wingback.md), [XecGuard](https://docs.litellm.ai/docs/proxy/guardrails/xecguard.md), [Zscaler AI Guard](https://docs.litellm.ai/docs/proxy/guardrails/zscaler_ai_guard.md), [Alerting / Webhooks](https://docs.litellm.ai/docs/proxy/alerting.md), [PagerDuty Alerting](https://docs.litellm.ai/docs/proxy/pagerduty.md), [Grafana Pyroscope CPU profiling](https://docs.litellm.ai/docs/proxy/pyroscope_profiling.md), [LiteLLM AI Gateway Prompt Management](https://docs.litellm.ai/docs/proxy/litellm_prompt_management.md), [Custom Prompt Management](https://docs.litellm.ai/docs/proxy/custom_prompt_management.md), [LiteLLM Prompt Management (GitOps)](https://docs.litellm.ai/docs/proxy/native_litellm_prompt.md), [Prompt Management](https://docs.litellm.ai/docs/proxy/prompt_management.md), [Arize Phoenix Prompt Management](https://docs.litellm.ai/docs/proxy/arize_phoenix_prompts.md), [Be an Integration Partner](https://docs.litellm.ai/docs/integrations/community.md), [Contribute Custom Webhook API](https://docs.litellm.ai/docs/contribute_integration/custom_webhook_api.md), [[BETA] Generic Prompt Management API - Integrate Without a PR](https://docs.litellm.ai/docs/adding_provider/generic_prompt_management_api.md)

### Providers

[AI Providers](https://docs.litellm.ai/docs/providers.md), [OpenAI](https://docs.litellm.ai/docs/providers/openai.md), [OpenAI - Response API](https://docs.litellm.ai/docs/providers/openai/responses_api.md), [OpenAI - Text-to-speech](https://docs.litellm.ai/docs/providers/openai/text_to_speech.md), [OpenAI Video Generation](https://docs.litellm.ai/docs/providers/openai/videos.md), [OpenAI Fast & Ultrafast mode](https://docs.litellm.ai/docs/providers/openai/ultrafast.md), [OpenAI (Text Completion)](https://docs.litellm.ai/docs/providers/text_completion_openai.md), [Anthropic](https://docs.litellm.ai/docs/providers/anthropic.md), [Tool Search](https://docs.litellm.ai/docs/providers/anthropic_tool_search.md), [Preserved Thinking Prefix Stability](https://docs.litellm.ai/docs/providers/anthropic_preserved_thinking.md), [Gemini - Google AI Studio](https://docs.litellm.ai/docs/providers/gemini.md), [Gemini Video Generation (Veo)](https://docs.litellm.ai/docs/providers/gemini/videos.md), [Gemini Lyria (music generation)](https://docs.litellm.ai/docs/providers/gemini/music.md), [Gemini File Search](https://docs.litellm.ai/docs/providers/gemini_file_search.md), [[BETA] Google AI Studio (Gemini) Files API](https://docs.litellm.ai/docs/providers/google_ai_studio/files.md), [Google AI Studio Image Generation](https://docs.litellm.ai/docs/providers/google_ai_studio/image_gen.md), [Gemini Realtime API - Google AI Studio](https://docs.litellm.ai/docs/providers/google_ai_studio/realtime.md), [VertexAI [Gemini]](https://docs.litellm.ai/docs/providers/vertex.md), [Vertex AI Video Generation (Veo)](https://docs.litellm.ai/docs/providers/vertex_ai/videos.md), [Vertex AI - Anthropic, DeepSeek, Model Garden, xAI](https://docs.litellm.ai/docs/providers/vertex_partner.md), [Vertex AI - Self Deployed Models](https://docs.litellm.ai/docs/providers/vertex_self_deployed.md), [Vertex AI Embedding](https://docs.litellm.ai/docs/providers/vertex_embedding.md), [Vertex AI Image Generation](https://docs.litellm.ai/docs/providers/vertex_image.md), [Vertex AI Text to Speech](https://docs.litellm.ai/docs/providers/vertex_speech.md), [Vertex AI Audio Transcription](https://docs.litellm.ai/docs/providers/vertex_transcription.md), [Vertex Batch APIs](https://docs.litellm.ai/docs/providers/vertex_batch.md), [Vertex AI OCR](https://docs.litellm.ai/docs/providers/vertex_ocr.md), [Vertex AI Agent Engine](https://docs.litellm.ai/docs/providers/vertex_ai_agent_engine.md), [Vertex AI Gemini Live - Realtime API](https://docs.litellm.ai/docs/providers/vertex_realtime.md), [Azure OpenAI](https://docs.litellm.ai/docs/providers/azure/index.md), [Azure Responses API](https://docs.litellm.ai/docs/providers/azure/azure_responses.md), [Azure OpenAI Embeddings](https://docs.litellm.ai/docs/providers/azure/azure_embedding.md), [Azure Text to Speech (tts)](https://docs.litellm.ai/docs/providers/azure/azure_speech.md), [Azure Video Generation](https://docs.litellm.ai/docs/providers/azure/videos.md), [Azure AI Studio](https://docs.litellm.ai/docs/providers/azure_ai.md), [Azure Model Router](https://docs.litellm.ai/docs/providers/azure_ai/azure_model_router.md), [Azure AI Foundry Agents](https://docs.litellm.ai/docs/providers/azure_ai_agents.md), [Azure AI OCR (Mistral, Cohere Parse)](https://docs.litellm.ai/docs/providers/azure_ocr.md), [Azure Document Intelligence OCR](https://docs.litellm.ai/docs/providers/azure_document_intelligence.md), [Azure AI Speech (Cognitive Services)](https://docs.litellm.ai/docs/providers/azure_ai_speech.md), [Azure AI Image Generation (Black Forest Labs - Flux)](https://docs.litellm.ai/docs/providers/azure_ai_img.md), [Azure AI Search - Vector Store (Unified API)](https://docs.litellm.ai/docs/providers/azure_ai_vector_stores.md), [Azure AI Search - Vector Store (Passthrough API)](https://docs.litellm.ai/docs/providers/azure_ai/azure_ai_vector_stores_passthrough.md), [AWS Bedrock](https://docs.litellm.ai/docs/providers/bedrock.md), [Bedrock Embedding](https://docs.litellm.ai/docs/providers/bedrock_embedding.md), [Bedrock Imported Models](https://docs.litellm.ai/docs/providers/bedrock_imported.md), [AWS Bedrock - Image Generation](https://docs.litellm.ai/docs/providers/bedrock_image_gen.md), [AWS Bedrock - Rerank API](https://docs.litellm.ai/docs/providers/bedrock_rerank.md), [Bedrock AgentCore](https://docs.litellm.ai/docs/providers/bedrock_agentcore.md), [Bedrock Agents](https://docs.litellm.ai/docs/providers/bedrock_agents.md), [Bedrock - Writer Palmyra](https://docs.litellm.ai/docs/providers/bedrock_writer.md), [Bedrock Batches](https://docs.litellm.ai/docs/providers/bedrock_batches.md), [Bedrock Realtime API](https://docs.litellm.ai/docs/providers/bedrock_realtime_with_audio.md), [AWS Polly Text to Speech (tts)](https://docs.litellm.ai/docs/providers/aws_polly.md), [Bedrock Knowledge Bases](https://docs.litellm.ai/docs/providers/bedrock_vector_store.md), [Amazon Bedrock Mantle](https://docs.litellm.ai/docs/providers/bedrock_mantle.md), [Mistral AI API](https://docs.litellm.ai/docs/providers/mistral.md), [Mistral AI Batch API](https://docs.litellm.ai/docs/providers/mistral_batches.md), [xAI](https://docs.litellm.ai/docs/providers/xai.md), [xAI Voice Agent (Realtime API)](https://docs.litellm.ai/docs/providers/xai_realtime.md), [xAI Batch API](https://docs.litellm.ai/docs/providers/xai_batches.md), [Deepseek](https://docs.litellm.ai/docs/providers/deepseek.md), [Groq](https://docs.litellm.ai/docs/providers/groq.md), [Ollama](https://docs.litellm.ai/docs/providers/ollama.md), [VLLM](https://docs.litellm.ai/docs/providers/vllm.md), [vLLM - Batch + Files API](https://docs.litellm.ai/docs/providers/vllm_batches.md), [LM Studio](https://docs.litellm.ai/docs/providers/lm_studio.md), [Llamafile](https://docs.litellm.ai/docs/providers/llamafile.md), [Docker Model Runner](https://docs.litellm.ai/docs/providers/docker_model_runner.md), [Lemonade](https://docs.litellm.ai/docs/providers/lemonade.md), [Triton Inference Server](https://docs.litellm.ai/docs/providers/triton-inference-server.md), [Xinference [Xorbits Inference]](https://docs.litellm.ai/docs/providers/xinference.md), [Infinity](https://docs.litellm.ai/docs/providers/infinity.md), [Nvidia NIM](https://docs.litellm.ai/docs/providers/nvidia_nim.md), [Nvidia NIM - Rerank](https://docs.litellm.ai/docs/providers/nvidia_nim_rerank.md), [Petals](https://docs.litellm.ai/docs/providers/petals.md), [Custom API Server (Custom Format)](https://docs.litellm.ai/docs/providers/custom_llm_server.md), [LiteLLM Proxy (LLM Gateway)](https://docs.litellm.ai/docs/providers/litellm_proxy.md), [OpenRouter](https://docs.litellm.ai/docs/providers/openrouter.md), [OpenCode Go](https://docs.litellm.ai/docs/providers/opencode_go.md), [Vercel AI Gateway](https://docs.litellm.ai/docs/providers/vercel_ai_gateway.md), [Helicone](https://docs.litellm.ai/docs/providers/helicone.md), [CometAPI](https://docs.litellm.ai/docs/providers/cometapi.md), [CLF AI Gateway](https://docs.litellm.ai/docs/providers/clf_ai_gateway.md), [Poe](https://docs.litellm.ai/docs/providers/poe.md), [Milvus - Vector Store](https://docs.litellm.ai/docs/providers/milvus_vector_stores.md), [MongoDB - Vector Store (BETA)](https://docs.litellm.ai/docs/providers/mongodb_vector_stores.md), [AWS S3 Vectors](https://docs.litellm.ai/docs/providers/s3_vectors.md), [Valkey - Vector Store](https://docs.litellm.ai/docs/providers/valkey_vector_stores.md), [RAGFlow Vector Stores](https://docs.litellm.ai/docs/providers/ragflow_vector_store.md), [LangGraph](https://docs.litellm.ai/docs/providers/langgraph.md), [Pydantic AI Agents](https://docs.litellm.ai/docs/providers/pydantic_ai_agent.md), [Abliteration](https://docs.litellm.ai/docs/providers/abliteration.md), [AI21](https://docs.litellm.ai/docs/providers/ai21.md), [AI/ML API](https://docs.litellm.ai/docs/providers/aiml.md), [Aleph Alpha](https://docs.litellm.ai/docs/providers/aleph_alpha.md), [Amazon Nova](https://docs.litellm.ai/docs/providers/amazon_nova.md), [Anyscale](https://docs.litellm.ai/docs/providers/anyscale.md), [Apertis AI (Stima API)](https://docs.litellm.ai/docs/providers/apertis.md), [AWS Sagemaker](https://docs.litellm.ai/docs/providers/aws_sagemaker.md), [Baseten](https://docs.litellm.ai/docs/providers/baseten.md), [Black Forest Labs Image Generation](https://docs.litellm.ai/docs/providers/black_forest_labs.md), [Black Forest Labs Image Editing](https://docs.litellm.ai/docs/providers/black_forest_labs_img_edit.md), [Bytez](https://docs.litellm.ai/docs/providers/bytez.md), [Cerebras](https://docs.litellm.ai/docs/providers/cerebras.md), [ChatGPT Subscription](https://docs.litellm.ai/docs/providers/chatgpt.md), [Chutes](https://docs.litellm.ai/docs/providers/chutes.md), [Clarifai](https://docs.litellm.ai/docs/providers/clarifai.md), [Cloudflare Workers AI](https://docs.litellm.ai/docs/providers/cloudflare_workers.md), [Codestral API [Mistral AI]](https://docs.litellm.ai/docs/providers/codestral.md), [Cognition](https://docs.litellm.ai/docs/providers/cognition.md), [Cohere](https://docs.litellm.ai/docs/providers/cohere.md), [CompactifAI](https://docs.litellm.ai/docs/providers/compactifai.md), [CoralBricks](https://docs.litellm.ai/docs/providers/coralbricks.md), [Crusoe](https://docs.litellm.ai/docs/providers/crusoe.md), [Dashscope API (Qwen models)](https://docs.litellm.ai/docs/providers/dashscope.md), [Databricks](https://docs.litellm.ai/docs/providers/databricks.md), [DataRobot](https://docs.litellm.ai/docs/providers/datarobot.md), [Deepgram](https://docs.litellm.ai/docs/providers/deepgram.md), [DeepInfra](https://docs.litellm.ai/docs/providers/deepinfra.md), [Eden AI](https://docs.litellm.ai/docs/providers/edenai.md), [ElevenLabs](https://docs.litellm.ai/docs/providers/elevenlabs.md), [EmpirioLabs AI](https://docs.litellm.ai/docs/providers/empiriolabs.md), [Fal AI](https://docs.litellm.ai/docs/providers/fal_ai.md), [Featherless AI](https://docs.litellm.ai/docs/providers/featherless_ai.md), [Fireworks AI](https://docs.litellm.ai/docs/providers/fireworks_ai.md), [FriendliAI](https://docs.litellm.ai/docs/providers/friendliai.md), [Galadriel](https://docs.litellm.ai/docs/providers/galadriel.md), [Github](https://docs.litellm.ai/docs/providers/github.md), [GitHub Copilot](https://docs.litellm.ai/docs/providers/github_copilot.md), [GMI Cloud](https://docs.litellm.ai/docs/providers/gmi.md), [GradientAI](https://docs.litellm.ai/docs/providers/gradient_ai.md), [Heroku](https://docs.litellm.ai/docs/providers/heroku.md), [Hugging Face](https://docs.litellm.ai/docs/providers/huggingface.md), [HuggingFace Rerank](https://docs.litellm.ai/docs/providers/huggingface_rerank.md), [Hyperbolic](https://docs.litellm.ai/docs/providers/hyperbolic.md), [Inception](https://docs.litellm.ai/docs/providers/inception.md), [Inference APIs](https://docs.litellm.ai/docs/providers/inferenceapis.md), [Jina AI](https://docs.litellm.ai/docs/providers/jina_ai.md), [Lambda AI](https://docs.litellm.ai/docs/providers/lambda_ai.md), [LlamaGate](https://docs.litellm.ai/docs/providers/llamagate.md), [Manus](https://docs.litellm.ai/docs/providers/manus.md), [Meta Model API](https://docs.litellm.ai/docs/providers/meta.md), [Meta Llama](https://docs.litellm.ai/docs/providers/meta_llama.md), [Microsoft 365 Copilot](https://docs.litellm.ai/docs/providers/microsoft_365_copilot.md), [MiniMax](https://docs.litellm.ai/docs/providers/minimax.md), [Moonshot AI](https://docs.litellm.ai/docs/providers/moonshot.md), [Morph](https://docs.litellm.ai/docs/providers/morph.md), [Nadir](https://docs.litellm.ai/docs/providers/nadir.md), [NanoGPT](https://docs.litellm.ai/docs/providers/nano-gpt.md), [Nebius AI Studio](https://docs.litellm.ai/docs/providers/nebius.md), [Neosantara](https://docs.litellm.ai/docs/providers/neosantara.md), [NLP Cloud](https://docs.litellm.ai/docs/providers/nlp_cloud.md), [Novita AI](https://docs.litellm.ai/docs/providers/novita.md), [Nscale (EU Sovereign)](https://docs.litellm.ai/docs/providers/nscale.md), [Nvidia Riva (Speech-to-Text)](https://docs.litellm.ai/docs/providers/nvidia_riva.md), [Oracle Cloud Infrastructure (OCI)](https://docs.litellm.ai/docs/providers/oci.md), [OpenInfer](https://docs.litellm.ai/docs/providers/openinfer.md), [OVHCloud AI Endpoints](https://docs.litellm.ai/docs/providers/ovhcloud.md), [Perplexity AI (pplx-api)](https://docs.litellm.ai/docs/providers/perplexity.md), [Perplexity Embeddings](https://docs.litellm.ai/docs/providers/perplexity_embedding.md), [Predibase](https://docs.litellm.ai/docs/providers/predibase.md), [Prism](https://docs.litellm.ai/docs/providers/prism.md), [PublicAI](https://docs.litellm.ai/docs/providers/publicai.md), [QwenCloud and Qianwen AI Platform (Qwen models)](https://docs.litellm.ai/docs/providers/qwencloud.md), [RAGFlow](https://docs.litellm.ai/docs/providers/ragflow.md), [Recraft](https://docs.litellm.ai/docs/providers/recraft.md), [Refuseless](https://docs.litellm.ai/docs/providers/refuseless.md), [Reka](https://docs.litellm.ai/docs/providers/reka.md), [Replicate](https://docs.litellm.ai/docs/providers/replicate.md), [RunwayML - Image Generation](https://docs.litellm.ai/docs/providers/runwayml/images.md), [RunwayML - Video Generation](https://docs.litellm.ai/docs/providers/runwayml/videos.md), [SambaNova](https://docs.litellm.ai/docs/providers/sambanova.md), [SAP Generative AI Hub](https://docs.litellm.ai/docs/providers/sap.md), [Sarvam.ai](https://docs.litellm.ai/docs/providers/sarvam.md), [ScaleDown](https://docs.litellm.ai/docs/providers/scaledown.md), [Scaleway](https://docs.litellm.ai/docs/providers/scaleway.md), [SCX.ai](https://docs.litellm.ai/docs/providers/scx_ai.md), [Snowflake Cortex](https://docs.litellm.ai/docs/providers/snowflake.md), [Stability AI](https://docs.litellm.ai/docs/providers/stability.md), [Strands Decider](https://docs.litellm.ai/docs/providers/strands_decider.md), [Synthetic](https://docs.litellm.ai/docs/providers/synthetic.md), [Tencent TokenHub](https://docs.litellm.ai/docs/providers/tencent.md), [Tensormesh](https://docs.litellm.ai/docs/providers/tensormesh.md), [Together AI](https://docs.litellm.ai/docs/providers/togetherai.md), [Topaz](https://docs.litellm.ai/docs/providers/topaz.md), [v0](https://docs.litellm.ai/docs/providers/v0.md), [Volcano Engine (Volcengine)](https://docs.litellm.ai/docs/providers/volcano.md), [VoyageAI by MongoDB](https://docs.litellm.ai/docs/providers/voyage.md), [Weights & Biases Inference](https://docs.litellm.ai/docs/providers/wandb_inference.md), [IBM watsonx.ai](https://docs.litellm.ai/docs/providers/watsonx/index.md), [WatsonX Audio Transcription](https://docs.litellm.ai/docs/providers/watsonx/audio_transcription.md), [Xiaomi MiMo](https://docs.litellm.ai/docs/providers/xiaomi_mimo.md), [Z.AI (Zhipu AI)](https://docs.litellm.ai/docs/providers/zai.md), [Calling Finetuned Models](https://docs.litellm.ai/docs/guides/finetuned_models.md), [Integrate as a Model Provider](https://docs.litellm.ai/docs/provider_registration/index.md), [Add Model Pricing & Context Window](https://docs.litellm.ai/docs/provider_registration/add_model_pricing.md)