How to Set Up LiteLLM Proxy: Budgets, Fallbacks and Virtual Keys (2026)

This guide is for platform and backend engineers who want one OpenAI-compatible endpoint in front of their model providers, with per-team keys, spend caps and automatic fallbacks. At the end you will have LiteLLM proxy v1.106.0 (released 18 October 2026) running with Postgres, a config.yaml with load balancing and fallbacks, virtual keys with budgets and rate limits, traces going to Langfuse or OpenTelemetry, and the same setup ready for Kubernetes via the official Helm chart. I ran the Docker steps, the key and budget API calls and the OTel export against the v1.106.0 image on 20 October 2026, and the outputs shown are from that run. You need Docker with Compose v2, curl, and at least one provider API key; Helm 3 and kubectl for the Kubernetes step. If you are still deciding whether LiteLLM is the right gateway at all, read the companion piece on AI gateways in 2026 first.

Step 1: Pick a LiteLLM version and verify the image

Do not run latest or main-stable in production, and do not pip install litellm unpinned. In March 2026 the PyPI releases 1.82.7 and 1.82.8 were backdoored for about 40 minutes, and the project's security advisories list 17 issues published in 2026, three of which are on CISA's Known Exploited Vulnerabilities list (CVE-2026-42208, CVE-2026-42271, CVE-2026-59822). The newest critical one, GHSA-7hp6-4w63-5g45, is fixed in 1.103.1 and later. This guide pins v1.106.0, the current stable release.

Images have been cosign-signed since v1.83.0. The verification command and key come straight from the v1.106.0 release notes:

# cosign v2.x; the key URL is pinned to an immutable commit
cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:v1.106.0

# Resolve the tag to a digest so a re-pushed tag can never change what you run
docker buildx imagetools inspect ghcr.io/berriai/litellm:v1.106.0 | grep '^Digest'

Expected output of the second command on 20 October 2026:

Digest:    sha256:2e4c0371f3a8c28e4da131d77abaa8341cd3b0c2fcc4d34d7c5900431af7b9d0

The default image runs as root (id -u returns 0 inside it). A non-root variant is published as ghcr.io/berriai/litellm-non_root:v1.106.0 if your cluster policy requires it.

Step 2: Run LiteLLM and Postgres with Docker Compose

Postgres is what makes virtual keys, teams, budgets and spend logs work; without a database budgets are not enforced at all. Create a working directory and generate the secrets:

mkdir litellm-gw && cd litellm-gw
cat > .env <<EOF
POSTGRES_PASSWORD=$(openssl rand -hex 16)
LITELLM_MASTER_KEY=sk-$(openssl rand -hex 24)
LITELLM_SALT_KEY=sk-$(openssl rand -hex 24)
OPENAI_API_KEY=replace-me
ANTHROPIC_API_KEY=replace-me
EOF
chmod 600 .env

LITELLM_MASTER_KEY is the admin key. LITELLM_SALT_KEY encrypts provider credentials stored in the database; the deploy docs say to set it once and never change it, because changing it makes stored credentials unreadable. Back it up with the same care as the database.

Now compose.yaml, adapted from the repository's own docker-compose.yml with the image pinned and the port bound to loopback:

services:
  db:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set in .env}
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d litellm -U litellm"]
      interval: 2s
      timeout: 5s
      retries: 15

  litellm:
    # v1.106.0, pinned by index digest (resolved 2026-10-20)
    image: ghcr.io/berriai/litellm:v1.106.0@sha256:2e4c0371f3a8c28e4da131d77abaa8341cd3b0c2fcc4d34d7c5900431af7b9d0
    restart: unless-stopped
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "127.0.0.1:4000:4000"      # loopback only; put real ingress/TLS in front
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    env_file: .env
    environment:
      DATABASE_URL: postgresql://litellm:${POSTGRES_PASSWORD}@db:5432/litellm
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')\""]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

volumes:
  pgdata:

Write the config in Step 3 before starting, then run docker compose up -d and check readiness:

curl -s http://localhost:4000/health/readiness
# {"status":"healthy","db":"connected"}

Step 3: Write config.yaml with model_list and load balancing

Every entry in model_list maps a public model_name (what clients send) to a provider model in litellm_params. Two entries with the same model_name become one load-balanced group. os.environ/NAME reads a secret from the environment, so no keys live in the file.

model_list:
  # Two deployments of one public name = a load-balanced group
  - model_name: chat-default
    litellm_params:
      model: openai/gpt-5-mini
      api_key: os.environ/OPENAI_API_KEY
      rpm: 500                      # deployment-level hint for routing
  - model_name: chat-default
    litellm_params:
      model: azure/gpt-5-mini       # second deployment, e.g. Azure OpenAI
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: os.environ/AZURE_API_VERSION
      rpm: 500

  # A different provider, used below as the fallback group
  - model_name: chat-fallback
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Drop the Azure entry if you only have one provider; a group with a single deployment is fine. Model identifiers change often, so check your provider's current model list rather than copying mine blindly. Restart with docker compose restart litellm after any config change; startup logs list the loaded groups:

LiteLLM: Proxy initialized with Config, Set models:
    chat-default
    chat-fallback

Step 4: Add fallbacks and retries

Retries happen inside a model group; fallbacks move the request to a different group once retries are exhausted. The reliability docs put fallbacks under router_settings and retry, timeout and cooldown knobs under litellm_settings:

router_settings:
  enable_pre_call_checks: true           # needed for context-window checks
  fallbacks:
    - chat-default: ["chat-fallback"]
  context_window_fallbacks:
    - chat-default: ["chat-fallback"]

litellm_settings:
  num_retries: 2            # per group, before falling back
  request_timeout: 30       # seconds
  allowed_fails: 3          # failures per minute before a deployment cools down
  cooldown_time: 30         # seconds a failing deployment is skipped

To test it, do not rely on mock_testing_fallbacks: the docs say the mock_testing_* flags have been stripped from proxy requests since v1.85.0. Instead, in a throwaway config, point the primary at an address that cannot answer and send a request. My test config had two groups, primary (with api_base: http://10.255.255.1:9/v1 and timeout: 3) falling back to backup. The response headers tell you what happened:

curl -s -D - -o /dev/null http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"primary","messages":[{"role":"user","content":"hi"}]}' \
  | grep -iE '^HTTP|x-litellm-(model-group|attempted)'

HTTP/1.1 200 OK
x-litellm-model-group: backup
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 1

The request was asked for primary and served by backup. Alert on x-litellm-attempted-fallbacks in your logs; a silent fallback to a pricier model is a cost incident waiting to happen.

Step 5: Create virtual keys, team budgets and rate limits

Teams hold a shared budget and limits; keys belong to teams. All of this is plain HTTP against the admin API with the master key. budget_duration accepts values such as 30s, 30m, 24h, 30d and 1mo, per the budgets docs.

export LITELLM=http://localhost:4000
export MASTER="Authorization: Bearer $(grep LITELLM_MASTER_KEY .env | cut -d= -f2)"

# A team with a $50 monthly budget and team-wide limits
curl -s $LITELLM/team/new -H "$MASTER" -H 'Content-Type: application/json' -d '{
  "team_alias": "search-team",
  "max_budget": 50,
  "budget_duration": "30d",
  "tpm_limit": 200000,
  "rpm_limit": 100,
  "models": ["chat-default", "chat-fallback"]
}'

# A key for one service in that team, with its own daily cap and RPM limit
curl -s $LITELLM/key/generate -H "$MASTER" -H 'Content-Type: application/json' -d '{
  "team_id": "<team_id from the previous response>",
  "key_alias": "search-api-prod",
  "max_budget": 5,
  "budget_duration": "1d",
  "rpm_limit": 60,
  "models": ["chat-default"],
  "metadata": {"owner": "search"}
}'

The key response contains "key": "sk-..."; that string is what the service uses as its OpenAI API key, with base_url set to the proxy. In my test, with deliberately tiny limits, the three enforcement paths returned:

# budget exceeded (max_budget 0.000001); messages trimmed
HTTP 422  {"error":{"message":"Budget has been exceeded! Key=search-api-prod (sk-...995Q) Current cost: 1.32e-05, Max budget: 1e-06","type":"budget_exceeded","code":"422"}}

# rpm_limit 2, third request in the same minute
HTTP 429  {"error":{"message":"Rate limit exceeded for api_key: ... Limit type: requests. Current limit: 2, Remaining: 0. ...","type":"throttling_error","code":"429"}}

# model not in the key's models list
HTTP 403  {"error":{"message":"The requested model 'primary' is not available for this API key, ...","type":"key_model_access_denied","code":"403"}}

Two things I noticed. The docs' examples show 400 or 401 for budget errors, but v1.106.0 returned 422, so match on error.type, not the status code. And a key restricted to ["primary"] was still served by the backup group through a fallback, so do not treat a key's models list as a hard boundary if fallbacks point outside it.

Keys with a team_id are charged against the team budget and not the owner's personal budget, per the docs. Check spend with curl -s "$LITELLM/key/info?key=sk-..." -H "$MASTER"; spend is written asynchronously, so it can lag a request by a few seconds.

Step 6: Share limits across replicas with Redis

Rate limits are per process unless the instances share state. The docs state that limits are enforced across instances through Redis, and that if Redis is down each instance falls back to local enforcement (or returns 503 with fail_closed_rate_limit_enforcement: true). As soon as you run more than one replica, add Redis:

router_settings:
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD

Step 7: Send traces to Langfuse and OpenTelemetry

The v1.106.0 image already bundles langfuse 4.15.2 and the OpenTelemetry SDK 1.33.1, so no custom image is needed. From the logging docs:

litellm_settings:
  success_callback: ["langfuse"]   # needs LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST
  callbacks: ["otel"]              # needs OTEL_EXPORTER, OTEL_ENDPOINT, OTEL_HEADERS
  turn_off_message_logging: true   # keep spend and metadata, drop prompt and response text

# .env additions
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com
OTEL_EXPORTER=otlp_http
OTEL_ENDPOINT=http://otel-collector:4318/v1/traces
OTEL_SERVICE_NAME=litellm-gw

With OTEL_EXPORTER=console you can see the spans in docker compose logs litellm. One request produced spans named Received Proxy Server Request, auth, proxy_pre_call, router, litellm_request, raw_gen_ai_request and postgres, with attributes such as gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Send them to a collector rather than straight to a vendor; the OpenTelemetry Collector on Kubernetes guide covers that side.

Step 8: Deploy LiteLLM on Kubernetes with Helm

The monolithic chart is published at the same version as the app (ghcr.io/berriai/litellm-helm:1.106.0). Create secrets first, then point the chart at an existing Postgres:

kubectl create namespace litellm
kubectl -n litellm create secret generic litellm-master \
  --from-literal=masterkey="sk-$(openssl rand -hex 24)"
kubectl -n litellm create secret generic litellm-env \
  --from-literal=LITELLM_SALT_KEY="sk-$(openssl rand -hex 24)" \
  --from-literal=OPENAI_API_KEY="$OPENAI_API_KEY"
kubectl -n litellm create secret generic litellm-postgres \
  --from-literal=username=litellm --from-literal=password="$PG_PASSWORD"

# values.yaml for oci://ghcr.io/berriai/litellm-helm --version 1.106.0
replicaCount: 2
image:
  repository: ghcr.io/berriai/litellm
  tag: "v1.106.0@sha256:2e4c0371f3a8c28e4da131d77abaa8341cd3b0c2fcc4d34d7c5900431af7b9d0"
  pullPolicy: IfNotPresent
masterkeySecretName: litellm-master
masterkeySecretKey: masterkey
environmentSecrets:
  - litellm-env
db:
  useExisting: true
  deployStandalone: false
  endpoint: postgres.data.svc.cluster.local
  database: litellm
  secret:
    name: litellm-postgres
    usernameKey: username
    passwordKey: password
proxy_config:
  model_list:
    - model_name: chat-default
      litellm_params:
        model: openai/gpt-5-mini
        api_key: os.environ/OPENAI_API_KEY
  general_settings:
    master_key: os.environ/PROXY_MASTER_KEY   # the chart injects PROXY_MASTER_KEY

helm install litellm oci://ghcr.io/berriai/litellm-helm --version 1.106.0 \
  -n litellm -f values.yaml
kubectl -n litellm rollout status deploy/litellm

I rendered these values with helm template against chart 1.106.0: they produce a Deployment using the digest-pinned image, a migrations Job, and DATABASE_URL assembled from the secret. Note the chart reads the master key as PROXY_MASTER_KEY, not LITELLM_MASTER_KEY. Keep the Service as ClusterIP and expose it only through an internal ingress; restrict it with network policies.

Troubleshooting

Symptom	Likely cause	Fix
401 ... Malformed API Key passed in. Ensure Key has Bearer prefix	Empty or unquoted key variable in the header	Echo the variable; send Authorization: Bearer sk-...
422 budget_exceeded	Key, team or user max_budget reached	Raise the budget, wait for budget_duration reset, or move the key to another team
429 throttling_error	rpm_limit or tpm_limit hit on key or team	Raise limits, spread across keys, or add client backoff
403 key_model_access_denied	Model not in the key's or team's models list	Add the model_name (not the provider model ID) to models
Limits look per pod, not global	No shared Redis between replicas	Set redis_host, redis_port, redis_password in router_settings
Budgets never enforced	Proxy started without DATABASE_URL	Set database_url; check readiness shows "db":"connected"
Stored provider credentials suddenly fail to decrypt	LITELLM_SALT_KEY changed after models were added	Restore the original salt key from backup
Requests silently cost more	Fallbacks to a pricier group	Alert on x-litellm-attempted-fallbacks; review cooldown settings
Admin calls fail on a Helm install despite a correct secret	Config uses LITELLM_MASTER_KEY but chart injects PROXY_MASTER_KEY	Use os.environ/PROXY_MASTER_KEY in proxy_config
mock_testing_fallbacks does nothing	Flag stripped from proxy requests since v1.85.0	Trigger a real failure in a non-production config

FAQ

Do I need Postgres for LiteLLM proxy?

Only if you want virtual keys, teams, budgets or spend tracking, which is the main reason to run the proxy. Without a database the proxy still routes requests and applies fallbacks, but budgets are not enforced.

Which LiteLLM version should I run in 2026?

A current stable release that is newer than every patched version in the advisory list, verified with cosign and pinned by digest. As of 20 October 2026 that is v1.106.0. Never run 1.82.7 or 1.82.8; if any machine ever installed them, rotate every credential it could reach.

What is the difference between router_settings and litellm_settings?

router_settings configures the routing layer: fallbacks, routing strategy and the Redis connection for shared state. litellm_settings configures library-wide behaviour such as retries, timeouts, cooldowns, callbacks and message logging. The docs show fallbacks in both places, so pick one and stay consistent.

Can LiteLLM enforce a budget per end user of my app?

Yes, with customer budgets. The docs describe creating a budget with /budget/new and pointing max_end_user_budget_id at it, so it applies to any end user without their own budget. Customer budgets are global per deployment and the budget object is cached for 60 seconds, so a burst can overshoot slightly.

When I set up a gateway like this, the first thing I check is not routing, it is exposure: the admin UI and management routes reachable only from a private network, the image pinned to a digest on a release newer than the last critical advisory, and the salt key backed up. Then I set team budgets low and raise them on request, because the first surprise bill always comes from a fallback nobody noticed. The official references are the LiteLLM proxy docs, the reliability page, the budgets and rate limits page, the logging page and the GitHub releases. For the CI side of the same supply-chain story, see GitHub Actions security in 2026.

Related: AI Gateways in 2026: Six Options Compared After LiteLLM's 40-Minute Backdoor, FinOps for AI in 2026: 98% Manage Token Spend, 3 in 4 Can't Prove the Value, GPUs Run at 5%, and FOCUS Gets a Token Column in 1.5 and Agent Observability in 2026: The Spans Are Standard, the Standard Isn't Stable, and Nobody Reads the Logs.