AI Gateways in 2026: Six Options Compared After LiteLLM's 40-Minute Backdoor

On 24 March 2026, for about 40 minutes, pip install litellm gave you a credential stealer. The package that sits between your applications and every model provider, holding every provider key you own, was the one that got backdoored. I have run some kind of LLM gateway in front of model APIs for a while now, and that week changed how I think about them: an AI gateway is not a convenience library, it is a privileged piece of infrastructure, and it deserves the same patching, pinning and isolation as your ingress controller or your secrets manager. This post explains what an AI gateway (or LLM gateway) actually does, compares the six options people search for most (LiteLLM, Portkey, Kong AI Gateway, Cloudflare AI Gateway, Envoy AI Gateway and OpenRouter), walks through the 2026 supply-chain and CVE record, and ends with how I would decide between building, self-hosting and buying.

What is an AI gateway, and how is it different from an API gateway?

An AI gateway is a reverse proxy that speaks the model providers' APIs. Applications send an OpenAI-style (or Anthropic-style) request to one endpoint; the gateway decides which provider and model serve it, attaches the real provider credential, and records what happened. "LLM gateway" and "AI gateway" mean the same thing in practice; vendors that now also proxy MCP tool calls and agent-to-agent traffic have started saying "agent gateway".

A classic API gateway routes by path and counts requests. An AI gateway has to understand the body: count tokens rather than requests (one request can cost a fraction of a cent or several dollars), know model prices, stream server-sent events without buffering, and map provider-specific failures, such as a context-window error that should go to a bigger model rather than be retried.

The feature list has converged across vendors:

• Unified API and routing. One schema for many providers, load-balanced across deployments.
• Retries and fallbacks. Retry a flaky deployment, then fall back to another model or provider.
• Virtual keys, budgets and rate limits. Each team gets its own key with a spend cap and token limit; real provider keys stay in the gateway.
• Caching. Exact-match, and sometimes semantic caching on embedding similarity.
• Guardrails. PII redaction, prompt-injection classifiers and content filters.
• Logging and cost attribution. Per-request traces to Langfuse, OpenTelemetry or a dashboard, tagged by team and key.

If that list sounds like FinOps and security controls wearing a proxy costume, that is because it is. The FinOps for AI problem (who spent what on which model) is mostly solved or unsolved at this layer.

The six AI gateways people actually compare

The market splits into three shapes: open-source proxies you run yourself (LiteLLM, Portkey's open-source gateway, Envoy AI Gateway, Kong), an edge service you configure (Cloudflare), and a hosted marketplace that resells inference (OpenRouter). Here is where each stood in October 2026, from the sources I could check.

Gateway	Shape	License / pricing	2026 status	Source
LiteLLM	Self-hosted Python proxy plus SDK	Open-source core, enterprise features licensed separately	v1.106.0 released 18 Oct 2026; about 92M PyPI downloads in the last month	releases, pypistats
Portkey	Self-hosted or SaaS	MIT on GitHub; production gateway merged into the open-source repo on 24 Mar 2026	Acquired by Palo Alto Networks, closed 29 May 2026	announcement, press release
Kong AI Gateway	Kong data plane plus AI plugins	Apache-2.0 Kong core; AI Gateway runtime has its own release line	Runtime 2.2.0 on 30 Sep 2026; 2.0.0 removed ai-proxy in favour of ai-proxy-advanced	changelog
Cloudflare AI Gateway	Managed edge service	Core features free on all plans; 5% fee on Unified Billing credit purchases	Dynamic routing with budget and rate-limit nodes	pricing, dynamic routing
Envoy AI Gateway (now Agent Router)	Kubernetes, built on Envoy Gateway	Apache-2.0	1.0 on 23 Jun 2026; renamed Agent Router and moved to the Agentic AI Foundation on 10 Sep 2026; v1.2.0 on 7 Oct	README, blog
OpenRouter	Hosted marketplace	5.5% fee on card credit purchases; BYOK free up to $25,000 a month, then 5%	Series B of $113M at about $1.3B (May 2026)	FAQ, Dealroom

A few notes the table cannot hold.

LiteLLM is the default most teams reach for: a YAML file away from working, with virtual keys, team budgets and spend tracking in the open-source proxy. The flip side is a very large Python codebase with an admin UI, SSO, MCP gateway and guardrail sandbox in one process, and that surface area shows up in the advisory list below.

Portkey merged its internal production gateway into the open-source repo in March, saying it processes 2 trillion tokens a day. Two months later Palo Alto Networks closed the acquisition, making Portkey the core gateway for Prisma AIRS. Self-hosters of the MIT code are unaffected for now; SaaS buyers now have a security vendor's roadmap.

Kong makes sense when you already run Kong for APIs. The AI MCP Proxy plugin gained OAuth-claim ACLs in 3.14.0.0 (April 2026), and the AI Gateway runtime added A2A v1.0 support in 2.0.0. The AI Gateway runtime has its own changelog and version line, separate from the open-source Kong 3.9.x releases on GitHub, so check which AI plugins the edition you run actually includes.

Cloudflare is the cheapest way to get logging, caching and rate limits in front of providers if your traffic already crosses Cloudflare. Under its legacy log storage, the free Workers plan is capped at 100,000 logs total across gateways (accounts whose first gateway was created on or after 24 September 2026 use Workers Logs pricing instead), which is a development budget, not a production one.

Envoy AI Gateway is the Kubernetes-native answer: CRDs such as AIGatewayRoute, and a two-tier design where the edge tier handles auth and global limits while a second tier fronts self-hosted inference. The rename to Agent Router changed no API groups, images or Helm charts, per its README.

OpenRouter is not a gateway you run; it is one API key for hundreds of models, with fallbacks via a models array, billed at whichever model served the request. Governance of your own teams and keys stays with you.

The LiteLLM PyPI backdoor: what happened in March 2026

The chain started with someone else's tool. Aqua Security's advisory for CVE-2026-33634 describes an attacker using compromised credentials to publish a malicious Trivy v0.69.4 release and force-push 76 of 77 version tags in aquasecurity/trivy-action on 19 March 2026. Any CI job that referenced the action by tag rather than commit SHA ran attacker code with whatever secrets the job could see. CISA added it to the Known Exploited Vulnerabilities catalogue on 26 March.

LiteLLM ran Trivy in its CI. Per the maintainers' incident post, malicious litellm==1.82.7 and litellm==1.82.8 were live on PyPI from 10:39 UTC on 24 March for about 40 minutes before PyPI quarantined them. They believe the compromise came through the Trivy dependency, and that the attacker bypassed the official workflows and uploaded directly to PyPI. Version 1.82.7 hid the payload in litellm/proxy/proxy_server.py; 1.82.8 added a litellm_init.pth file, which Python executes on every interpreter start whether or not LiteLLM is imported. Security firms attributed it to the group called TeamPCP, and Endor Labs' analysis as reported describes a payload that hunted SSH keys, cloud credentials, Kubernetes secrets and CI tokens.

Two details matter. The official Docker image was not affected, because it pins dependencies; the exposure was unpinned pip install, including LiteLLM pulled in transitively by other tools. And the response was sound: credentials rotated, Mandiant engaged, releases paused, and a clean v1.83.0 shipped on 30 March from a rebuilt pipeline, with cosign-signed images and a cosign verify command in every release note since.

If you want the broader pattern, the npm worm campaigns of the past year followed the same path: steal a CI token, publish from outside CI, target developer and agent machines that hold credentials.

LiteLLM CVEs in 2026: three on CISA's KEV list

The backdoor was not the only problem. LiteLLM's GitHub security page lists 17 advisories published in 2026 as of today: 4 critical, 6 high, 5 medium and 2 low. Three of them are on CISA's KEV catalogue, meaning exploitation was observed in the wild.

The pattern in the list is more useful than the count:

• CVE-2026-42208, critical: SQL injection in proxy API key verification, affecting 1.81.16 up to 1.83.7. The function whose job is to check keys was the hole.
• CVE-2026-42271, high: command execution through the MCP stdio test endpoints, fixed in 1.83.7. The v1.106.0 release notes include "disable stdio MCP servers by default", which closes that class of feature for new installs.
• CVE-2026-49468, critical: authentication bypass via Host header injection, fixed in 1.84.0.
• GHSA-7hp6-4w63-5g45, critical, published 30 September: one key sealed secrets at rest and minted session tokens, so an internal user could forge an admin token and reach the MCP stdio endpoint. Patched in 1.100.4, 1.101.3, 1.102.2 and 1.103.1.

None of this makes LiteLLM uniquely bad: it is popular, fast-moving, and publishes advisories with version ranges and credits. It does mean that "we run LiteLLM" has to come with "and we upgrade it like we upgrade the Kubernetes control plane", plus network rules so the admin UI and management routes are not on the public internet. The same thinking applies to MCP gateways, which now ship inside most AI gateways.

How to pin and verify an AI gateway

The concrete lesson from March is boring: pin by digest, verify signatures, and keep the gateway's own credentials short-lived. Here is what that looks like for LiteLLM, using the commands and key from the v1.106.0 release notes; the image digest is the one I resolved on 20 October 2026.

# 1. Verify the image signature against the key pinned to a commit (cosign v2+)
cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:v1.106.0

# 2. Resolve the tag to an immutable digest and deploy the digest, not the tag
docker buildx imagetools inspect ghcr.io/berriai/litellm:v1.106.0 | grep '^Digest'
# Digest:    sha256:2e4c0371f3a8c28e4da131d77abaa8341cd3b0c2fcc4d34d7c5900431af7b9d0

# 3. If you install from PyPI, pin exactly and require hashes
#    requirements.txt:  litellm[proxy]==1.106.0 --hash=sha256:<from pypi.org>
pip install --require-hashes -r requirements.txt

# 4. Check any host that ever ran an unpinned install in March 2026
find / -name 'litellm_init.pth' 2>/dev/null
pip show litellm 2>/dev/null | grep -E '^Version: 1\.82\.(7|8)$' && echo "ROTATE EVERY CREDENTIAL"

On the CI side, pin GitHub Actions to full commit SHAs; the Trivy tag rewrite only worked because consumers trusted mutable tags. The companion piece on GitHub Actions supply-chain security covers that in depth, and the step-by-step LiteLLM proxy setup guide uses a pinned image throughout.

Build vs buy: do you need an AI gateway at all?

My rule of thumb is that you need a gateway when you have more than one of: several teams sharing provider accounts, more than one provider, a finance person asking which feature costs what, or a compliance requirement to log prompts somewhere you control. A single service calling one provider does not need one; the provider SDK's retry logic and a dashboard are enough.

When you do need one, the choice is mostly about where it runs and who patches it:

• Self-host LiteLLM or Portkey OSS if you want budgets and virtual keys without a vendor, and you have someone who will read advisories weekly and upgrade monthly. Run it in a private network, behind your own ingress, with Postgres and Redis you already know how to operate.
• Use Envoy AI Gateway (Agent Router) or Kong if you already have Envoy Gateway or Kong in production. Reusing a data plane you understand beats adding a new Python service, especially for latency and for self-hosted inference behind it.
• Use Cloudflare for low-effort logging, caching and rate limiting at the edge.
• Use OpenRouter when the problem is model breadth and procurement; it also works as one more provider behind your own gateway.
• Build your own only when the gateway is your product. The hard parts are not the proxying but a current price table for hundreds of models, streaming token accounting and provider error mapping.

The trade-off missing from feature tables is blast radius. A gateway holds every provider key and sees every prompt, so I treat its upgrade cadence and network exposure as a security decision, not an ops chore.

FAQ

What is the difference between an AI gateway and an LLM gateway?

In practice none. Both terms describe a proxy that unifies model provider APIs and adds routing, budgets, caching and logging. "AI gateway" is the broader marketing term and increasingly includes MCP tool traffic and agent-to-agent calls.

Is LiteLLM safe to use after the 2026 supply-chain attack?

The malicious versions were 1.82.7 and 1.82.8 on PyPI only, and the official Docker image was not affected. Releases since v1.83.0 come from a rebuilt pipeline with cosign-signed images. It is safe in the sense any well-maintained project is: pin a current version (1.106.0 as of this writing), verify the signature, and keep up with the advisory feed.

Which AI gateway is best for Kubernetes?

If you already run Envoy Gateway, Envoy AI Gateway (now Agent Router) fits natively with CRDs and a two-tier design for self-hosted models. LiteLLM and Portkey both ship Helm charts and run fine on Kubernetes too; LiteLLM's chart is published at version 1.106.0 alongside the release.

Does an AI gateway add latency?

Some, usually small next to model latency measured in hundreds of milliseconds or seconds. Vendors publish overhead claims, for example Bifrost's repository claims under 200 microseconds at 5,000 requests per second, but measure it on your own traffic, including streaming time to first token, before trusting any number.

This quarter I would do three things. Inventory every place a gateway or gateway library runs, including developer laptops and CI images where LiteLLM arrives as a transitive dependency. Move any self-hosted gateway to a digest-pinned, signature-verified image on a current release, with the admin UI off the public internet. And put a monthly upgrade on the calendar, because the advisory list above says the next critical one is a question of when, not if.

Related: MCP in 2026: 475M SDK Downloads a Month, 39,492 Servers in a Registry Still in Preview, and 12 Gateways Selling the Same 3 Features, FinOps for AI in 2026: 98% Manage Token Spend, 3 in 4 Can't Prove the Value, GPUs Run at 5%, and FOCUS Gets a Token Column in 1.5 and The Worms Learned to Use Your AI Agent: A Year of npm Supply-Chain Attacks.