An LLM gateway is a single API endpoint that sits in front of many large-language-model providers. Instead of integrating Anthropic, Google and OpenAI separately — three SDKs, three keys, three invoices — your app sends one request shape to one base URL with one key, and the gateway routes it, retries on failure, meters usage and enforces spend limits. It is the control plane between your product and the models.
What an LLM gateway does
A production LLM gateway carries six responsibilities. The first is what makes the other five possible: a single, stable API surface.
| Capability | What it gives you |
|---|---|
| OpenAI-compatible API | One request/response shape — usually /v1/chat/completions — for every model, so any OpenAI SDK works by changing only base_url. |
| Model routing | Pick the provider/model per request; swap models by changing a string, not rewriting client code. |
| Fallback & failover | Reroute to a healthy upstream when a provider errors, rate-limits or degrades — resilience your app doesn't have to code. |
| Caching | Reuse stable prompts (e.g. Anthropic prompt caching) to cut repeat input cost, with an affinity router that keeps caches warm. |
| Billing & spend control | One wallet across providers, per-key budgets and rate limits so a leaked key or runaway loop can't drain the account. |
| Observability | Per-call logging of tokens, latency and cost — one dashboard instead of three provider consoles. |
One endpoint, every provider
The gateway turns “add a provider” into a one-string change. The same client reaches Claude, Gemini and GPT — you route by setting the model field:
from openai import OpenAI
# One LLM gateway endpoint + one key fronts every provider.
gateway = OpenAI(
api_key="sk-kunavo-...",
base_url="https://api.kunavo.com/v1", # the LLM gateway
)
# Route by changing a string — no per-provider SDK, key or client.
def ask(model: str, prompt: str) -> str:
r = gateway.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
)
return r.choices[0].message.content
print(ask("claude-sonnet-4-6", "Draft a release note.")) # Anthropic
print(ask("gemini-2-5-flash", "Classify this ticket.")) # Google
print(ask("gpt-5-4", "Summarize the thread.")) # OpenAILLM gateway vs API gateway
A traditional API gateway — Kong, AWS API Gateway, NGINX — routes generic HTTP and handles auth, quotas and rate limiting. An LLM gateway is purpose-built for model traffic: it understands tokens and streaming, does prompt caching and model routing, meters spend per key, and (for inference gateways) bills usage. They compose — some teams put an LLM gateway behind their existing API gateway. For the conceptual deep dive, see what is an AI gateway?, and Kunavo vs Kong AI Gateway for the enterprise API-management case.
Two kinds of LLM gateway
The category splits by who holds the provider relationship:
| Type | You bring | Examples | Best for |
|---|---|---|---|
| BYO-key gateway | Your own provider keys + billing | Portkey, Helicone, LiteLLM (open source) | Routing, guardrails and observability over accounts you hold |
| Inference gateway | Nothing — the gateway resells access | Kunavo, OpenRouter | One wallet and one key across providers, often below list price |
BYO-key gateways layer control and observability on top of your existing Anthropic, Google and OpenAI accounts — you still hold those keys. Inference gateways front the providers themselves: top up one balance, never touch a provider console. See Kunavo vs Portkey, Helicone, LiteLLM and OpenRouter for the full side-by-sides.
Security & compliance
A gateway is also the natural place to enforce security: keys never leave your server, per-key spend caps and rate limits contain blast radius, and centralized logging gives you one audit trail. For regulated deployments, Kunavo offers a DPA and Zero Data Retention routing through the providers' enterprise tiers — the region-by-region playbook is in the AI compliance guide.
Kunavo as an LLM gateway
Kunavo is an inference LLM gateway. It is OpenAI-wire-compatible at https://api.kunavo.com/v1, so if your code can call the OpenAI API it can call Kunavo by changing only the base URL. One sk-kunavo- bearer token reaches Claude, Gemini, GPT and 200+ models — priced roughly 60% under the providers' list rates, pay-as-you-go from a $5 top-up, one Stripe-billed wallet. Unlike a text-only router it also fronts image, video and audio generation on the same key.
Start with the quickstart, see the OpenAI-compatible API guide for how the wire format lets you move between local (Ollama) and hosted models without a code change, and the LLM cost calculator to estimate what a workload will cost.
FAQ
What is an LLM gateway?
A single API endpoint in front of many model providers. Your app uses one base URL and one key; the gateway handles routing, failover, caching, spend limits, usage logging and billing — so you integrate once instead of wiring each provider separately.
What's the difference between an LLM gateway and an API gateway?
An API gateway routes generic HTTP with auth and rate limits. An LLM gateway is model-aware: it speaks the OpenAI chat format, understands tokens and streaming, and adds prompt caching, model routing, per-key spend caps and usage metering.
What is the best LLM gateway?
Depends on whether you bring your own provider keys (Portkey, Helicone, LiteLLM) or want the gateway to resell access on one wallet (Kunavo, OpenRouter). Kunavo is an OpenAI-compatible inference gateway across text, image, video and audio at about 60% under list.
Is there an open-source LLM gateway?
Yes — LiteLLM is the common open-source, self-hosted choice; you run it with your own keys. Kunavo is hosted with nothing to run and one balance across every model.