LiteLLM, Portkey, and cloud AI gateways
Words made simple
A few useful meanings before you read the explanation.
- AI
- Software techniques for tasks such as recognizing patterns, understanding language or generating content.
- API
- A way for one piece of software to ask another for data or an action. A claims API might return the current status of a claim.
- Schema
- A description of the fields and value types that data should contain. It helps check that an answer has the expected shape.
More words from this lesson
- Streaming
- Sending an answer in small pieces as it is produced, rather than waiting for the complete response.
- Latency
- How long a request takes to return a result. Lower latency usually means a shorter wait for the user.
- HTTP
- The request-and-response protocol used on the web. HTTPS adds encryption to protect information in transit.
- Multimodal
- Working with more than one kind of input or output, such as text, images or audio.
Overview
- LiteLLM provides model-access abstraction and proxy/gateway capabilities.
- Portkey provides an AI gateway and operational controls.
- Cloud gateways can apply access and model traffic policies within an existing cloud API-management environment; exact AI features vary by product and edition.
Why this matters
Existing tools can provide central controls without building the entire gateway yourself.
An everyday example
Choose a managed travel desk, operate your own desk, or extend the one already in your company.
A technical example
Proxy multiple model providers behind an internal endpoint while preserving capability differences in your adapters.
How it works
- Configure routes/credentials
- Define budgets and allowlists
- Integrate client
- Test quotas/failure/streaming
- Monitor and audit.
Architecture and flow
Internal model contract → chosen gateway → approved route → provider.
The insurance assistant example
A Microsoft-centered insurer may evaluate its existing API-management stack; another team may use a model-specific proxy.
When to use it
When the chosen product meets concrete policy and operating requirements.
When to avoid it
Do not assume an OpenAI-compatible endpoint means identical schema, tool, reasoning, or multimodal behavior.
Interview answer
“I compare gateways by capability fidelity, policy controls, latency, privacy, availability, and operating cost.”
Benefits, trade-offs, and production considerations
- Validate support for each required feature and provider version.
- Compare self-hosted operation with managed cost and retention controls.
- LiteLLM, Portkey.
Can you explain it without the notes?
LiteLLM, Portkey, and cloud AI gateways
Use your own words. Explain an example and one limitation before checking the explanation.
Check your explanation
LiteLLM provides model-access abstraction and proxy/gateway capabilities. Portkey provides an AI gateway and operational controls. Cloud gateways can apply access and model traffic policies within an existing cloud API-management environment; exact AI…
Turn this concept into a clear answer.
Practice this conceptAdapted from 13_AI.MD · “LiteLLM, Portkey, and cloud AI gateways”. Examples are study fragments; framework behavior can depend on your version.