Why direct LLM API calls are this decade’s Lambda mistake — and what AWS, Azure, and Google are doing about it.
There is a specific moment in every technology cycle when an abstraction feels so complete that it stops being treated as a component. It starts being treated as a system.
That moment is almost always wrong.
I wrote about this, when I argued that the Lambda mistake — wiring critical workflows to an event-driven compute primitive without the operational scaffolding around it — was about to repeat with GenAI agents.
A year on, the repeat has begun. But it is not agents that are the most pervasive issue. It is something simpler.
Direct LLM API calls.
The same pattern. The same illusion of completeness. And at scale the same expensive consequences.
The new powerful abstraction
An API call to OpenAI, Anthropic, or Amazon Bedrock is the most concentrated form of architectural seduction the software industry has produced in years.
One HTTP request goes out. One JSON response comes back.
Inside that response is something that can reason, summarise, search, classify, translate, generate code, draft documents, follow instructions, and converse.
No infrastructure to provision. No model to train. No serving layer to maintain. No GPU capacity to plan for.
In a Jupyter notebook, it looks complete. In a proof of concept, it looks complete. In a demo to leadership, it looks complete.
It is not complete.
It is the most powerful abstraction in modern software, wrapped in a single endpoint and that is the dangerous part. Because the abstraction hides the system that should sit underneath it.
Direct LLM API calls — the broken pattern. Each application holds its own credentials and chooses its own provider, with no governance layer in between.
What "complete" actually requires
In my role leading AI Platform and Cloud Transformation, the question I keep returning to is not “which model should we use?” It is: what does it take to operate this in production without surprises?
The answer is uncomfortably long.
Concern | Why it cannot be skipped |
Routing | One model is rarely correct for every task. Classification does not need GPT-4. Long-context reasoning does not always fit Claude. Routing is a decision layer, not a default. |
Failover | Every model provider had outages in 2025. Without failover, your AI feature is down whenever their region is. |
Rate limiting and quotas | A single misbehaving loop can exhaust an enterprise’s entire monthly token budget in hours. |
Cost attribution | Without per-team, per-application accounting, no enterprise can answer “who spent what on AI last month?” — a question the CFO will eventually ask. |
Caching | Semantic caching alone reduces LLM spend by 30–50% in customer support workloads. Without it, identical questions are billed thousands of times. |
Guardrails | Prompt injection, PII exposure, hallucination detection, output filtering. In regulated industries, these are the compliance perimeter. |
Observability | Token usage, latency distributions, error rates, model selection patterns. Without these, you discover production failures from customer support tickets. |
Secrets management | Provider API keys must rotate, expire, and never live in application code. |
Lifecycle management | Models get deprecated. Pricing changes. Capabilities shift. Without an abstraction layer, every deprecation is an emergency refactor. |
That is the difference between an API call and a system.
The API call is the input. Everything in the table above is the system that must exist around it.
When teams skip the system, the abstraction does its job until it doesn’t. When it doesn’t, the failure mode is usually expensive, public, or both.
Where this becomes catastrophic: the scale break
The illusion of completeness survives the first ten applications.
It begins to crack at fifty.
The architecture diagram on the wall looks innocent. Each team has its own AI integration. Each integration is two screens of Python code. Each developer manages their own API key.
It works. Each team ships features. The company looks AI-forward. Leadership is satisfied.
Then the CIO walks into a meeting and asks four questions.
How much did we spend on AI last month? Answer: scattered across five providers and twelve teams. No single number exists.
Which model is serving production traffic right now? Answer: each application made its own choice. Some chose three months ago. Nobody has updated the registry.
Which applications are sending customer data to external AI providers? Answer: legal is about to ask the same question. The room goes quiet.
If OpenAI raises prices 40 percent next quarter, what is our exposure — and how fast can we switch? Answer: every application has the provider hard-coded. The honest estimate is months.
That is not a model problem. It is not a tooling problem.
It is an architecture problem. And it surfaces at scale because powerful abstractions hide the architectural question until the scale forces it into the open.
By then, the cost of fixing it is two years of technical debt.
The industry has converged on the answer
The most important development of 2026 is that the three major cloud providers have all arrived at the same architectural conclusion within six months of each other.
Amazon Web Services published the Multi-Provider Generative AI Gateway reference architecture in late 2025. It provides a single AWS-managed entry point for enterprise applications, with routing across Amazon Bedrock, Amazon SageMaker AI, and external providers including OpenAI, Anthropic, and Google. In parallel, AWS made Amazon Bedrock AgentCore generally available a full agentic platform with its own gateway, runtime, memory, identity, and observability stack, designed to work with any framework and any model.
Microsoft Azure shipped the API Management AI Gateway pattern as part of its 2026 reference architecture. The pattern fronts Azure AI Foundry with Azure API Management for token-based rate limiting, semantic caching, model routing, and multi-tenant governance. The same gateway pattern that managed microservices for a decade is now Microsoft’s recommended pattern for AI workloads.
Google Cloud continues to consolidate Vertex AI as the production infrastructure for GenAI engineers with managed RAG, Gemini-native workflows, and compliance postures. Google was named a Leader in the 2025 IDC MarketScape for GenAI Lifecycle Foundation Model Software.
And Gartner has been direct. In their Innovation Guide for Generative AI Engineering, Gartner identifies that the question has shifted: vendors are moving “well beyond prompting a model and into full-fledged GenAI platforms covering pipelines, context engineering, orchestration, and governance.”
Gartner’s prediction: by 2029, 70 percent of enterprises will deploy agentic AI in production — up from less than 5 percent in 2025. By 2028, half of all secure GenAI deployments will be driven by LLM observability investments. None of that is achievable without an orchestration and governance layer.
Truefoundry’s analysis of Gartner’s position is the clearest summary: the AI Gateway is the control plane that makes GenAI safe to scale.
When AWS, Microsoft, Google, IDC, and Gartner all arrive at the same architectural conclusion in the same window, the question stops being “is this needed?” The question becomes “how soon.”
The fix at a conceptual level
The pattern emerging now is structurally identical to the one the industry adopted a decade ago for microservices.
A decade ago, the principle was: never let clients call microservices directly. The fix was the API Gateway.
In 2026, the principle is: never let applications call models directly. The fix is the AI Gateway.
The architectural role is the same:
- One endpoint that all applications call
- Routing logic that decides which model handles each request
- Authentication, rate limiting, and quota enforcement
- Cost attribution per team, per application, per request
- Observability for latency, errors, and token consumption
- Semantic caching for repeated queries
- Guardrails for safety, PII protection, and compliance
- Failover when a provider goes down
- Lifecycle management as models deprecate
Applications no longer know which model they are calling. They no longer manage credentials. They no longer hard-code provider SDKs. They make one call to ai.company.com and the gateway handles everything else.
This is exactly the abstraction that API Management provided for microservices — and exactly the abstraction that direct LLM calls do not provide for AI.
The AI Gateway is what turns the powerful abstraction into a complete system.
The AI Gateway pattern. Applications converge through a single governance plane that handles routing, observability, cost attribution, guardrails, secrets, and lifecycle.
What this means for the next eighteen months
A decade ago, “should we use an API Gateway?” was a debate. By 2018, it was a default. Enterprises that delayed adoption paid for it with two years of compounding technical debt vendor lock-in, fragmented authentication, untracked costs, security gaps that took years to close.
The same window is open now.
The enterprises that establish their AI Gateway architecture in 2026 will spend the rest of the decade scaling their AI investment confidently. The enterprises that delay will spend the rest of the decade unwinding hard-coded provider dependencies, scattered API keys, and ungoverned costs.
In Part 2 of this series, I will walk through the actual reference architectures on AWS and Azure — what each component does, how they fit together, the operational concerns that almost nobody addresses until production forces the conversation, and the organisational question every enterprise must answer next: who owns the AI Gateway?
For now, the most useful principle I can leave you with is the one that has guided how I think about every architectural decision over two decades.
Powerful abstractions are not complete systems. The complete system is what survives production.
Still calling cloud and AI providers directly? WinfoCloudX gives enterprises a unified framework for cloud migration, cost optimisation, security, and governance across OCI, AWS, and Azure — the operational scaffolding this article is about.