Building the Complete System: The AWS Reference Architecture

AWS Reference Architecture

How an enterprise AI Gateway actually works in production secrets, identity, provider abstraction, and the operational decisions that separate a system from a demo.

Part 2 of the series · Architecture in the AI Era.

In Part 1, I argued that direct LLM API calls are this decade’s Lambda mistakea powerful abstraction mistaken for a complete system. I made the case that the AI Gateway is becoming the control plane for enterprise AI, the same way API Management became the control plane for microservices a decade ago.

That was the argument. This is the architecture.

In this part, I want to move from the why to the how. Specifically, I want to walk through the parts of an AWS reference implementation that most teams underestimate not the gateway itself, but the operational layer that has to exist around it before the gateway is safe to run in production.

Because here is the uncomfortable truth I keep returning to in my role leading AI Platform and Cloud Transformation: deploying the gateway is the easy part. The architecture lives in everything around the gateway.

The shape of the system

At the centre of the reference architecture is a gatewayin this implementation, LiteLLM running on Amazon EKS. Applications send their requests to a single endpoint. The gateway decides which model answers, enforces policy, tracks cost, and returns the response. The application never knows, and never needs to know, which provider served the request.

That is the part everyone draws on a whiteboard.

What rarely makes it onto the whiteboard is everything that has to be true for that gateway to operate safely:

  • Where do the provider credentials live, and who is allowed to read them?
  • How does the gateway authenticate to AWS without a static key sitting somewhere?
  • How do you add a second or third provider without coupling the application to any of them?
  • What happens when a provider raises prices, deprecates a model, or fails a compliance review?

These are not edge cases. They are the daily reality of running AI in an enterprise. And the quality of your answers to these questions is what determines whether your architecture survives its second year.

Let me take the two most important ones in turn.

Question One: Where do the credentials live?

Every team that puts an AI gateway into production has to answer a deceptively simple question. The gateway needs API keys to talk to model providers. Those keys have to live somewhere. Where?

I see three answers in the field, and all three are wrong.

The first is hardcoded credentials the API key sitting in source code, a config file, or a shell script. The danger is not that someone will see it. The danger is that once a credential enters version control, it is effectively permanent. Every clone, every backup, every CI/CD execution carries it. Removing the line does not remove the history.

The second is treating Kubernetes Secrets as the system of record. Most engineers assume Kubernetes Secrets are encrypted. They are not they are base64 encoded. Anyone with read access to the namespace can retrieve every credential. And the same logical secret ends up duplicated across dev, staging, and production clusters with no unified way to rotate it.

The third is environment variables committed to source control the .env file, the docker-compose.yml with a password in plain text, the Helm values file with the API key. Same root cause as the first: the credential ends up somewhere it can be read by people and systems that should never have had access.

Three different patterns. One common failure mode. The credential lives somewhere it should not live.

The pattern that replaces all three

The reference architecture uses AWS Secrets Manager as the single system of record. Every credential the gateway’s master key, each provider’s API key originates and is managed there. One place. One source of truth. One place to rotate.

That answers where the credentials live. It does not answer the harder question: how does the gateway, running as a pod inside EKS, read those secrets without a credential of its own?

The naive answer is to give the pod an AWS access key. But that simply moves the problem. You have now created a long-lived static credential whose entire job is to access your other credentials and that credential also has to be stored, rotated, audited, and protected.

The right answer is to give the workload an identity instead of a key. On AWS, that mechanism is IRSAIAM Roles for Service Accounts.

With IRSA, the pod never holds a static AWS credential. When it starts, EKS projects a short-lived token into the pod. The AWS SDK exchanges that token, through AWS STS, for temporary credentials that expire within the hour. AWS verifies which service account is asking, which namespace it belongs to, and which IAM role it is allowed to assume and only then issues credentials scoped to exactly what that workload needs.

No static AWS keys. No credentials in environment variables. No credentials in version control. The pod receives short-lived credentials only when it needs them.

The one line that does the security work

There is a single condition in the IAM trust policy that does most of the work, and it is the part teams most often get wrong. It binds the role to one specific service account, in one specific namespace, on one specific cluster.

Without that condition, any workload anywhere on the cluster can assume the role and read the gateway’s secrets. The blast radius of a single compromised pod becomes the blast radius of the entire AWS account.

With it, the gateway has a true workload identity. It can prove it is what it claims to be, and AWS grants it nothing more than the specific secrets it is allowed to read.

This is what I mean when I say the architecture lives around the gateway, not in it. The gateway is a container running open-source software. The security of the whole system rests on an identity model that has nothing to do with the gateway itself.

Question Two: How do you add a provider without creating new lock-in?

The applications calls an alias. The architecture chooses the provider.

The second question is subtler, and it is where many gateway implementations quietly fail.

The reference architecture treats Amazon Bedrock as the primary provider. The reasons are specifically enterprise reasons. Bedrock authenticates through the same AWS identity model, so there are no separate API keys to manage for the main provider. Requests are processed inside your AWS region, so data does not leave your account. Bedrock does not use your prompts to train its models. And every call appears in AWS Cost Explorer alongside the rest of your AWS spend.

Identity. Data residency. Training policy. Billing. Four questions every enterprise architect has to answer before any AI provider reaches production and Bedrock answers all four within the AWS boundary.

So why support a second provider like OpenAI at all? Two reasons. Some models are simply not available in the Bedrock catalogue. And teams already integrated directly with OpenAI need a migration path that does not require rewriting their applications.

But and this is the part that separates an extension of the architecture from a violation of it how you add the second provider determines everything.

The Test

Here is the test I apply. Can you switch a model from one provider to another with zero application code changes?

If the answer is yes, you have an AI gateway. If the answer is no, you have a proxya much smaller thing.

In the reference architecture, every model is exposed to applications through an alias. An application asks for a model by its alias and sends a standard request. It never references a provider. It never holds a provider’s credentials. It never imports a provider’s SDK.

Behind that alias, the gateway configuration maps the alias to an actual provider and model. Today, an alias might route to OpenAI. Tomorrow because of a pricing change, a compliance decision, or a better-suited Bedrock model that same alias can be repointed to Bedrock by changing a single section of the gateway configuration.

Same alias. Same API contract. Same client code. Different provider. Different bill. Different data path.

Zero changes in any application.

This is the property that makes the gateway worth building. Not that it can make a model callany SDK can do that. It is that the gateway preserves your ability to change your mind.

Why this matters at enterprise scale

Consider what this looks like across an organisation with fifty applications and several model providers.

Without the gateway, a provider change is a coordination problem across fifty codebases, owned by different teams on different release cycles. In practice, that migration takes many months and frequently never fully completes. The organisation becomes locked into whatever pricing and roadmap its providers choose, not because anyone decided to be, but because changing direction became too expensive.

With the gateway, a provider change is one configuration change, in one repository, reviewed by one team, deployed in one rolling update. What was an organisational constraint becomes a governance decision.

That is the difference between an enterprise that moves with the market and one that is held in place by its own integration debt. The gateway is not an integration convenience. It is the architectural mechanism that preserves strategic optionality the ability to change providers as the market, your costs, and your compliance requirements change.

The principle underneath all of it

Everything in this architecture comes back to a single principle, and it is the one I keep returning to after watching this pattern repeat with data centres, with APIs, and now with AI:

The application should not choose the provider. The architecture should.

When provider selection lives in the application, every change in the AI landscape becomes a development project. When it lives in the architecture behind a gateway, behind an identity model, behind a governance layer every change becomes a decision you can actually make.

That is what it means to build the complete system. Not the powerful abstraction that demos well, but the operational layer that lets the abstraction survive contact with production, with auditors, with cost reviews, and with a market that will not stop changing.

What Comes Next

This part covered the AWS reference implementation the secrets and identity model, and the provider abstraction that prevents lock-in. The reference implementation, including the configuration, the IAM policies, and the architecture documentation, is openly available for anyone who wants to study how the pieces fit together.

In a future part, I will take the same architectural principles to a second cloud implementing the equivalent pattern on Microsoft Azure using API Management and Azure AI Foundry to demonstrate that a well-designed AI Gateway is not an AWS pattern or an Azure pattern. It is a cloud-agnostic architecture. The principle travels. Only the services change.

Because in the end, the test of an architecture is not whether it works on one cloud. It is whether the thinking behind it holds anywhere.

Want to see what this looks like as a complete system — not just a diagram? WinfoCloudX brings the gateway, identity, governance, and FinOps layers together across AWS, Azure, OCI, and GCP — so your AI architecture preserves optionality from day one.

About the Author

Picture of Winfo

Winfo

What do you think?

From our blog

Articles & insights

Cloud-Based HRMS Software: Complete Guide for Modern Businesses Organizations today need more than payroll processing and employee record management. HR teams are expected to support
For many organisations, Oracle E-Business Suite remains the operational backbone of finance, procurement, manufacturing, supply chain, projects and other business-critical functions. It is stable. It
On-Demand Webinar Ensure Your Oracle EBS Survives the Next Cyber Attack Building Resilience, Security and Recovery with WinfoCloudX 20 August 2026 | 11:00 AM EST