Cloud architecture: what it is, how it works, and what it quietly costs you

In March 2023, an engineering team at Amazon Prime Video published a blog post that made half the internet spit out its coffee.

The Video Quality Analysis team, the folks who monitor thousands of live streams across 240+ countries and territories for freezes and glitches, admitted that their trendy serverless microservices setup was eating them alive. AWS Step Functions orchestration and constant Tier-1 calls to S3, shuttling video frames between distributed components, had become the bill nobody wanted to open.

So they did the unthinkable. They collapsed the whole thing into a monolith running on Amazon ECS and EC2. Same workload. Same team. Different architecture.

As senior engineer Marcin Kolny put it: “Moving our service to a monolith reduced our infrastructure cost by over 90%.”

Amazon, the company that basically invented service-oriented architecture as a religion, saved 90% by breaking its own commandments. Basecamp co-founder David Heinemeier Hansson couldn’t resist piling on, writing that “microservices pose perhaps the biggest siren song for needlessly complicating your system.”

Here’s the part most coverage missed: nothing about the product changed. The only thing that changed was cloud architecture. Which means architecture was never just an engineering diagram. It’s a line item. Possibly your biggest one.

This guide covers what cloud architecture actually is, how it works, the types you’ll choose between, and how AI is currently rearranging all of it. And because we’re CloudZero, we’ll keep asking the question the whiteboard never asks: what does this box cost?

What is cloud architecture?

Cloud architecture is the blueprint for how all the pieces of a cloud environment fit together: hardware, virtual resources, networks, applications, and the services that glue them into something users can actually touch.

Think of it like a restaurant. Diners see the dining room. They never see the kitchen, the walk-in freezer, or the supplier trucks arriving at 5 AM. But the whole operation only works, and only turns a profit, if every part is designed to work with the others.

The cloud architecture components break down into four pieces:

That fourth row deserves a beat. Whether you consume infrastructure as a service, platform as a service, or software as a service decides where your responsibility ends and your bill begins.

We’ve broken down the delivery models in our guide to the types of cloud computing, so we won’t rehash them here.

The formal definition, if you need one for a slide: NIST’s SP 800-145 defines cloud computing through five essential characteristics, three service models, and four deployment models. It’s the closest thing the industry has to a cloud reference architecture, and it’s held up remarkably well since 2011.

But definitions don’t explain why Prime Video’s bill dropped 90%. For that, you need to see how the pieces actually talk to each other.

How does cloud computing architecture work?

Cloud computing architecture works by abstracting physical hardware into virtual resources you can rent, resize, and release on demand. Three mechanisms make that possible.

Virtualization slices physical servers into virtual machines and containers, so one box in a data center can serve dozens of customers. This is the original magic trick, the one that launched EC2 back in 2006 and started the whole industry.

Orchestration decides what runs where and when. Kubernetes schedules containers, autoscalers add capacity when traffic spikes, and services like Step Functions coordinate multi-step workflows. Orchestration is powerful and, as Prime Video discovered, orchestration is billable. Every state transition, every handoff, every retry has a price tag.

APIs and middleware let components communicate without knowing each other’s internals. Your front end asks an API for data, the API asks a database, and nobody has to care what hardware any of it lives on.

Here’s the pattern worth tattooing somewhere visible: every abstraction that makes your life easier meters something. Virtual machines meter hours. Containers meter vCPU-seconds. APIs meter requests. Data moving between all of them meters gigabytes. Cloud computing architecture isn’t just a system design. It’s a pricing surface.

Which is exactly why the shape you choose matters so much. Let’s look at the shapes.

What are the main types of cloud architecture?

There are four main types of cloud architecture, and most companies larger than a food truck end up running some blend of them.

A pure public cloud architecture is where most companies start, and a fully private cloud architecture is where a few regulated ones stay.

Public cloud dominance explains where the money goes: our breakdown of the biggest cloud service providers shows just how concentrated the market has become among AWS, Azure, and Google Cloud.

A quick word on the last two rows, because they’re where architecture and accounting file for divorce. Hybrid setups mean your teams reconcile on-prem depreciation against cloud consumption billing. And multi-cloud fragmentation is real enough that we wrote a whole guide to multi-cloud management tools for teams trying to see across providers.

One more architectural fork that shows up inside these types: whether each customer gets their own isolated stack or shares infrastructure with others. That’s the single-tenant versus multi-tenant decision, and for SaaS companies it directly shapes gross margin per customer.

If you’re building SaaS specifically, our SaaS architecture guide goes deep on that whole territory.

So those are the deployment shapes. But there’s a design philosophy question that cuts across all of them, and it’s the one Prime Video wrestled with.

What is cloud native architecture?

Cloud native architecture means building applications specifically for the cloud rather than lifting-and-shifting old software into it. In practice: microservices instead of monoliths, containers instead of VMs, managed services instead of self-hosted everything, and automation instead of humans SSH-ing into servers at 2 AM.

The building blocks are familiar. Containers packaged and scheduled by Kubernetes (our nodes versus pods explainer covers the moving parts). Serverless functions like AWS Lambda that run code without provisioning anything. Container services like ECS, and yes, choosing between ECS and EC2 is itself an architecture-and-cost decision, as Prime Video demonstrated in the most public way possible.

For cloud application architecture, cloud native buys you real things: independent deployments, fault isolation, and the ability to scale horizontally instead of vertically so failures stay small and traffic spikes stay survivable.

But here’s the plot twist the Prime Video saga taught everyone: cloud native is a toolbox, not a loyalty oath. Netflix famously went all-in on microservices after a 2008 database corruption took down its monolith for days. Prime Video went the other direction for one specific workload and saved 90%. Both were right. Architecture serves the workload, not the conference keynote.

And the finance angle: cloud native’s granularity cuts both ways. Billing by the function-invocation and vCPU-second gives you incredibly precise cost data. It also means your bill has gone from 50 line items to 50,000, and someone has to map those back to products and customers.

If cloud native was the last decade’s architectural earthquake, the current one is bigger. And it has a GPU shortage.

How is AI rewriting cloud architecture?

For fifteen years, cloud architecture meant CPUs, storage, and networking. AI added an entirely new wing to the building, and it’s the expensive wing.

The scale is honestly hard to overstate. The five largest hyperscalers, Amazon, Microsoft, Alphabet, Meta, and Oracle, plan to spend over $600 billion on capital expenditure in 2026, a roughly 36% jump over 2025, with CreditSights estimating around 75% of it (about $450 billion) going directly to AI infrastructure. Amazon alone guided to roughly $200 billion, up from $125 billion the year before.

And it’s not slowing down. McKinsey projects the global data center buildout could require $6.7 trillion by 2030, with $5.2 trillion of that going to AI-ready capacity alone. Analysts now peg hyperscaler capital intensity at 45% to 57% of revenue, ratios that look more like utilities than software companies.

Your architecture diagram now has boxes it didn’t have in 2022:

The uncomfortable part? The money is moving faster than the visibility. In CloudZero’s 2026 AI ROI survey of 260 finance leaders, 55% said they ran over their AI budget last year, and 32% overshot by more than 20%. That’s not a discipline problem. That’s what happens when token-metered architecture gets bolted onto infrastructure faster than anyone updates the cost model.

“Every AI architecture decision is a unit economics decision,” says Erik Peterson, CloudZero founder and CTO.

So how do you design for a world where the architecture and the invoice are the same document? Glad you asked.

How do you design a cloud architecture that doesn’t bankrupt you?

Cloud architecture design traditionally optimizes for five things: scalability, reliability, security, performance, and operability. All still true. But the Prime Video lesson, and the $600 billion AI buildout, argue for promoting a sixth principle to the varsity squad: cost as a first-class design constraint.

Here’s what that looks like in practice, decision by decision:

A few cloud architecture best practices that flow from treating cost as a design input: architect for scalability in both directions, because scaling down is where savings actually live.

Design data flows to minimize transfer, since egress is the toll booth of the cloud. Tag and structure resources so spend maps to products and teams from day one, not as a forensic exercise later.

Above all, revisit architecture decisions when volume changes, because the right answer at 1x traffic is often the wrong answer at 100x. Prime Video’s original serverless design was genuinely correct for a prototype. It just stopped being correct at scale, and they were paying attention.

One practical note on cloud architecture diagrams: draw them with billing units on the arrows, not just data flows.

A diagram that shows “video frames → S3 → detector” looks fine. A diagram that shows “video frames → S3 (Tier-1 calls, $) → detector” is how you spot a 90% savings opportunity before it ships.

Design principles get you a sound structure. But sound structures still surprise you at invoice time, which brings us to the part where we stop pretending we’re neutral.

How CloudZero turns cloud architecture into unit economics

Everything above ends in the same place: modern cloud architecture generates cost data faster than humans can interpret it.

Kubernetes bills in vCPU-seconds, model APIs bill in tokens, GPUs bill by the hour, and your CFO asks one question none of those units answer: what does it cost us to serve a customer?

That translation is what CloudZero does.

CloudZero’s CostFormation engine maps raw billing data from every provider, AWS, Azure, GCP, Snowflake, Databricks, and AI platforms, to the dimensions your business actually thinks in: products, features, teams, and customers. The allocation engine handles the messy parts, like splitting shared Kubernetes clusters and multi-tenant infrastructure fairly, without demanding a perfect tagging strategy first.

For the AI wing of your architecture, AI Hub tracks GPU, model API, and training spend in one view, so the Bedrock endpoint, the OpenAI invoice, and the H100 cluster stop living in three different spreadsheets.

And anomaly detection catches architecture-driven cost spikes, the orchestration loop or runaway retry pattern, in hours instead of at month-end. It’s how we help companies managing over $15 billion in combined cloud and AI spend answer the whiteboard question and the invoice question at the same time.

If your architecture diagram and your cloud bill feel like two unrelated documents, get a demo and we’ll show you how they connect. Prefer to poke around solo? Take the self-guided tour. Or start with a free cloud cost assessment and find out what your current architecture is really costing you per unit.

Frequently Asked Questions about cloud architecture

Author Spotlight

Lyne Carolyne

Lyne Carolyne has several years of experience in AI and cloud economics and brings that understanding into the content she creates. Outside work, she's an avid explorer.