Coder Legion - Recent posts tagged aws

From Writing the Spec to Approving It: What AI-DLC Looks Like on a Real Build

Fri, 30 Oct 2026 10:36:49 +0000

🤖 From Writing the Spec to Approving It: What AI-DLC Looks Like on a Real Build

TL;DR

AI-DLC is public and open source: awslabs/aidlc-workflows. The method comes from this AWS DevOps blog post.

My only written input was one sentence. Everything else I did was pick options and approve gates.

The run produced requirements, four architecture decisions, a unit plan, a traceability file, and 12 tests that passed when I ran them, in about 44 minutes from start to the code approval.

The human role moves from author to decision-maker. The decisions are still yours, and the record shows where they changed the outcome.

Most AI-assisted development still follows the old split. A human writes the spec, a human writes the code, and the AI helps with autocomplete or generates tests at the end. AI-DLC (the AI-Driven Development Life Cycle) reverses that. The AI drafts the requirements, the design, and the code. The human answers short questions and approves each stage before the next one starts.

To see what that looks like in practice, I built a small command-line tool with it and then read the record the workflow left behind. This post walks through that record, folder by folder.

The app

The tool is a Spooky Date Fact Generator. It picks one of 10 spooky dates (or takes --date MM-DD) and prints a spooky event for that day from Wikipedia's "On this day" feed. It is small on purpose.

The whole request was this sentence:

/aidlc build random date generator with fact around that number

The flip, in one table

	Traditional with AI help	AI-DLC
Requirements	Human writes them	AI drafts them from multiple-choice answers
Design	Human sketches it	AI proposes, human picks between options
Code	Human writes, AI assists	AI writes against an approved plan
Tests	Human or AI at the end	AI writes them to the stated requirements
Human job	Author	Decide, validate, approve
Evidence	Pull request diff	Audit log, decisions, traceability file

Where everything was written

Two locations matter. The generated app sits in its own folder, and the workflow record sits in an aidlc/ folder.

app/                                  <- the generated tool
  src/spooky-facts.ts      core: dates, fetch, keyword filter, nearest-date search
  src/cli.ts               command line: options, output, exit codes
  test/                    12 tests, offline, with a fake fetch
  package.json, biome.json, tsconfig.json, README.md

aidlc/
  codekb/                             <- 9 docs about the existing repo
  intents/<date>-date-fact-generator/ <- the record for this build
    aidlc-state.md                    stage checklist and settings
    audit/                            every event, decision, and approval
    inception/
      reverse-engineering/            scan of the existing repo
      requirements-analysis/          questions + requirements.md
      domain-design/                  questions + components.md + decisions.md
      units-generation/               unit-of-work files
    construction/u1-spooky-date-cli/code-generation/
      code-generation-plan.md, code-summary.md,
      unit-test-instructions.md, traceability.json

Each stage writes a -questions.md file, and your answers sit in the same file as the AI's questions. That makes the decision trail readable without any tooling.

Stage by stage: what it did and why it helps

Initialization. The workflow scaffolded the record folder, scanned the workspace, and classified it as brownfield TypeScript on Bun. This is automatic. It helps because every later stage starts from facts about the repo instead of guesses.

Reverse Engineering. A developer agent scanned the existing repo and wrote nine knowledge-base documents (architecture, technology stack, code structure and others). This matters most in a brownfield repo. The later code followed the repo's Bun, TypeScript, and Biome conventions because the AI had read them first.

Requirements Analysis. The AI asked six questions, such as where facts come from, what to do when a date has no spooky event, and whether this should be a web page or a command line. Each had lettered options. I answered, and one answer included a free-text note: generate dates that have spooky facts, from 10 such scenarios. The result is requirements.md with 15 functional and 5 non-functional requirements, each with acceptance criteria and a tag pointing back to the answer that produced it, such as [Q5]. This stage turns one vague sentence into testable statements before any code exists.

Domain Design. Four more questions produced components.md and four architecture decision records. The AI recommended splitting the tool into four parts. I chose two. The decision record states that the four-part split was rejected as more structure than needed. This is the clearest example of the validator role, because the human overrode the AI's recommendation and the reason is recorded.

Units Generation. One question: build as one unit or two. I chose one, approved the plan, and got unit-of-work.md with boundaries, responsibilities, and constraints. It fixes the blast radius. The unit is limited to apps/date-fact-generator/, and nothing in the framework folders may change.

Code Generation. The AI wrote a plan, then the code, then the tests. It asked one thing here: Bun was not installed, so should it install Bun or switch to Node. I chose Bun. The code-summary.md lists files created, decisions made, and deviations from the plan. One deviation worth noting: the AI added word forms such as "dies" and "sank" to the keyword list, because whole-word matching would otherwise miss "Houdini dies". The summary also states that some planned checks were grouped into fewer tests to match the 12-test count.

Build and Test. In the record I read, this stage was still marked in progress, so its summary was not written yet. I ran the checks myself from the app folder: 12 tests passed with 93 assertions, and lint and typecheck were clean.

What the human actually did

Reading the audit log, the human work was this:

Typed one sentence to start.
Approved a recomposed plan. The workflow's planner found the default express plan was missing inputs for later stages, and proposed adding Domain Design and Units Generation while dropping deployment and monitoring.
Answered 11 multiple-choice questions across three stages, plus two more decisions: build inside the existing repo, and install Bun.
Approved 5 stage gates: reverse engineering, requirements, domain design, units, and code generation.

That is the flip. Nothing in that list is writing a spec or writing code. All of it is deciding and checking.

Why this holds up better than a prompt

Two details make the validation real instead of a rubber stamp.

First, traceability. traceability.json maps every requirement ID (FR1.1 through NFR5, 20 in total) to the file that satisfies it. If a requirement has no target, it shows up.

Second, the record is complete. The audit log holds the timestamp of each question, answer, and approval. A reviewer six months later can see why the design has two components and not four.

What this run does not show

This was an express-scope build of a 658-line app, with plan approval, sensors, and learnings switched off by the scope. It shows the shape of the method, not how it behaves on a large system with many units and teams. The Build and Test stage had not completed in the record, and the tests are offline, so only one manual live smoke run, recorded in the code summary, exercised the real Wikipedia feed.

The questions are only as good as the options offered. A weak option list can steer a hurried approver, so reading the alternatives still matters.

Takeaway

The useful change is not that the AI writes more code. It is that the human's time moves to the points where judgment counts: what the tool should do, which design to accept, and whether the result meets the requirements. If you want to try it, start with a small brownfield repo and read the -questions.md files first. They show the method faster than any description.

I automated the hardest parts of Amazon SES, so you can send from your own AWS account in two minute

Wed, 28 Oct 2026 06:00:57 +0000

Amazon SES is the most cost-effective way to send transactional email. But a production setup
takes real work: IAM, the sandbox, bounce and complaint handling, logs, and watching your
reputation. I built SendFleet to do that work inside your own AWS account.

What it does

One-click connection. A CloudFormation template creates a scoped IAM role. Paste its ARN and
you are connected. No AWS keys are ever shared.
Finds your sending region. Production access is per region, so SendFleet checks them all and
picks the one that can send. If none can, it requests production access for you.
Tracking and webhooks. It sets up the configuration set and SNS topic in your account, keeps
them healthy, and turns events into an email log and signed webhooks.
Reputation protection. It pauses sending before your bounce or complaint rate reaches the
levels where AWS starts reviewing accounts.
Cost controls. You choose which SES events your account publishes, and set a monthly cap.
Transparency. Every change SendFleet makes in your account is listed in the dashboard.

How it is built

A Rust API for sending (POST /v1/send).
A Django dashboard.
Python Lambdas in a separate AWS account: a relay that assumes your role to send, a control
plane for setup and self-healing, and an event worker.
Postgres on Neon. Message bodies never reach the database.

Pricing

Free for 3,000 emails a month. $5 a month with 20,000 emails included, then $0.10 per 1,000. You
pay AWS for SES directly.

Try it at https://sendfleet.net. I'm keen to hear what would make you trust a tool like this with
your AWS account.

Organization-Wide EC2 DNS Automation with EventBridge & AWS Config Aggregator: One Rule Per Account

Tue, 27 Oct 2026 19:19:28 +0000

🌐 Organization-Wide EC2 DNS Automation with EventBridge and AWS Config Aggregator: One Rule Per Account

TL;DR

Detecting EC2 launches and terminations across a whole AWS Organization for DNS is not the five-minute single-account job it looks like.

The EC2 state-change event carries only an instance ID and a state, no IP, so it is useless for DNS on its own.

A hub-and-spoke design keeps the per-account footprint to a single EventBridge rule; all logic lives in one hub account.

The hub enriches each event by querying the AWS Config Aggregator you already run, which holds every instance's IP, tags, and state org-wide. No new data pipeline.

An aws:PrincipalOrgID resource policy on the hub's custom bus gives cross-account fan-in without per-account config.

SQS between the rule and the Lambda absorbs bulk launches and gives you retries plus a dead-letter queue.

How to detect every EC2 launch and termination across an entire AWS Organization, and turn it into automated DNS registration, without deploying repeated infrastructure in every account.

I spent longer than I would like to admit staring at this problem before the answer got simple. So if you are wiring up DNS across an AWS Organization right now, maybe this saves you a weekend.

Here is the short version of what I landed on. I needed near real-time DNS registration for every EC2 instance across a multi-account, multi-region org, feeding an on-prem enterprise DDI system (think Infoblox). The two obvious ways to do it both annoyed me: one puts infrastructure in every account, the other leans on a fragile cross-account API-scraping job.

What I used instead is a hub-and-spoke event setup:

Each spoke account gets exactly one EventBridge rule. That is the whole footprint.
One hub account does everything else: a custom event bus, an SQS queue, one Lambda.
The Lambda fills in the missing details by querying the AWS Config Aggregator, which, and this is the part I almost missed, already has every instance's IP, tags, and state for the entire org. No new data pipeline required.

Onboarding a new account came down to a one-line deploy. Below is the whole thing: the architecture, why I made the calls I made, and the code.

The problem (why the single-account version does not scale)

In one account this is a five-minute job. EC2 fires a state-change event, you point an EventBridge rule at a Lambda, done. I have written that Lambda a dozen times.

Across an organization it falls apart for a few reasons:

Instances come up in dozens of accounts and several regions.
The people who run DNS are not the people who own the workload accounts, so "just add it to the account" is not really on the table.
The EC2 state-change event does not include the IP address. You get an instance ID and a state. That is it. Useless for DNS on its own.
Nobody wants to deploy, version, and babysit a Lambda plus queue plus IAM role in every account just to catch these events.

The usual reason you end up here is a move to an enterprise DDI platform. Every instance that boots needs a DNS record, every instance that dies needs its record cleaned up, org-wide, in near real-time, with as little junk in each account as you can manage.

So two things really drove the design:

Adding an account should be almost free.
The IP is not in the event, so I have to look it up, and that lookup cannot be flaky.

The approaches I threw out

Option A: put the whole stack in every account. Rule, queue, Lambda, IAM role, shipped everywhere with StackSets. It works. But now you own N copies of the same code, and every bug fix or IAM tweak is an N-account rollout. I have maintained fleets like that. No thanks.

Option B: a central poller that assumes a role into the source account and calls DescribeInstances for the IP. Logic is centralized, which is nice, but you need an assume-role trust in every account and you are paying a cross-account API hop on every single event. More IAM surface, more things to break.

Option C: hub-and-spoke, enriched from the Config Aggregator. This is the one. All the logic sits in the hub. Spokes just forward. And instead of building anything to fetch the IP, the hub asks the Config Aggregator, which is already sitting there aggregating EC2 config (IP, tags, state) from every account and region.

Option C won for a boring but decisive reason: the hard part, the enrichment data, already existed. I did not have to build it. I just had to notice it was there.

How it fits together

Spokes forward EC2 state changes to the hub's custom bus; the hub enriches via the org Config Aggregator, then calls the DDI system. Spokes are dumb, the hub is smart, and the data already exists. That is the whole idea in one line.

That second hub rule is easy to forget, by the way. If you only listen on the custom bus, the hub account's own instances never get DNS records, because they never leave home to reach the custom bus. Ask me how I know.

Filling in the blanks

Most single-account tutorials skip this bit, which is exactly the bit that matters at scale. Here is what the event actually gives you:

{
  "source": "aws.ec2",
  "detail-type": "EC2 Instance State-change Notification",
  "detail": {
    "instance-id": "i-1234567890",
    "state": "running"
  }
}

An ID and a state. No IP, no tags, no hostname. So the Lambda goes and gets the rest, from the aggregator, not from the source account.

The nice thing: the aggregator is org-level, so one query resolves an instance no matter which account or region it lives in. No assume-role, no per-account permissions to manage.

The code

Everything here is vendor-neutral and anonymized. Swap the Infoblox call for whatever you actually talk to: Route 53, a CMDB, an IPAM API.

The spoke's one and only rule (Terraform)

This is the entire per-account footprint. Ship it with StackSets or a Terraform module.

# --- Deployed in EVERY spoke account/region ---

resource "aws_cloudwatch_event_rule" "forward_ec2_state" {
  name           = "forward-ec2-state-to-hub"
  description    = "Forward EC2 running/terminated events to the hub account"
  event_bus_name = "default"

  event_pattern = jsonencode({
    source        = ["aws.ec2"]
    "detail-type" = ["EC2 Instance State-change Notification"]
    detail = {
      state = ["running", "terminated"]
    }
  })
}

resource "aws_cloudwatch_event_target" "to_hub_bus" {
  rule           = aws_cloudwatch_event_rule.forward_ec2_state.name
  event_bus_name = "default"
  arn            = "arn:aws:events:${var.region}:${var.hub_account_id}:event-bus/ec2-dns-hub-bus"
  role_arn       = aws_iam_role.eventbridge_cross_account.arn
}

# Minimal role that lets EventBridge PutEvents to the hub bus
resource "aws_iam_role" "eventbridge_cross_account" {
  name = "eventbridge-forward-to-hub"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "events.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "put_events" {
  role = aws_iam_role.eventbridge_cross_account.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = "events:PutEvents"
      Resource = "arn:aws:events:${var.region}:${var.hub_account_id}:event-bus/ec2-dns-hub-bus"
    }]
  })
}

The hub's custom bus, with an org-scoped resource policy

This policy is the thing that makes "one rule per spoke" actually work: any account in the org can publish, and the hub needs zero per-account config.

# --- Deployed ONCE in the hub account ---

resource "aws_cloudwatch_event_bus" "hub" {
  name = "ec2-dns-hub-bus"
}

resource "aws_cloudwatch_event_bus_policy" "allow_org" {
  event_bus_name = aws_cloudwatch_event_bus.hub.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "AllowOrgAccountsToPutEvents"
      Effect    = "Allow"
      Principal = "*"
      Action    = "events:PutEvents"
      Resource  = aws_cloudwatch_event_bus.hub.arn
      Condition = {
        StringEquals = {
          "aws:PrincipalOrgID" = var.aws_org_id
        }
      }
    }]
  })
}

Do not skip the aws:PrincipalOrgID condition. It scopes that Principal = "*" down to your organization instead of, you know, the entire internet. It is the one line standing between "elegant pattern" and "open event bus."

Rules, queue, and Lambda wiring (hub)

resource "aws_sqs_queue" "events" {
  name                       = "ec2-dns-events"
  visibility_timeout_seconds = 120
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 5
  })
}

resource "aws_sqs_queue" "dlq" {
  name = "ec2-dns-events-dlq"
}

# Rule on the custom bus (events from spokes)
resource "aws_cloudwatch_event_rule" "hub_custom" {
  name           = "ec2-state-custom-bus"
  event_bus_name = aws_cloudwatch_event_bus.hub.name
  event_pattern = jsonencode({
    source        = ["aws.ec2"]
    "detail-type" = ["EC2 Instance State-change Notification"]
    detail        = { state = ["running", "terminated"] }
  })
}

resource "aws_cloudwatch_event_target" "custom_to_sqs" {
  rule           = aws_cloudwatch_event_rule.hub_custom.name
  event_bus_name = aws_cloudwatch_event_bus.hub.name
  arn            = aws_sqs_queue.events.arn
}

# Second rule on the default bus catches the hub account's OWN instances
resource "aws_cloudwatch_event_rule" "hub_default" {
  name           = "ec2-state-default-bus"
  event_bus_name = "default"
  event_pattern = jsonencode({
    source        = ["aws.ec2"]
    "detail-type" = ["EC2 Instance State-change Notification"]
    detail        = { state = ["running", "terminated"] }
  })
}

resource "aws_cloudwatch_event_target" "default_to_sqs" {
  rule = aws_cloudwatch_event_rule.hub_default.name
  arn  = aws_sqs_queue.events.arn
}

resource "aws_lambda_event_source_mapping" "sqs_to_lambda" {
  event_source_arn = aws_sqs_queue.events.arn
  function_name    = aws_lambda_function.dns_registrar.arn
  batch_size       = 10
}

The enrichment Lambda (Python)

This is the heart of it: resolve the instance against the Config Aggregator, then act on the state.

import json
import boto3

config = boto3.client("config")

AGGREGATOR_NAME = "OrganizationConfigAggregator"

def get_instance_details(instance_id: str) -> dict | None:
    """Resolve IP, tags, account, and region from the org Config Aggregator."""
    expression = (
        "SELECT resourceId, accountId, awsRegion, "
        "configuration.privateIpAddress, "
        "configuration.publicIpAddress, tags "
        f"WHERE resourceId = '{instance_id}' "
        "AND resourceType = 'AWS::EC2::Instance'"
    )
    resp = config.select_aggregate_resource_config(
        Expression=expression,
        ConfigurationAggregatorName=AGGREGATOR_NAME,
        Limit=1,
    )
    results = resp.get("Results", [])
    return json.loads(results[0]) if results else None


def handler(event, _context):
    for record in event["Records"]:
        body = json.loads(record["body"])
        detail = body["detail"]
        instance_id = detail["instance-id"]
        state = detail["state"]

        details = get_instance_details(instance_id)
        if not details:
            # Terminated instances may already be gone from the aggregator;
            # fall back to event data for deregistration.
            if state == "terminated":
                deregister_dns(instance_id)
            continue

        ip = (
            details.get("configuration", {}).get("privateIpAddress")
            or details.get("configuration", {}).get("publicIpAddress")
        )

        if state == "running" and ip:
            register_dns(instance_id, ip, details.get("tags", []))
        elif state == "terminated":
            deregister_dns(instance_id)


def register_dns(instance_id, ip, tags):
    # Replace with your Infoblox / Route 53 / IPAM call.
    # Infoblox example: POST to the WAPI record:a endpoint.
    print(f"REGISTER {instance_id} -> {ip}")


def deregister_dns(instance_id):
    print(f"DEREGISTER {instance_id}")

One thing worth calling out: that SELECT against the aggregator resolves a resource across every account and region in a single call. That is the capability that collapses the whole cross-account enrichment headache into a one-liner. I keep coming back to it for other problems now.

And notice the fallback in the handler. A terminated instance has often already dropped out of the aggregator by the time you look, so you cannot rely on the lookup for deregistration. You fall back to the instance ID from the event. Small thing, but it will bite you in testing if you miss it.

Why there is an SQS queue in the middle

You can target the Lambda straight from the rule, and for a demo I would. In production I put SQS in between on purpose:

A bulk launch, a scale-out, a StackSet rollout, can throw hundreds of events at you at once. SQS soaks that up and lets the Lambda drain at its own pace instead of getting hammered.
When Infoblox or the aggregator has a bad minute, messages retry and eventually land in a DLQ you can actually look at. Nothing quietly disappears.
Batching. Ten events per invocation, fewer cold starts, lower cost.

Decoupling ingest from processing is most of what separates "worked in the demo" from "still working at 2am during a mass launch."

What actually came out of it

Detection and DNS updates happened within seconds, across the whole org.
Adding an account or region was a single forwarding rule. No Lambda, no queue, no Config setup per account.
Registration and cleanup were fully automated across the instance lifecycle, nobody touching DNS by hand.
Cost tracked instance activity, not the clock, because there is no idle compute.
The shape is reusable. Same hub-spoke plus aggregator-enrichment works for CMDB sync, security tagging, IPAM updates, anything org-wide and event-driven.

The part worth stealing

Take Infoblox out of the picture and you are left with a pattern I now reach for often:

Fan org-wide events into one hub, then enrich them from a data source the organization already has, instead of building a new one.

Two moves do the work:

A custom event bus with an aws:PrincipalOrgID resource policy gives you cross-account fan-in and a one-rule spoke footprint.
The Config Aggregator, used as an enrichment lookup, resolves any resource in any account or region without an assume-role fan-out.

So the next time you are about to copy the same Lambda into 40 accounts, stop and ask whether a fan-in plus an aggregator you already run could do it with one rule per account instead. Usually it can.

Wrapping up

Honestly, none of the individual pieces here are clever. Cross-account EventBridge is documented. Config aggregators are documented. The win was in composing them so the per-account cost dropped to near zero and the enrichment data was something we already had lying around. That is where most of the good architecture calls seem to live, not in the shiny new component, but in the one you talked yourself out of building.

If you end up adapting the enrichment step to Route 53, a CMDB, or your own IPAM, I am happy to talk through it in the comments.

Kubernetes Security Is Three Problems, Not One: Admission, Identity, Runtime

Thu, 22 Oct 2026 03:19:32 +0000

🛡️ Kubernetes Security Is Three Problems, Not One: Admission, Identity, Runtime

TL;DR

Most Kubernetes compromises start inside the perimeter, where the firewall verifies nothing.

Treat security as three independent problems: admission (what deploys), identity (what a pod can reach), runtime (what a pod actually does).

Admission: Pod Security Admission as a free floor, Kyverno for per-resource policy, Gatekeeper where you need Rego. Run in audit mode first, then enforce.

Identity: IRSA kills static AWS keys, ExternalSecrets kills hardcoded app secrets. Scope roles per workload.

Runtime: Falco watches syscalls via eBPF and catches what the manifest never declared. Budget time to tune per-workload rules.

Supply chain is the layer most teams skip: Cosign signing plus Kyverno verifyImages closes the mutable-tag gap, and SLSA provenance plus SBOMs answer "which running images have this CVE?"

Kubernetes security failures usually do not look like network intrusions. An attacker who gets code execution inside a container, through an application bug, a poisoned dependency, or a policy gap that let a privileged pod schedule, is already inside the perimeter the firewall was watching. From there the firewall verified nothing: which secrets the process can reach, which syscalls it can make, whether the running image is even the one you signed.

That is the useful reframe. Kubernetes security is not one problem at the boundary. It is three problems inside it, and each one needs its own control that keeps working when the others fail:

Admission decides what is allowed to deploy, and with what configuration.
Identity decides what each running pod is allowed to reach.
Runtime watches what a pod actually does, which is often not what its manifest claimed.

None of the individual tools below are new, and none of this is a new technique. What is worth sharing is how the layers fit together, so a failure in any one of them does not open an exploitable gap. The model maps cleanly onto NIST SP 800-207 (zero trust) and NIST SP 800-190 (container security), and you can hold it to the CIS Kubernetes Benchmark as a measurable floor.

Layer 1: admission, decide what gets in

Start with what Kubernetes gives you for free. Pod Security Admission (PSA) enforces security profiles per namespace with no external tooling. The Restricted profile blocks host namespace sharing, privilege escalation, root contexts, and unsafe seccomp settings. A practical split is Restricted for externally facing namespaces and Baseline for internal platform namespaces with documented exceptions.

PSA is a floor, not a scalpel. It enforces at the namespace level, so it cannot say "this specific Deployment must run as UID 1000 with this seccomp profile." That is where Kyverno comes in. Its policies are plain YAML, they live in the same GitOps repo as your manifests, and they go through the same pull request review as any other change. Four rule types cover most needs:

validate rejects non-compliant resources (for example, require every image to come from an approved ECR registry).
mutate corrects omissions automatically, which turns "you forgot a security context" from an admission failure into a silent fix.
generate creates dependent resources on creation.
verifyImages enforces signature checks at admission time (more on that below).

For constraint logic that depends on the state of other cluster resources, or cross-namespace rules, OPA Gatekeeper and Rego express things Kyverno's YAML cannot. Running both is a reasonable choice: Kyverno for the common case, Gatekeeper where you need Rego's relational power.

One rollout lesson worth stating plainly: both tools have an audit mode, and you should use it first. Measure violations with enforcement off, remediate systematically, then switch to enforce. Flipping straight to enforcement is how a security rollout turns into a production incident.

Layer 2: identity, kill the static credential

A static AWS key in a ConfigMap or an environment variable is a persistent risk with a wide blast radius. IRSA (IAM Roles for Service Accounts) removes the category entirely. The AWS SDK in a pod annotated with an IRSA service account exchanges a short-lived, audience-scoped JWT for temporary IAM credentials through STS. Nothing static is created, stored, or rotated. The IAM trust policy validates the token's subject claim against a specific service account and namespace, so a compromised pod in one namespace cannot assume another namespace's role.

The payoff is blast-radius reduction you can describe concretely. Without IRSA, a compromised pod inherits whatever the node's instance profile can reach. With IRSA, it is limited to that one pod's scoped role.

Application secrets need a parallel mechanism, and that is the ExternalSecrets Operator. It syncs values from AWS Secrets Manager or Parameter Store into Kubernetes Secrets and refreshes them on the source's rotation schedule. When Secrets Manager rotates a database password, the new value propagates without a deployment. IRSA plus ExternalSecrets together removes every static credential: cloud access through IRSA, application secrets through ESO, nothing hardcoded in a Helm values file.

Scaling per-workload IAM roles across hundreds of workloads is the part teams tend to defer. Attribute-based access control with IAM permission boundaries makes it tractable: a role that expresses "this service account, in this namespace, for this environment" can be generated from workload metadata instead of hand-authored each time.

Layer 3: runtime, watch what actually happens

Admission decides what can deploy. Identity decides what it can reach. Neither sees what a pod does after it is running. Falco does, using eBPF probes at the kernel level to observe every syscall from every container on the node. A shell spawned in a container whose image has no shell in its approved process list raises an alert, regardless of how the compromise happened or whether the attacker edited the deployment manifest. Because eBPF sits below the container runtime, it keeps working even if the runtime is tampered with, and it needs no change to the image or the application.

Two practical notes. First, uniform rules across every workload generate false positives that erode trust in the alerts. A data-processing pod that legitimately reads many files will trip rules tuned for an application server. Per-workload rule sets, stored in Git next to the manifests, let you tighten baselines for payment or secret-handling services and loosen them for CI runners. Second, route Falco alerts into a SIEM so individual detections become correlated events. A shell-spawn alert followed within seconds by an outbound connection to an unknown IP is a pattern a correlation engine can act on, including automated network-policy quarantine of the affected pod.

Plan for a calibration period. Default Falco rules produce false-positive volume that takes iterative tuning, with both security and application knowledge, before the layer runs at production-grade sensitivity.

The layer most teams skip: supply chain

An image that passes admission is trusted to be what its manifest says, but only if the reference is cryptographically verified. Tags are mutable. Someone can overwrite the content behind a trusted tag and the cluster will deploy it under the same name, which bypasses every admission policy that checks the image's properties rather than its identity.

Cosign signs images at the registry, and Kyverno verifyImages rules reject anything at admission that is not signed by an approved identity. That closes the tag-mutability gap. SLSA provenance is the next step up: signing proves an image came from your build system, provenance proves what went into that build (source commit, build platform, resolved dependencies). A Software Bill of Materials attached as an attestation then lets you answer "which running images contain this newly disclosed CVE?" by querying the SBOM store instead of inspecting containers one by one.

Putting it together

The point of three independent layers is that no single failure is fatal. A policy gap that lets a bad pod through is still caught by scoped identity and runtime monitoring. A runtime compromise still cannot reach credentials the pod's IAM role never granted. A swapped image is rejected before it schedules.

If you are starting from scratch, a reasonable order is:

Turn on Pod Security Admission at Baseline or Restricted. Free, immediate.
Add Kyverno in audit mode, learn your violations, then enforce.
Move workloads to IRSA and ExternalSecrets to delete static credentials.
Deploy Falco, then budget real time to tune per-workload rules.
Add Cosign signing and verifyImages, then SLSA provenance and SBOMs.

The honest tradeoff is maintenance. Policy-as-code needs upkeep as Kubernetes APIs and workloads change, and Falco rule tuning is ongoing work. The reason to pay that cost is coverage no single control gives you: admission, identity, and runtime failures are different problems, and defending only one of them leaves the other two open.

How AWS Is Designed: Regions, Availability Zones, and Where a Workload Runs

Wed, 21 Oct 2026 09:20:05 +0000

Most AWS learning starts with picking a service. The first decision is actually where the workload runs. Region and Availability Zone choices shape availability, speed, compliance, and cost for everything built on top of them.

WHAT

AWS is organized into Regions (geographic areas), each containing multiple Availability Zones (isolated data center groups), with edge locations placed close to end users.

AWS infrastructure has three layers:

Region: a geographic area such as Mumbai, Virginia, or Frankfurt. Each Region operates independently of the others.
Availability Zone (AZ): one or more physically separate data centers inside a Region, each with its own power, cooling, and networking. AZs in the same Region are connected by fast, low-latency links (very short delay between them).
Edge location: a small site close to end users that caches content for faster delivery. Amazon CloudFront, AWS's content delivery service, uses this network.

A Region decides where a workload lives. AZs decide how well it survives failures.

WHY

Any single data center can fail, and Regions exist because distance and law both matter.

Power, network, and natural events can take a data center offline. An application running in only one location goes down with it. Because AZs are physically separated, an application spread across two or more keeps running when one fails.

Regions solve a different problem. Choosing the Region affects:

Latency (response delay): users in Singapore experience real delay reaching a server in Virginia.
Data residency: some countries and contracts require data to stay within specific borders.
Service availability: not every AWS service or feature exists in every Region.
Compliance: regulatory frameworks may restrict where data can be stored and processed.
Cost: prices for the same service differ by Region.

These choices are hard to change later, and mistakes in Region choice or redundancy often surface only during an outage or an audit.

WHEN

Choose the Region by user location and compliance, then use multiple AZs whenever downtime has a business cost.

Region and AZ decisions come before most service decisions, because they constrain what follows. Some services, such as IAM and Route 53, are global, but most workloads are tied to the Region they are built in.

Region decision order

If a legal or contractual rule fixes the location, that decides it.
Otherwise, pick the Region closest to most users.
Then confirm the required services are available there, and compare cost.

Single AZ vs. multiple AZs

A single-AZ setup is simpler and cheaper but cannot survive a data center failure, so it suits development, testing, or a proof of concept (POC, a small trial build). For anything a customer depends on, multi-AZ is the baseline expectation rather than an upgrade. The distinguishing signal: if an hour of downtime would cost money, hurt customers, or breach an agreement, the workload should span at least two AZs.

Multi-AZ vs. multi-Region

Multi-AZ protects against a data center-level failure. Multi-Region protects against a whole Region becoming unavailable, but adds significant cost and complexity. Most workloads need only multi-AZ.

HOW

Run copies of each layer in two AZs, so a failure in one is absorbed by the other.

Example scenario: a production setup in the Mumbai Region.

EC2 instances (virtual servers rented from AWS) run in two AZs, one in each.
A load balancer (a traffic distributor that sends requests only to healthy servers) sits in front and receives all user requests.
The database runs on RDS (AWS's managed database service) in Multi-AZ mode, which keeps an automatic standby copy in the second AZ.
If one AZ fails, the load balancer stops sending traffic there, and the standby database is promoted to primary.
The web layer shifts almost instantly, and the database typically recovers within a minute or two.

Flow: Users → Load Balancer → EC2 instances in AZ-a and AZ-b → RDS primary (AZ-a) with standby (AZ-b)

A multi-AZ setup in one Region. If AZ-a fails, the load balancer routes all traffic to AZ-b and the RDS standby is promoted to primary.

Where the Services Fit

AWS services are building blocks in layers, and the Region and AZ layout decides how they are placed.

Memorizing every service matters less than knowing which layer each one covers:

Compute: EC2, Lambda, ECS, EKS
Storage: S3, EBS, EFS
Database: RDS, Aurora, DynamoDB
Networking: VPC (a private network inside AWS), Route 53, CloudFront
Security: IAM (access control), KMS, WAF, GuardDuty
Integration: SQS, SNS, EventBridge
Monitoring: CloudWatch, CloudTrail

A sound design answers six questions:

Where should the workload run?
How will it remain available?
How will users access it?
How will it scale?
How will it be secured?
What will it cost?

The first question shapes the other five. This is where cloud architecture moves beyond knowing individual services and starts designing solutions.

Takeaway

The one thing to remember: Choose the Region for compliance and latency, then spread across Availability Zones for resilience, because that one decision shapes every architecture choice that follows.

AWS PyTorch: Build, Train, and Deploy Deep Learning Models in the Cloud

Wed, 21 Oct 2026 06:46:29 +0000

title: "AWS PyTorch: Build, Train, and Deploy Deep Learning Models in the Cloud"
published: true
description: "Learn how to build a PyTorch image classifier, store model artifacts in Amazon S3, and move toward managed machine learning training and deployment with Amazon SageMaker AI."

tags: aws, pytorch, machinelearning, python

PyTorch makes it easier to experiment with neural networks, computer vision, natural language processing, and deep learning.

But building a model is only one part of a machine learning project.

When it's time to move from experimentation to production, you need reliable compute, data storage, repeatable training, model artifacts, deployment, monitoring, and cost controls.

This is where AWS can help.

By combining PyTorch, Amazon SageMaker AI, Amazon S3, and Amazon CloudWatch, developers can build machine learning workflows that move from local experiments toward scalable cloud deployments.

In this article, we'll build a simple image classifier and explore how to extend the project into an AWS-based machine learning pipeline.

1. What Is PyTorch on AWS?

PyTorch is an open-source deep learning framework that provides tensor operations, automatic differentiation, GPU acceleration, and tools for creating neural networks.

AWS provides several services that can support PyTorch workloads:

Amazon SageMaker AI: Run managed training jobs and deploy machine learning models.
Amazon S3: Store datasets, model artifacts, and training outputs.
Amazon EC2: Run PyTorch on virtual machines, including suitable GPU instances.
AWS Deep Learning Containers: Use supported, preconfigured machine learning environments.
Amazon CloudWatch: Monitor supported metrics, logs, and application behavior.
AWS IAM: Control access to training data, infrastructure, and model artifacts.

You don't need all these services for every project. Start with what your workload requires and add complexity when it provides a clear benefit.

2. Understand the Architecture

A typical workflow looks like this:

        Training Dataset
               |
               v
         Amazon S3
               |
               v
      Amazon SageMaker AI
          Training Job
               |
               v
         PyTorch Model
               |
               v
       Model Artifacts in S3
               |
               v
       Deployment / Inference
               |
               v
       Application Predictions

Amazon S3 provides durable storage for datasets and artifacts. SageMaker AI can run managed training jobs, while a suitable inference service can expose the trained model to applications.

CloudWatch and related AWS monitoring capabilities help you investigate failures, latency, and resource usage.

For small experiments, you can train locally first. For larger workloads, managed training can reduce infrastructure-management overhead.

3. Hands-On Project: Build a Digit Classifier

We'll train a small neural network on the MNIST handwritten-digit dataset.

Our model will learn to classify grayscale images into one of ten categories, representing the digits from 0 to 9.

Prerequisites

You'll need:

Python 3.
Basic Python knowledge.
An internet connection to download the dataset.
An AWS account only when you move to the cloud steps.

The first part runs locally and does not require AWS resources.

Step 1: Create a project environment

Create a project directory and enter it:

mkdir aws-pytorch-demo
cd aws-pytorch-demo
python -m venv .venv

Activate the virtual environment on Windows:

.venv\Scripts\Activate.ps1

On Linux or macOS:

source .venv/bin/activate

Install the dependencies:

pip install torch torchvision

For GPU training, install a PyTorch build compatible with your GPU and CUDA configuration. Follow the official installation instructions: https://pytorch.org/get-started/locally/.

Step 2: Create the training script

Create train.py:

import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

print(f"Training device: {device}")

transform = transforms.ToTensor()

train_data = datasets.MNIST(
    root="data",
    train=True,
    download=True,
    transform=transform,
)

test_data = datasets.MNIST(
    root="data",
    train=False,
    download=True,
    transform=transform,
)

train_loader = DataLoader(
    train_data,
    batch_size=64,
    shuffle=True,
)

test_loader = DataLoader(
    test_data,
    batch_size=256,
    shuffle=False,
)


class DigitClassifier(nn.Module):
    def __init__(self):
        super().__init__()

        self.network = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28 * 28, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
        )

    def forward(self, x):
        return self.network(x)


model = DigitClassifier().to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(
    model.parameters(),
    lr=0.001,
)

# Training loop
for epoch in range(3):
    model.train()
    total_loss = 0.0

    for images, labels in train_loader:
        images = images.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()

        outputs = model(images)
        loss = criterion(outputs, labels)

        loss.backward()
        optimizer.step()

        total_loss += loss.item()

    average_loss = total_loss / len(train_loader)

    print(
        f"Epoch {epoch + 1}/3, "
        f"Loss: {average_loss:.4f}"
    )

# Evaluation
model.eval()
correct = 0
total = 0

with torch.no_grad():
    for images, labels in test_loader:
        images = images.to(device)
        labels = labels.to(device)

        outputs = model(images)
        predictions = outputs.argmax(dim=1)

        correct += (predictions == labels).sum().item()
        total += labels.size(0)

accuracy = 100 * correct / total
print(f"Test accuracy: {accuracy:.2f}%")

# Save model parameters
torch.save(
    model.state_dict(),
    "digit_classifier.pth",
)

print("Saved digit_classifier.pth")

Step 3: Train the model

Run:

python train.py

The script downloads MNIST, trains the neural network, evaluates it against the test dataset, and saves the learned parameters.

Your output should include:

Training device: cpu
Epoch 1/3, Loss: ...
Epoch 2/3, Loss: ...
Epoch 3/3, Loss: ...
Test accuracy: ...
Saved digit_classifier.pth

The loss and accuracy values depend on the training run. If a compatible GPU is available, the script can use it automatically.

Step 4: Make a prediction

Create predict.py:

import torch
from torch import nn
from torchvision import datasets, transforms


class DigitClassifier(nn.Module):
    def __init__(self):
        super().__init__()

        self.network = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28 * 28, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
        )

    def forward(self, x):
        return self.network(x)


device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

model = DigitClassifier().to(device)

state = torch.load(
    "digit_classifier.pth",
    map_location=device,
    weights_only=True,
)

model.load_state_dict(state)
model.eval()

dataset = datasets.MNIST(
    root="data",
    train=False,
    download=True,
    transform=transforms.ToTensor(),
)

image, actual_label = dataset[0]

with torch.no_grad():
    logits = model(image.unsqueeze(0).to(device))
    predicted_label = logits.argmax(dim=1).item()

print(f"Predicted digit: {predicted_label}")
print(f"Actual digit: {actual_label}")

Run the prediction:

python predict.py

Example output:

Predicted digit: 7
Actual digit: 7

These values are illustrative. The actual result depends on your trained model.

You now have a basic end-to-end deep learning workflow: train, evaluate, save, reload, and predict.

4. Move the Project to AWS

Once the model works locally, you can move the workflow to AWS.

Step 1: Create an S3 bucket

Choose a globally unique bucket name:

aws s3 mb s3://YOUR-UNIQUE-PYTORCH-BUCKET \
  --region us-east-1

Replace the placeholder with your actual bucket name.

Upload the training script:

aws s3 cp train.py \
  s3://YOUR-UNIQUE-PYTORCH-BUCKET/code/train.py

For this example, the script downloads MNIST directly. A production pipeline should manage dataset preparation, storage, and versioning more explicitly.

Step 2: Run training with SageMaker AI

Amazon SageMaker AI supports PyTorch training jobs using supported framework environments and training configurations.

At a high level, you need to:

Configure an execution role with appropriate permissions.
Prepare your training script and dependencies.
Select a supported PyTorch environment.
Configure the training input and output locations.
Choose an appropriate instance type.
Set runtime limits and launch the job.
Review the training logs and saved artifacts.

Use the current official guide for the SDK version you install:

https://docs.aws.amazon.com/sagemaker/latest/dg/pytorch.html

SageMaker SDK interfaces evolve, so use examples matching your installed SDK version rather than mixing older estimator APIs with newer SDK workflows.

Step 3: Store model artifacts in S3

For a locally trained model, you can upload the saved parameters:

aws s3 cp digit_classifier.pth \
  s3://YOUR-UNIQUE-PYTORCH-BUCKET/models/digit_classifier.pth

For a managed training job, configure the output location so SageMaker AI saves the model artifacts automatically.

Keep a record of the dataset version, source-code version, training parameters, framework version, and evaluation metrics. This makes it easier to reproduce and audit experiments.

5. Deploy the Model for Inference

Training and inference are separate stages.

For an application to request predictions, you need a serving interface that loads the model, validates input, executes inference, and returns a response.

With Amazon SageMaker AI, you can deploy supported PyTorch models to a managed endpoint. The required model packaging and inference code depend on the selected deployment approach.

A typical inference flow is:

Load the approved model artifact.
Convert the input into the tensor format expected by the model.
Run the model in evaluation mode.
Convert the output into a response your application can consume.
Log relevant errors and operational metrics.

For the MNIST model, input should represent a grayscale image with dimensions of 28 × 28 pixels. Validate the request before running inference.

For workloads that don't need immediate responses, consider batch inference instead of keeping an online endpoint available continuously.

6. Production Best Practices

A model that works locally isn't automatically production-ready.

Cost optimization

Start with CPU training for small models when practical.
Use GPU instances when benchmarks show a meaningful benefit.
Set job time limits.
Stop endpoints and compute resources when they are no longer required.
Evaluate batch inference for offline workloads.
Include storage, logging, data transfer, and supporting services in cost estimates.

Security

Apply least-privilege IAM policies.
Keep S3 buckets private.
Encrypt sensitive datasets and artifacts.
Avoid embedding credentials in source code.
Restrict access to training data and deployed models.

Reproducibility

Track dataset versions, dependencies, training parameters, random seeds, and model artifacts. Store code in version control and preserve evaluation results for each model version.

Monitoring

Monitor training failures, compute utilization, inference latency, and application errors using supported AWS monitoring capabilities.

Also evaluate model quality over time. A service can remain operational while its predictions become less accurate because real-world data has changed.

Model evaluation

Don't rely on training loss alone.

For classification tasks, consider precision, recall, F1 score, confusion matrices, and performance on representative held-out data. Select metrics based on the real business problem.

7. Where Can You Use PyTorch on AWS?

Common applications include:

Computer vision: Image classification and defect detection.
Natural language processing: Text classification and domain-specific model fine-tuning.
Recommendation systems: Ranking and personalization.
Forecasting: Demand prediction and time-series analysis.
Generative AI: Training or fine-tuning compatible neural network models.

For large-scale distributed training, additional planning is needed for GPU memory, network communication, checkpointing, and data loading.

Final Thoughts

PyTorch provides the flexibility to build deep learning models, while AWS offers several ways to run training workloads, store model artifacts, and deploy predictions.

Start with a small experiment, validate the results, and then introduce cloud infrastructure where it provides measurable value.

The real goal is not just to train a model in the cloud. It's to create a secure, reproducible, observable machine learning workflow that can grow with your application.

Official Resources

AWS #PyTorch #MachineLearning #DeepLearning #AmazonSageMaker

A Field Guide to Beating Amazon Bedrock Throttling

Tue, 20 Oct 2026 20:53:15 +0000

🛡️ A Field Guide to Beating Amazon Bedrock Throttling

TL;DR

Know your quota first: check Service Quotas, learn which limits are adjustable (soft) and which are not (hard).

Retry with exponential backoff and full jitter on throttling errors. This is the floor, not the ceiling.

Cap concurrency so you stop generating load the quota cannot absorb.

Add a circuit breaker so a throttle storm backs off instead of hammering.

Handle timeouts separately: shrink the batch for model timeouts, tune and retry for client read timeouts.

Use cross-region inference profiles to spread token demand across regions.

Reserve provisioned throughput only when on-demand plus the above is not enough.

If you run real LLM traffic through Amazon Bedrock, throttling is not an edge case. It is the normal operating condition at scale. A ThrottlingException or TooManyRequestsException is Bedrock telling you that your request rate or token rate crossed a quota for that model in that region. The fix is rarely one setting. It is a layered defense: understand the quota, retry intelligently, cap concurrency, degrade gracefully, handle timeouts, and know what to escalate to AWS.

This is the set of strategies I run across production LLM workloads, written as a generic guide. No workload specifics, just the patterns and when each one earns its place.

Step one: know the quota you are hitting

Before writing a line of retry code, find the actual limit. Bedrock quotas are per model, per region, and split by type: tokens per minute (TPM), requests per minute (RPM), and for some models separate input/output token quotas.

Quotas for Amazon Bedrock — the overview, including that default quotas vary by account age, usage history, and region.
Quotas for the bedrock-runtime endpoint — the runtime invoke limits you hit at inference time. View them in the Service Quotas console under Amazon Bedrock.
Scaling and throughput best practices — how per-model TPM and RPM quotas interact.

Soft limits vs hard limits

This distinction decides your whole strategy.

Soft limits (adjustable). In the Amazon Bedrock endpoints and quotas table these are marked Adjustable: Yes. You can raise them. Request an increase through Service Quotas; smaller increases often auto-approve, larger ones route to AWS Support for review.
Hard limits (not adjustable). Marked Adjustable: No. No amount of asking moves these. Your only lever is architectural: reduce demand, spread it across regions, or change the access pattern.

The practical rule: if the limit is soft and you are consistently near it, file the increase and also build the defenses below, because the increase takes time and never makes throttling impossible. If the limit is hard, skip the request entirely and go straight to architecture.

One often-missed detail: Bedrock reserves TPM quota at the start of a request based on input tokens plus maxTokens. If you leave maxTokens unset, it can default to the model maximum and reserve far more quota than your call actually needs, so you throttle earlier than your real token use suggests. Set maxTokens to a realistic ceiling.

Layer one: retry with backoff and jitter

The baseline defense. Catch the throttling error codes (ThrottlingException, TooManyRequestsException), wait, and try again. Two details separate a good retry from a harmful one:

Exponential backoff. Each attempt waits longer: base * 2^attempt, capped at a ceiling. This gives the quota window time to refill.
Full jitter. Randomize the delay inside that window. Without jitter, every throttled caller retries at the same instant and you get a synchronized stampede that throttles again. With jitter, retries spread out.

A compact pattern:

import random, time
from botocore.exceptions import ClientError

THROTTLE = {"ThrottlingException", "TooManyRequestsException"}

def call_with_backoff(fn, *, base=2.0, cap=15.0, max_retries=5):
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except ClientError as exc:
            code = exc.response.get("Error", {}).get("Code", "")
            if code not in THROTTLE or attempt == max_retries:
                raise
            delay = min(base * 2 ** attempt, cap)
            time.sleep(random.uniform(0, delay))  # full jitter

A note on the boto3 client's own retries. Boto3 has a built-in retry mode (standard or adaptive) you set via botocore.config.Config(retries=...). You can lean on it, or disable it (max_attempts=0) and own the retry loop yourself. Owning it is worth the effort when you want the backoff, jitter, and error classification to match the rest of your system rather than two retry layers fighting each other.

The same retry in a Step Functions workflow

If your Bedrock calls run inside a Step Functions state machine, you do not write the backoff loop by hand. You declare it on the task's Retry block and the service applies exponential backoff with jitter for you. Step Functions error handling added a JitterStrategy field so wait times are spread instead of synchronized, the same reason full jitter matters in code.

"InvokeBedrock": {
  "Type": "Task",
  "Resource": "arn:aws:states:::bedrock:invokeModel",
  "Retry": [
    {
      "ErrorEquals": ["ThrottlingException", "TooManyRequestsException"],
      "IntervalSeconds": 2,
      "MaxAttempts": 15,
      "BackoffRate": 2.0,
      "MaxDelaySeconds": 30,
      "JitterStrategy": "FULL"
    }
  ],
  "Catch": [
    { "ErrorEquals": ["States.ALL"], "Next": "HumanReview" }
  ],
  "Next": "NextStep"
}

The fields map onto the code version: IntervalSeconds is the base delay, BackoffRate is the exponential multiplier, MaxDelaySeconds is the cap, MaxAttempts is the retry count, and JitterStrategy: FULL randomizes each wait. The Catch routes to a fallback once retries are exhausted, for example a human-review step, so an exhausted retry degrades gracefully instead of failing the whole execution.

Layer two: cap concurrency

Retries handle the request that already failed. Concurrency caps stop you from generating more load than the quota can absorb in the first place. If 50 workers all call the same model at once, you will throttle no matter how clever the backoff is.

Put a ceiling on in-flight Bedrock calls. In async Python a semaphore does it:

import asyncio

semaphore = asyncio.Semaphore(10)  # max concurrent Bedrock calls

async def bounded_call(fn):
    async with semaphore:
        return await fn()

Upstream of the application, cap concurrency at the infrastructure layer too. If documents arrive as events, put them on an SQS queue and let the SQS-to-Lambda event source mapping enforce a maximum_concurrency. Work waits in the queue until a slot frees, with no polling and no custom counters:

resource "aws_lambda_event_source_mapping" "sqs_to_worker" {
  event_source_arn = aws_sqs_queue.work.arn
  function_name    = aws_lambda_function.worker.arn
  batch_size       = 1
  scaling_config {
    maximum_concurrency = 5
  }
}

Match the concurrency ceiling to the quota, not to how many workers you could theoretically run. The right number is the one that keeps steady-state token rate under the TPM limit.

Layer three: a circuit breaker for throttle storms

Backoff and concurrency caps handle normal pressure. A sustained throttle storm needs something that steps back harder. A circuit breaker watches the throttle rate and, once it crosses a threshold inside a time window, pauses all calls for a cooldown, then resumes at reduced concurrency before restoring full throughput.

The state machine:

Closed (normal): calls flow, throttles are counted.
Open (tripped): after N throttles in the window, pause all calls for a cooldown and halve the concurrency limit.
Half-open (recovering): after the cooldown, resume at the reduced limit; if clean for a while, restore full concurrency; if it throttles again, reopen.

This converts a feedback loop that makes throttling worse (retry, throttle, retry, throttle) into one that actively sheds load until Bedrock recovers. It pairs with the concurrency limiter: the breaker is what turns the limit down and back up.

Layer four: handle timeouts, they are not throttling

Throttling rarely travels alone, and the loudest neighbor is the timeout. There are two distinct kinds, and they want opposite fixes.

Model timeout: shrink the work

A ModelTimeoutException means the request itself was too large to finish in the model's time budget. It is not a rate problem, so retrying the same payload will time it out again. The fix is to reduce the work: halve the batch or chunk the input, then retry smaller.

async def call_shrinking(fn, items, *, max_retries=3):
    batch = list(items)
    for attempt in range(max_retries + 1):
        try:
            return await fn(batch)
        except Exception as exc:
            code = getattr(exc, "response", {}).get("Error", {}).get("Code", "")
            if code != "ModelTimeoutException" or attempt == max_retries:
                raise
            batch = batch[: max(1, len(batch) // 2)]  # halve and retry

Client timeout: tune the budget, then retry the transient ones

A client-side ReadTimeoutError or ConnectTimeoutError is different again: the SDK gave up waiting before the model answered. Large prompts and long generations routinely exceed the boto3 default read timeout, which produces timeouts that look like failures but are really an impatient client.

Two moves:

Set a realistic read_timeout on the boto3 client so a slow-but-valid response is not cut off.
Treat the transient connection timeouts as retryable, alongside throttling, rather than letting them bubble up as hard failures.

from botocore.config import Config

client = boto3.client(
    "bedrock-runtime",
    config=Config(read_timeout=300, retries={"max_attempts": 0}),
)

RETRYABLE = {"ThrottlingException", "TooManyRequestsException"}
# also retry ReadTimeoutError / ConnectTimeoutError by exception class name

Classify before reacting. A throttle wants backoff, a model timeout wants a smaller batch, a client timeout wants a longer budget plus a retry, and a ValidationException wants no retry at all.

Layer five: spread demand with cross-region inference profiles

Quotas are per region. A cross-region inference profile lets a single on-demand call be served from one of several regions, so your token demand is spread across each region's quota instead of piling onto one. For on-demand workloads this is often the highest-leverage change: no capacity to buy, and the effective ceiling goes up because you are drawing from multiple regional pools.

It is not infinite headroom, and it does not change a hard per-model limit in a single region, but for spiky on-demand traffic it buys real room.

Layer six: provisioned throughput, only when you need it

If on-demand plus everything above still throttles under predictable heavy load, Bedrock offers Provisioned Throughput: reserved capacity (model units) with a guaranteed token rate, bought by commitment. It removes throttling for that reserved slice because the capacity is yours.

Treat it as the last layer, not the first. It costs whether or not you use it, so it only makes sense for steady, predictable volume. For bursty or unpredictable traffic, retries, concurrency caps, and cross-region profiles on on-demand are usually the better economics.

Tuning in practice

The layers above are the toolbox. How hard you turn each knob depends on the workload, and a couple of patterns are worth stating plainly.

Match the aggressiveness of retries to the cost of a lost call. Heavy, expensive calls justify more retry attempts, a higher backoff ceiling, and tighter concurrency, because losing one is costly. High-volume, cheaper work is better served by an upstream queue that caps concurrency and a workflow-level self-retry that ends in a manual-review fallback, keeping the per-call retries light. There is no single correct number; the right setting is the one that keeps steady-state token rate under the quota for that workload.

Two caveats that are easy to get wrong:

A "fallback model" can quietly resolve to the same model. If both the primary and fallback point at the same model id through shared config, the fallback functions as extra retries against one model, not as model diversity. Calling it a fallback overstates what it does until the config points at a genuinely different model.
Retry, backoff, and concurrency are not the same as reserved capacity. A defense built entirely from retries, backoff, concurrency caps, and on-demand cross-region inference profiles has no reserved-capacity floor under a true spike. That can be a deliberate on-demand posture, but name it as one, because on-demand plus retries still has a ceiling.

How to decide what to reach for

A short decision order when you see throttling:

Check the quota. Is the limit soft or hard? If soft and you are near it, file the increase now, in parallel with the rest.
Confirm retries have backoff and jitter. If not, fix that first. It is the cheapest win.
Cap concurrency to keep steady-state rate under the TPM limit, at both the app and the infrastructure layer.
Add a circuit breaker if throttle storms are recurring rather than occasional.
Separate timeouts from throttles. Shrink the batch for model timeouts, tune the read timeout and retry for client timeouts.
Spread across regions with a cross-region inference profile for on-demand traffic.
Buy provisioned throughput only for steady, predictable, high volume the above cannot cover.

Throttling is not a wall you hit once and clear. It is a pressure you manage continuously. The systems that stay up are the ones that treat the quota as a first-class design constraint, not an exception to catch.

The Bottleneck Was Not the Model. It Was the Prompt Lookup.

Sun, 18 Oct 2026 23:08:00 +0000

🔑 The Bottleneck Was Not the Model. It Was the Prompt Lookup.

TL;DR

A managed prompt-management service was throttling prompt retrieval on the hot path, which capped the whole pipeline.

The model was never the bottleneck. Fetching the prompt was.

Move prompt serving into DynamoDB: a single keyed read, no shared service quota.

Keep the managed service as the authoring source of truth, and sync into DynamoDB write-through on a checksum, so only real changes get written.

DynamoDB can do versioning too: a sort-key version plus a current pointer buys rollback and audit while keeping the hot path a single keyed read.

Prompts feel like configuration, so it is tempting to park them in whatever managed prompt-management service your platform hands you. You get versioning, a console, and a clean API, and for a while that is the right call. The trouble shows up later, under load, when prompt retrieval sits on the hot path of every inference call and the service starts pushing back.

That is the wall I hit. The prompt-management layer was throttling requests. Because every call had to fetch its prompt before doing any real work, that throttle became a ceiling on the whole pipeline. The model was not the bottleneck. Prompt lookup was.

I moved prompt serving into DynamoDB and kept the managed service as the authoring source of truth. The throttling went away, and the design has been running well since.

Why a prompt-management service becomes the bottleneck

Prompt-management services are built for authoring and governance, not for being read thousands of times a minute on a latency-sensitive path. They come with request quotas, and those quotas are usually shared across the account rather than scaled to one workload's read traffic. When read volume climbs, you hit the quota long before you hit any limit that has to do with the prompts themselves.

The tell is subtle at first. You start seeing throttling and too-many-requests errors on prompt retrieval, so you add retries with exponential backoff and maybe a fallback for when the service says no. That is reasonable hygiene, but once you are wrapping a plain config lookup in retry logic, the store has quietly become the throughput limit of the application. You can optimize the model, batch the work, and tune everything downstream, and none of it matters, because the request is already stalled waiting for its prompt.

Why DynamoDB fits this access pattern

A prompt lookup is about the simplest access pattern there is. You have a key, and you want the item back. That is a single-item read by primary key, which is exactly what DynamoDB is good at.

Read throughput scales with the table, not a fixed service quota. On-demand capacity absorbs spikes and provisioned capacity with auto scaling handles steady load. The shared quota that was throttling me no longer sat on the path.
The access pattern is a key lookup. Partition on the prompt identifier and every fetch is a single GetItem. No scans, no filtering, predictable single-digit-millisecond reads.
Reads and writes separate cleanly. High-volume reads hit DynamoDB. The rare authoring writes stay in the managed service where review and governance already live, and a sync step pushes changes down into DynamoDB.

The part that makes it safe: checksum-based sync

Caching prompts is easy. Keeping the cache honest is the part worth getting right.

The managed service stays the source of truth for prompt content. DynamoDB holds the serving copy that the hot path reads. To keep them in agreement without re-uploading everything on every deploy, I sync on a checksum.

The sync is write-through. When it detects a change at the source, it writes the new prompt into DynamoDB first, and readers pick it up from there afterward. The flow is simple:

For each prompt in the source of truth, compute a checksum of its content.
Compare that checksum against what is already stored for that prompt.
If they match, do nothing. If they differ, the prompt changed, so write the new content into DynamoDB and record the new checksum.

This turns every sync into a cheap compare instead of a blind rewrite. Unchanged prompts cost a comparison and no write. Only a genuine edit at the source triggers a write. Because the write lands before any read needs it, a new prompt propagates cleanly: the serving copy is already updated by the time the hot path asks for it.

The nice property is that the checksum is the change signal. You do not need a separate "is this dirty" flag or a manual version bump. If the content is identical, the checksum is identical, and the write is skipped. If someone edits a prompt, the checksum moves and the new text is written on the next sync.

If you need rollback or audit: versioning in DynamoDB

DynamoDB does not hand you versioning the way a prompt-management service does, but you can model it, and it is worth saying so plainly because "DynamoDB has no versioning" is a common misread.

If a single current copy per prompt is all you need, the checksum layout above is enough. If you want history, rollback, or an audit trail, add a version to the key:

Partition key: the prompt identifier.
Sort key: the version. A monotonic integer, an ISO timestamp, or the checksum itself all work.
A current pointer: one extra item per prompt (for example sort key CURRENT) that names the active version.

The hot path reads the current pointer by key, so serving stays a single keyed read. History stays queryable with a Query on the partition key. The checksum still decides when a new version row gets written, so you are not stacking a new version for an unchanged prompt. If concurrent syncs are possible, a conditional write (ConditionExpression on the stored checksum) keeps a stale sync from overwriting a newer one.

The tradeoff is simple: the single-copy layout is less to manage, the versioned layout buys rollback and audit. Both are key reads on the hot path, so neither reintroduces the throttling problem.

The shape of the change

The migration itself was small, which is part of why it was the right call.

Model the table around the lookup. Partition key on the prompt identifier. That is the whole schema, because the access pattern was known up front.
Read by key on the hot path. Each inference call does a single keyed read. No scan runs against this table in production, so cost stays proportional to the reads you actually make.
Sync by checksum, not on a timer or on every deploy. The source of truth drives updates, and only real changes cross into DynamoDB.

The result is that prompt retrieval stopped being a shared, rate-limited dependency and became a key read against a store that scales with the traffic hitting it.

What I would tell someone before they copy this

This is not an argument against prompt-management services. They earn their place for authoring, review, and governance. The mistake is letting that same service sit on a high-volume read path it was never sized for.

A few things worth keeping in mind:

The symptom is a throughput ceiling that tracks request volume, not model latency. If you are wrapping prompt retrieval in throttling retries, that is the tell. No downstream tuning will move it.
Design the table around the access pattern first. A key lookup wants a key schema. Pick the partition key from the query you actually run, and keep scans out of production.
Make the cache self-correcting. A checksum compare against the source of truth means the write-through sync updates the serving copy only when content actually changes, without manual version juggling and without rewriting unchanged data.
Model versioning if you need it, do not assume it is missing. A sort-key version plus a current pointer gives you rollback and audit while keeping the hot path a single keyed read.

Moving prompts into DynamoDB was a small change with a disproportionate payoff. The throttling that was capping the pipeline is gone, retrieval is a clean key lookup, and the checksum sync keeps the serving copy honest against the source of truth. It has held up in production since.

Leveraging AWS CodePipeline pipeline and Serverless Framework to deploy a Flask API.

Wed, 14 Oct 2026 18:02:28 +0000

For the past couple of days. I have been working with the Serverless framework. I came to know that you can actually host your flask API that integrates several AWS services such as Lambda, API Gateway, and DynamoDB using this framework. I had to spend a lot of time to understand the process at first since there is not a lot of content regarding this implementation. So, here’s me contributing a little.

To make this swift, I wanted to integrate the entire process with a pipeline that contains the code and a build environment. Here’s how everything is carried out. Please understand that this is not a beginner level tutorial and the whole process is written in assumption that the readers have knowledge and experience with AWS services and Python Flask.

First, you need to have a flask api. An API that works. I couldn’t share the actual code that I first used since it is confidential . So, I have decided to go with a template of serverless framework. You can also convert your flask api to be compatible with serverless using the serverless-wsgi plugin. For this, you may need to make some changes to your serverless.yaml configurations. You can check it here to achieve that

Now that you have your api , lets upload it again to an aws codecommit repository. You can clone an empty repo, add the files, commit and push. Check it here.

Next, we need to create a build project in AWS CodeBuild.

In this codebuild project, use the following buildspec.yaml file. You can choose an Ubuntu environment for the build project to run.

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: 14
    commands:
        - npm install -g serverless
        - npm install serverless-compose
        - serverless --version
        - npm install serverless-deployment-bucket
        - sls plugin install -n serverless-wsgi
        - sls plugin install -n serverless-python-requirements

  build: 
    commands: 
      - serverless deploy

Remember: In this case, Codebuild is only used to run the deploy command. All the resources are built by serverless by automatically running a cloudformation.

Now, we’re also gonna need another yaml file. This one is the configuration file for the serverless framework which will be deployed once we deploy it using the Aws pipeline.

Here’s the configuration code.

service: new12

frameworkVersion: '3'

custom:
  wsgi:
    app: app.hello

provider:
  name: aws
  runtime: python3.9

functions:
  api:
    handler: wsgi_handler.handler
    events:
      - httpApi: '*'

plugins:
  - serverless-wsgi
  - serverless-python-requirements

Also keep your flask api file in the same directory. Make sure to configure codebuild to look for the specific branch in the specific repo.

Once everything is created, we need to make sure that our Codebuild service role has enough permission to use cloudformation and all the other services associated with the resources built through the cloudformation stack.

In my case, I faced a lot of failed builds due to permission issues.Always make sure to check your build logs when your build fails, so that you can get a clear understanding of the underlying reasons and issues.

So, to make it quick I added some permission to the codebuild service role. I suggest to create your own permission policies based on the least privilege access policy rather than assigning managed policies with Full access.

Once the permissions are set, your pipeline will run swiftly. Here's how my pipeline looks like with the two phases.

Here are the outputs from the cloudformation stack.

You invoke URL will be generated in Amazon API Gateway. Here in the build logs, you can see an invoke URL for the api.

The serverless Framework will also help you monitor your resources and their performance through the console. All you need to do is login to the console and connect to your aws account. The console will run another cloudformation stack to integrate with the resources in our aws account. Here is how the dashboard looks like.

Thank you for making it to the end, hope You found what you were looking for!

AWS Step Functions: Creating a ‘Busy Waiting’ flow to wait for successful lambda executions.

Wed, 14 Oct 2026 18:01:55 +0000

Hi everyone, welcome to this article. In the past few days, I had to create a logic in AWS Step functions which involved waiting for a lambda that queries athena before going forward in the flow.

When I was discussing the logic, it felt like an easy thing to accomplish in minutes but I was surprised upon knowing that Step Functions does not have its native technique/solve to wait for a state to succeed before starting another. This made me search for hours before coming up with a solution.

Here, I will share with you a technique which will help you in the future.

First of all, I will create a simple lambda function. All we need to know from the lambda is whether it did its job. So, this simple function will make sure to return its status.

Lambda function:

import json

def lambda_handler(event, context):
    response = {
        'status': 'success',
        'message': 'Hello, World!'
    }
    
    return response

You can copy both the function and the Step function definition which I will share below to replicate all of this in your own environment.

Now, lets go ahead and create a state machine in AWS Step Functions.

Click on ‘Create state machine’.

Choose the option to write your own definition and choose ‘standard’.

Now, in the definition section, copy and paste the following definition.

Step Functions definition:

{
  "Comment": "A Hello World example of the Amazon States Language using Pass states",
  "StartAt": "Main",
  "States": {
    "Main": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "OutputPath": "$.Payload",
      "Parameters": {
        "Payload.$": "$",
        "FunctionName": "arn:aws:lambda:us-east-1:031342435657:function:stepfunctionlambda_test:$LATEST"
      },
      "Retry": [
        {
          "ErrorEquals": [
            "Lambda.ServiceException",
            "Lambda.AWSLambdaException",
            "Lambda.SdkClientException",
            "Lambda.TooManyRequestsException"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Next": "WaitforInvocation"
    },
    "WaitforInvocation": {
      "Type": "Wait",
      "Seconds": 2,
      "Next": "CheckInvocationsStatus"
    },
    "CheckInvocationsStatus": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.status",
          "StringEquals": "success",
          "Next": "Final"
        }
      ],
      "Default": "WaitforInvocation"
    },
    "Final": {
      "Type": "Succeed"
    }
  }
}

Save the definition and click next.

Specify the name of your state machine and then leave everything else as default. A new execution role will be created if you don’t specify any.

Save it and then you will have created a state machine which looks like this:

Visual flow:

To explain the definition above, I have pointed out how this state machine works:

The state machine starts at the "Main" state.
The "Main" state is a Task state that invokes a Lambda function specified by the "FunctionName" parameter. The input to the Lambda function is the entire input passed to the state machine.
The "OutputPath" parameter is set to "$.Payload" to capture the output of the Lambda function.
If the Lambda function invocation fails with certain exceptions specified in the "Retry" section (Lambda.ServiceException, Lambda.AWSLambdaException, Lambda.SdkClientException, Lambda.TooManyRequestsException), the state machine will retry the invocation up to 6 times with a backoff interval of 2 seconds.
After invoking the Lambda function, the state machine proceeds to the "WaitforInvocation" state.
The "WaitforInvocation" state is a Wait state that pauses the execution for 2 seconds.
After the wait period, the state machine goes to the "CheckInvocationsStatus" state.
The "CheckInvocationsStatus" state is a Choice state that checks the value of the "$.status" field in the output of the Lambda function. If the status is "success", the state machine proceeds to the "Final" state. Otherwise, it goes back to the "WaitforInvocation" state.
The "Final" state is a Succeed state that indicates the successful completion of the state machine.

Now, lets go ahead and execute it!

Click on ‘Start execution’ and leave the input field empty.

Incase of a lambda that requires input, you can insert your input here as a Json

After you start your execution, each state will turn into different colours which indicates the following.

Green: Indicates successful completion of a state or step.
Red: Indicates a failure or error occurred during the execution of a state or step.
Yellow: Indicates a state is in a waiting or paused state.
Blue: Represents a decision or choice point where the state machine evaluates conditions to determine the next state.
Gray: Represents a passive state that simply passes input to output without performing significant work.
White: Represents transitions or connections between states, indicating the flow of execution from one state to another.

Finally this is how the execution will look like:

Thank you. Hope you can now go ahead and apply the “Busy Waiting” logic in your step functions.

How I Built and Deployed a Production ready MERN Application on AWS using Modular Terraform and Keyless GitHub Actions OIDC

Sat, 10 Oct 2026 20:34:17 +0000

Deploying a full-stack web application into production is often one of the most daunting steps for modern developers. You build a complete application locally, but when it comes to hosting it securely on cloud infrastructure with HTTPS, custom domains, and automated deployment pipelines, things can quickly get overwhelming.

In this post, I will walk you through how I built and deployed an end-to-end full-stack MERN application ("Little List")—from the frontend user interface and Express REST API to provisioning production AWS cloud infrastructure with Terraform and automating deployments with Keyless GitHub Actions OIDC.

Whether you are a beginner looking to understand full-stack architecture & cloud deployment or an experienced engineer reviewing multi-environment Terraform design, this guide covers the entire end-to-end workflow!

What is "Little List"? (Application Overview)

"Little List" is a full-stack personal productivity and management application built from scratch:

React SPA Frontend (Vite): Modern, responsive UI with state management, JWT auth flows, interactive diary logging, and custom URL shortener interfaces.
Node.js & Express API Backend: Modular REST API with secure authentication, refresh tokens, MongoDB integration, and route validation.
MongoDB Database: Persistent storage for users, diary entries, and shortened link analytics.

Prerequisites

Before diving in, here is what you will need if you want to replicate this deployment architecture:

AWS Account: An active AWS account with permissions to manage EC2, S3, CloudFront, ECR, IAM, and Route 53.
Terraform CLI: Installed on your machine (>= 1.16.2).
AWS CLI: Installed and configured with your credentials (aws configure).
Docker: Installed locally for testing container builds.
Node.js & npm: Installed locally for building the frontend.
GitHub Account: A repository containing your MERN codebase.
Custom Domain (Optional): A domain registered on Namecheap, GoDaddy, or Route 53 if you want custom SSL support (e.g., ideategudy.tech).

️ The Cloud Architecture Overview

Before writing code, let us look at the high-level architecture diagram of what we are provisioning on AWS:

                               ┌──────────────────────┐
                               │    GitHub Actions    │
                               └──────────┬───────────┘
                                          │
                        ┌─────────────────┴─────────────────┐
                        │                                   │
                  Frontend deploy                      Backend deploy
                        │                                   │
                        ▼                                   ▼
                ┌───────────────┐                    ┌───────────────┐
                │      S3       │                    │      ECR      │
                │ React build   │                    │ Docker image  │
                └───────┬───────┘                    └───────┬───────┘
                        │                                    │
                        │                                    │ docker pull
                        │                                    ▼
                        │                            ┌───────────────┐
                        │                            │      EC2      │
                        │                            │ Express/Docker│
                        │                            └───────┬───────┘
                        │                                    │
                        │                            ┌───────┴───────┐
                        │                            │      ASG      │
                        │                            │  EC2 fleet    │
                        │                            └───────┬───────┘
                        │                                    │
                        ▼                            ┌───────▼───────┐
                ┌───────────────┐                    │      ALB      │
                │  Route 53 DNS │                    │ Load Balancer │
                └───────┬───────┘                    └───────▲───────┘
                        │                                    │
                        ▼                                    │
                ┌───────────────┐                            │
                │  CloudFront   │────────────────────────────┘
                │  CDN (HTTPS)  │ Path-based routing (/api/*)
                └───────────────┘

Core Components Explained for Beginners:

AWS Route 53: Highly available DNS service that manages custom domain records (dev.ideategudy.tech) and routes client requests directly to CloudFront via A-record Aliases.
Amazon S3: Hosts the static, compiled single-page React frontend (dist/ build files) privately.
Amazon CloudFront: A global Content Delivery Network (CDN) that serves the React app over HTTPS and acts as a single reverse proxy for both client and backend requests.
Amazon ECR (Elastic Container Registry): Private Docker image registry to store backend container builds.
AWS ALB (Application Load Balancer): Receives API traffic from CloudFront and balances requests across EC2 instances.
AWS Auto Scaling Group (ASG): Manages an auto-healing fleet of EC2 instances running Canonical Ubuntu 24.04 LTS and Docker.
GitHub OIDC (OpenID Connect): Allows GitHub Actions to obtain short-lived security tokens to deploy to AWS without storing permanent AWS Secret Access Keys in GitHub.

Repository & Project Structure

Here is how the project files are structured:

.
├── client/                     # Vite + React SPA Frontend
├── server/                     # Node.js + Express API Backend (Dockerized)
├── infra/                      # Modular Terraform Infrastructure
│   ├── main.tf                 # Root Terraform Module
│   ├── variables.tf            # Root Variables & Configuration Defaults
│   ├── outputs.tf              # Root Terraform Outputs
│   ├── versions.tf             # AWS & Provider Requirements
│   ├── dev.tfvars.example      # Development Environment Settings
│   ├── prod.tfvars.example     # Production Environment Settings
│   └── modules/
│       ├── vpc/                # VPC, Public Subnets (2 AZs), IGW, Route Tables
│       ├── backend/            # ECR, ALB, ASG, Launch Template, EC2 IAM Role
│       ├── frontend/           # S3 Bucket, CloudFront OAC, CDN Distribution, Route53/ACM
│       └── github_oidc/        # GitHub OIDC Identity Provider & IAM Role
└── .github/
    └── workflows/
        └── deploy.yml          # Parallel CI/CD Workflow (Frontend & Backend)

Key Technical Deep Dives

1. Unified Single-Domain Routing with CloudFront

One common challenge with SPA + API setups is CORS and handling multiple URLs (api.domain.com vs domain.com). We solved this by serving both frontend and backend under the exact same domain (https://dev.ideategudy.tech) via CloudFront Path-Based Routing:

Request Path Pattern	Origin Target	Description
`/` or `/assets/*`	`s3-frontend`	Serves static React SPA files
`/api/`, `/s/`, `/healthz`	`alb-api`	Forwards API requests directly to the EC2/Express backend

Pro-Tip: When forwarding requests through CloudFront to an ALB, ensure your CloudFront cache behavior explicitly forwards the Authorization, Accept, Content-Type, Origin, and Referer headers! Otherwise, CloudFront strips the Authorization: Bearer <jwt> header by default, leading to 401 Unauthorized errors on authenticated routes.

2. Zero Secrets in GitHub via AWS OIDC

Instead of creating long-lived IAM user keys (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) and pasting them into GitHub Secrets, we configured an AWS IAM OIDC Identity Provider.

GitHub Actions assumes a temporal AWS IAM Role dynamically per pipeline run using OpenID Connect authentication. If your pipeline is compromised, there are no static credentials to leak!

Custom Domain, Subdomain Setup (`dev.ideategudy.tech`) & SSL Configuration

Primary infrastructure resources (VPC, EC2, ASG, ALB, ECR, S3) are deployed in eu-north-1 (Stockholm). HTTPS is terminated at the CloudFront CDN level using AWS Certificate Manager (ACM) in us-east-1 (required by CloudFront for global edge distributions).

Why Use a Subdomain (`dev.ideategudy.tech`)?

Instead of pointing the root/apex domain (ideategudy.tech) directly to this environment, we configured a dedicated subdomain (dev.ideategudy.tech) for several key reasons:

Environment Isolation (dev vs prod): Using subdomains allows us to run isolated development (dev.ideategudy.tech) and production (ideategudy.tech or app.ideategudy.tech) infrastructure in separate Terraform workspaces without DNS conflicts.
Subdomain NS Delegation in Namecheap: By adding custom NS records for dev.ideategudy.tech in Namecheap, we delegate only the dev subdomain DNS resolution to AWS Route 53 while keeping the parent apex domain (ideategudy.tech) managed at Namecheap.
Multi-Environment Testing: It enables continuous testing of AWS infrastructure changes (CloudFront, ALB, ASG) on a real HTTPS domain without risking downtime for main production domain traffic.

Connecting Namecheap Subdomain (`dev.ideategudy.tech`) to AWS:

In infra/dev.tfvars:

domain_name         = "dev.ideategudy.tech"
create_route53_zone = true

Run terraform apply --var-file="dev.tfvars".

Copy the 4 output nameservers from Terraform:

route53_nameservers = [
  "ns-1143.awsdns-14.org",
  "ns-1698.awsdns-20.co.uk",
  "ns-263.awsdns-32.com",
  "ns-847.awsdns-41.net"
]

Log into Namecheap -> Domain List -> Manage ideategudy.tech -> Advanced DNS -> Add 4 NS Records for Host dev pointing to the Route 53 Nameservers.

Figure: Configured NS records in Namecheap delegating dev.ideategudy.tech to AWS Route 53.

In minutes, free auto-renewing SSL is active across https://dev.ideategudy.tech!

⚡ Automated CI/CD Pipeline (`deploy.yml`)

When code is pushed to main, GitHub Actions triggers .github/workflows/deploy.yml which executes two parallel jobs:

Note for Beginners: You do not need to manually build the React frontend or Docker container on your local computer before pushing. GitHub Actions automatically compiles the React production bundle and builds/pushes the Docker image in the cloud during the pipeline run!

deploy-frontend:
- Builds Vite/React bundle.
- Authenticates keylessly to AWS via OIDC.
- Syncs static assets to the private S3 bucket (aws s3 sync).
- Invalidates CloudFront edge cache (aws cloudfront create-invalidation).
deploy-backend:
- Builds Docker image for Express API.
- Authenticates to Amazon ECR via OIDC.
- Pushes new Docker tag to ECR.
- Syncs runtime application secrets into AWS SSM Parameter Store (/mern-deploy/dev/...).
- Executes zero-downtime container updates across EC2 ASG instances via AWS SSM RunShellScript.

Step-by-Step Deployment Instructions

Step 1: Provision AWS Infrastructure with Terraform

cd infra

# Copy example variables
cp dev.tfvars.example dev.tfvars

# Edit values in dev.tfvars (domain_name, github_org, github_repo)
terraform init
terraform plan --var-file="dev.tfvars"
terraform apply --var-file="dev.tfvars" --auto-approve

Figure 1: Successful terraform apply showing outputs for S3, CloudFront, ECR, and ALB.

Step 2: Configure GitHub Repository Secrets & Variables

In your GitHub repo under Settings -> Secrets and variables -> Actions:

Tip: You can retrieve all your Terraform output values anytime by running terraform output inside the infra/ folder.

Figure 2: GitHub Actions repository variables and secrets configured for OIDC deployment.

Secrets:
- MONGODB_URI: MongoDB connection string
- ACCESS_TOKEN_SECRET: JWT access secret
- REFRESH_TOKEN_SECRET: JWT refresh secret
Variables:
- AWS_ROLE_ARN: Output github_actions_role_arn
- AWS_REGION: eu-north-1
- ECR_REPOSITORY: Output ecr_repository_name
- FRONTEND_BUCKET: Output frontend_bucket
- CLOUDFRONT_DISTRIBUTION_ID: Output cloudfront_distribution_id
- API_ASG_NAME: Output api_asg_name
- SSM_PREFIX: Output ssm_prefix
- CLIENT_URL: https://dev.ideategudy.tech
- SERVER_URL: https://dev.ideategudy.tech

Step 3: Trigger Automated Build & Deployment

Push your updates to the repository:

git add .
git commit -m "Deploy production architecture to AWS"
git push origin main

Figure 3: Parallel CI/CD execution for deploy-frontend and deploy-backend jobs.

Live Application Demo

Figure 4: The live full-stack application running on https://dev.ideategudy.tech served securely over CloudFront HTTPS.

⚡ Key Benefits & Architectural Trade-Offs

Key Benefits

Choosing this cloud setup over monolithic single-server hosting provides several high-value production advantages:

Zero-Downtime & Ultra-Fast Global Delivery:
By hosting the React SPA on Amazon S3 + CloudFront, static assets are cached globally at edge locations close to users. CloudFront delivers lightning-fast page loads while shielding your backend EC2 instances from static file requests.
Extreme Cost Optimization (Free Tier Friendly):
- S3 & CloudFront: Nearly zero cost for static SPA bandwidth.
- EC2 Auto Scaling: Scales t3.micro instances based on CPU utilization and traffic demands without paying for oversized idle instances.
- Amazon ECR Lifecycle Policies: Configured to auto-expire old image tags, keeping container storage well under free-tier limits.
Enterprise-Grade Keyless Security (OIDC):
By avoiding permanent AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY credentials in GitHub Secrets, your deployment security posture is greatly hardened. GitHub Actions requests temporary, short-lived IAM credentials via OIDC scoped strictly to the resources it needs.
️ High Availability & Auto-Healing:
The Application Load Balancer (ALB) and Auto Scaling Group (ASG) automatically monitor EC2 instance health. If an instance crashes or fails health checks, ASG automatically terminates it and launches a fresh, containerized replacement instance seamlessly.
Modular & Multi-Environment Ready:
The modular Terraform layout (vpc, backend, frontend, github_oidc) makes it effortless to spin up separate dev, staging, and production environments in minutes with zero manual AWS Console clicking.

⚠️ Architectural Trade-Offs & Considerations

No cloud architecture is silver bullet. Understanding the trade-offs helps you decide when to choose this pattern over alternatives:

Initial Setup Complexity vs. Single-Server EC2:
- Trade-off: Setting up S3, CloudFront OAC, ALB, Target Groups, ACM SSL, and OIDC requires more initial Terraform code than launching a single EC2 instance running Docker Compose + Nginx.
- Mitigation: The modular Terraform structure (modules/) abstracts away this complexity so it only needs to be written once.
Cold Start / Provisioning Time for New EC2 Instances:
- Trade-off: When ASG scales up a new instance, the user-data script installs Docker, pulls the image from ECR, and fetches SSM parameters before serving traffic (~2–3 minutes).
- Mitigation: Setting api_min_size = 1 ensures at least one instance is always warm and ready. For even faster scaling, custom pre-built AMIs (via Packer) can be used.
CloudFront Cache Invalidation Lag:
- Trade-off: Updating the React SPA requires invalidating CloudFront caches (/*), which takes ~30–60 seconds to propagate across all edge locations worldwide.
- Mitigation: Automated in the CI/CD pipeline (aws cloudfront create-invalidation) so developers don't have to trigger it manually.
Stateful Session Management (Sticky Sessions):
- Trade-off: Because requests are load-balanced across multiple EC2 instances by the ALB, the backend API must remain stateless.
- Mitigation: Authentication is handled using stateless JWT tokens and external MongoDB storage rather than in-memory server sessions.

Conclusion

Building cloud infrastructure using modular Terraform modules and automated keyless CI/CD pipelines transforms complex AWS operations into predictable, repeatable, and maintainable software engineering workflows.

By decoupling the architecture into public VPC subnets, S3/CloudFront SPA hosting, and auto-scaling EC2 container fleets, the application stays fast, cost-optimized, and resilient.

Looking for a simpler, single-server EC2 deployment pattern? Check out the companion article:

How I Built and Deployed a Full-Stack MERN App on AWS EC2 using Docker Compose, Terraform, and GitHub Actions

This guide covers deploying the full-stack MERN application on a single AWS EC2 instance using Docker Compose, Nginx reverse proxy, Certbot (Let's Encrypt SSL), and SSH-based GitHub Actions CI/CD.

Connect & Follow Me

If you found this guide helpful or have any questions about AWS, Terraform, Docker, or MERN stack architecture, let us connect!

GitHub: github.com/ideateGudy
LinkedIn: linkedin.com/in/ideategudy
️ Source Code: github.com/ideateGudy/terraform-docker-compose-ci-cd-mern-aws-deploy

Drop a reaction on Dev.to and feel free to star the GitHub repository! Happy coding!

A Prompt-Injection Firewall for AWS: What I'd Build, and What I Measured

Sat, 10 Oct 2026 20:30:16 +0000

🛡️ A Prompt-Injection Firewall for AWS: What I'd Build, and What I Measured

TL;DR

Prompt injection is OWASP's #1 LLM risk across every edition, and a 2026 proof shows pure input-wrapper defenses face a hard trilemma: they cannot be continuous, utility-preserving, and complete at the same time.

The space is no longer empty. NeuralTrust ships a commercial "Generative Application Firewall," Check Point acquired Lakera, Cloudflare has Firewall for AI, and open-source projects exist. The honest gap is a cloud-native, AWS-reference, agent-aware design, not "nobody has done this."

My proposal: a multi-layer ensemble with a weighted decision engine, output and tool-call enforcement, and a feedback loop, built the trilemma into the design rather than pretending to beat it.

I built the offline core (heuristic + vector detectors + decision engine) and benchmarked it on 285 labeled prompts. The measured trade-off is stark: at the threshold that catches 92% of attacks, 25% of legitimate prompts get flagged; tighten it until false positives are near zero and detection falls to 39%. No setting gives both. That is the trilemma, measured, and the argument for the semantic layer.

Prompt injection has sat at the top of the OWASP Top 10 for LLM Applications since the list began in 2023, and it is still number one in the 2025 and 2026 editions. That is unusual. Most vulnerability classes get a real fix and slide down the list. SQL injection had parameterized queries. XSS had output encoding and CSP. Prompt injection has no equivalent, because the flaw is architectural: an LLM reads the system prompt, the user's input, and any retrieved documents or tool output as one undifferentiated stream of text, with no built-in boundary between trusted instructions and untrusted data.

I work on LLM systems on AWS, and this is the problem I keep coming back to. This post is not a product announcement. It is the design I would build if I were building a prompt-injection firewall today, grounded in what the research actually says and an honest account of what already exists. I will be clear throughout about which parts are established fact and which are my proposed approach.

Why this problem resists a clean fix

A 2026 paper, "Why Prompt Injection Defense Wrappers Fail?", formalizes something practitioners had felt for a while. It argues that a defense that wraps the input faces a trilemma of three properties that cannot all hold at once:

Continuity — a small change to the input should not flip the defense decision.
Utility preservation — legitimate prompts must still get through and work.
Completeness — every injection attempt must be caught.

You can get two. Not three. The paper is explicit that this bounds input-wrapper style defenses specifically, and that it does not rule out training-time alignment, architectural changes, or defenses that deliberately trade away some utility. That nuance matters, and I will come back to it, because it is the single most important design constraint.

The practical consequence is that any honest firewall should stop promising completeness. It should instead be explicit about where on the continuity/utility/completeness surface it chooses to sit, and make that choice configurable.

What already exists (the part the hype usually skips)

When I first sketched this idea, my note to myself said "no one has built this." That was wrong, and it is worth correcting in public, because anyone who works in LLM security will know the landscape.

Lakera Guard — commercial, API-first prompt-injection and jailbreak detection. Check Point acquired Lakera in September 2025 and folded it into its platform. Its headline accuracy and sub-50ms latency are the vendor's own figures, not something I measured.
NeuralTrust — ships a commercial product it literally calls a Generative Application Firewall, an inline enforcement layer for LLM apps and agents. So the "GAF" concept is not paper-only; it is a shipping product.
Cloudflare Firewall for AI — a commercial layer that blocks unsafe prompts and addresses several OWASP LLM risks at the edge.
Open-source — several GitHub projects named prompt-shield (and the one-word promptshield) implement detector ensembles, reverse-proxy enforcement, and threat scoring. LLM Guard is an active rule-based scanner set, and NeMo Guardrails (NVIDIA) handles conversation-flow control. Rebuff, an earlier multi-layer project, is archived.

The "GAF" framing itself comes from a 2026 paper, "Introducing the Generative Application Firewall (GAF)", which proposes a single enforcement point that unifies prompt filters, output validators, and data masking, maintains session context, and covers agents and their tool calls, by analogy to a WAF for web traffic.

So the honest gap is narrower and more specific than "nobody has built a firewall." It is: an open, cloud-native reference architecture on AWS primitives that is agent-aware (intercepts tool calls, not just prompts), uses a multi-detector ensemble rather than a single technique, and is explicit about the trilemma trade-off it makes. That is a real and useful gap. It is not a claim of inventing the category.

The architecture I'd build

The design is a pipeline: detect, decide, enforce, learn. Each stage maps cleanly onto AWS primitives, which is the point, it should be deployable without inventing new infrastructure.

          ┌───────────────────────────────────────────────┐
 request  │  1. DETECTION  (ensemble, run in parallel)      │
 ───────► │   heuristic · ML classifier · LLM-as-judge      │
          │   vector similarity to known attacks            │
          └───────────────────────┬───────────────────────┘
                                   ▼
          ┌───────────────────────────────────────────────┐
          │  2. DECISION  (weighted vote + session context) │
          │   configurable threshold → the trilemma dial    │
          └───────────────────────┬───────────────────────┘
                                   ▼
          ┌───────────────────────────────────────────────┐
          │  3. ENFORCE   allow / block / sanitize / review │
          │   also: scan the OUTPUT, gate TOOL CALLS         │
          └───────────────────────┬───────────────────────┘
                                   ▼
          ┌───────────────────────────────────────────────┐
          │  4. LEARN   log verdicts, feed back FPs,         │
          │   grow the attack-vector store                   │
          └───────────────────────────────────────────────┘

On AWS, I'd map it like this: API Gateway as the entry point; the detector ensemble as Lambda functions, with the ML classifier on SageMaker and the LLM-as-judge on Bedrock; the attack-vector store on OpenSearch; the decision engine as a small Step Functions or Lambda orchestration reading session history from DynamoDB; and CloudWatch plus S3 for the audit trail. None of that is exotic, which is deliberate.

Why an ensemble, and what each layer is actually good at

No single detector is enough, and the reasons are well documented. The CAITLYN paper puts the trade-off bluntly: deterministic signature and regex filters run at sub-millisecond speed with zero token cost but fall to trivial obfuscation and semantic rewrites. So the ensemble is not padding; each layer covers a different failure mode of the others.

A heuristic layer is the cheap first pass. Illustrative, not a complete ruleset:

SUSPICIOUS = [
    r"ignore (all )?previous instructions",
    r"disregard the (system|above) prompt",
    r"you are now [a-z ]+",          # role-override attempts
    r"reveal (your )?(system )?prompt",
]

def heuristic_score(text: str) -> float:
    hits = sum(bool(re.search(p, text, re.I)) for p in SUSPICIOUS)
    return min(hits / 2, 1.0)   # saturate; this is a signal, not a verdict

That catches the lazy attacks for free and catches nothing clever, which is exactly why it only contributes a signal. A semantic layer (an ML classifier, or an LLM-as-judge on Bedrock) catches the paraphrased and indirect attacks the regex misses, at real token cost and latency. A vector-similarity layer compares the input against an embedding store of known attack patterns, so a variant of something you have seen scores high even if the wording is new.

The decision engine is where the trilemma becomes a dial

This is the heart of the honest version. Instead of a single detector shouting "block," the engine combines the signals into one score and compares it to a threshold you set per tenant or per use case:

WEIGHTS = {"heuristic": 0.15, "classifier": 0.35, "judge": 0.35, "vector": 0.15}

def decide(signals: dict[str, float], threshold: float) -> str:
    score = sum(WEIGHTS[k] * v for k, v in signals.items())
    if score >= threshold:            return "block"
    if score >= threshold * 0.6:      return "flag_for_review"
    return "allow"

The threshold is the trilemma choice, made explicit. A bank's internal agent can run it high and tolerate more false positives, because a missed injection is expensive. A low-stakes public chatbot runs it lower to protect utility. The firewall does not pretend to be complete; it hands you the knob and logs where you set it. That framing is the design taking the arXiv result seriously instead of marketing past it.

The weights above show the full four-detector design. The benchmark in the next section only exercises the two detectors I actually built offline (heuristic and vector), so it runs with two weights rather than four. The semantic detectors in this snippet are the layer I have not built yet.

I built the offline core and measured it

Design claims are cheap, so I implemented the two offline layers (the heuristic detector and the vector-similarity detector) plus the decision engine, and ran them against a labeled corpus of 285 prompts: 192 injection and 93 benign. The injection set is deliberately held out, paraphrases, obfuscations (spaced-out and hyphenated triggers), and indirect-via-document payloads that are not the examples the vector detector was fitted on, so the catch rate reflects generalisation rather than memorisation. The benign set includes hard negatives that mention "instructions," "system prompt," "override," and "ignore" innocently. Everything is pure standard-library Python with a fixed seed, so the run is deterministic and reproducible.

Here is the actual output, sweeping the decision threshold (weights: heuristic 0.4, vector 0.6):

 thresh  detect%    FP%  precision     F1
--------------------------------------------
   0.15      98%    52%       0.80   0.88
   0.18      92%    25%       0.88   0.90
   0.20      84%    14%       0.93   0.88
   0.25      60%     8%       0.94   0.74
   0.30      39%     1%       0.99   0.55
   0.40      10%     0%       1.00   0.18
   0.60       5%     0%       1.00   0.09

Read that top to bottom and the trilemma stops being an abstraction. At a threshold of 0.18 the firewall catches 92% of the injections, but it also flags a quarter of legitimate prompts, the hard benign negatives about writing system prompts and documenting team instructions get swept up. Tighten the threshold to 0.30 and false positives drop to 1% (precision 0.99), but detection collapses to 39%: most attacks walk straight through. There is no row where detection is high and false positives are low at the same time. The best F1 (0.90) sits near the aggressive end, where you have already accepted a real false-positive cost.

That is not a tuning failure I can polish away with better weights. It is the shape the 2604.06436 result predicts for this class of defense, and watching it fall out of real numbers is more convincing than any single headline accuracy figure. The honest reading is specific: a cheap offline ensemble reliably catches the obvious and the lexically similar, and genuinely struggles to separate a subtle paraphrase of an attack from a legitimate request that happens to talk about prompts. That overlap is exactly the region a semantic layer, an ML classifier or an LLM-as-judge on Bedrock, is supposed to resolve, and it is why I treat that layer as the next build step rather than a nice-to-have.

A note on honesty, because it is the whole point of this post: the corpus is synthetic and template-generated, so its diversity is bounded by the templates; the numbers are operational rather than a formal benchmark against published baselines; and the implementation is not open-sourced. I would not quote these figures without having run the code, and I would not dress them up as production-grade. They are a measured illustration of a trade-off, nothing more and nothing less.

The two things that make it agent-aware

Most of the field still treats this as "scan the user's prompt." For modern agentic systems that is half the surface. Two additions matter:

Output scanning. The model's response can carry an injected instruction or leak a system prompt. Scanning the output, not just the input, is where several designs (including NeuralTrust's and the GAF paper's) converge, and I'd treat it as mandatory rather than optional.

Tool-call gating. This is the one I care most about, and where indirect injection lives. When an agent decides to call a tool, that decision can be the payload: a poisoned document retrieved via RAG tells the agent to call send_email or delete_record. Gating the tool call, checking the proposed action and arguments against policy before execution, is a different and arguably more important checkpoint than filtering the prompt:

def gate_tool_call(tool: str, args: dict, policy) -> str:
    if tool in policy.denied:                       return "block"
    if tool in policy.requires_human and args_risky(args, policy):
        return "human_review"
    return "allow"

The "Firewalls to Secure Dynamic LLM Agentic Networks" work takes this further, projecting cross-agent messages onto a structured, task-scoped protocol so that persuasive framing and embedded instructions are stripped by construction. That is a stronger idea than scoring, and it is the direction I would want the tool-call layer to grow toward.

Honest limitations, up front

Because a firewall that oversells itself is worse than none:

It cannot be complete. The 2604.06436 result makes that a theorem for input-wrapper defenses, not a matter of effort. Anyone claiming 100% is wrong or redefining the problem.
The semantic layers cost latency and tokens. The sub-50ms figures you see are vendor numbers for narrower, single-layer detectors; a multi-layer ensemble with an LLM-judge step will cost more, and I would not quote a latency number I had not measured.
It is defense in depth, not a boundary. The durable lesson from the evaluation literature is that security boundaries belong in application code and permissions. A firewall lowers the hit rate; least-privilege tool scoping and human approval on destructive actions are what contain the ones that get through.

On the compliance angle

It is tempting to attach urgency to regulation, so here is the accurate version. The EU AI Act's high-risk obligations were pushed back by the 2026 Digital Omnibus to December 2027 and August 2028, not August 2026 as early drafts implied. What did take effect on August 2, 2026 is the transparency regime (Article 50) and penalty powers for general-purpose models. So the regulatory pressure is real but phased, and the honest pitch is that security controls for LLM apps are becoming table stakes over the next two years, not that a deadline hits tomorrow.

The path forward

The offline core and the trade-off measurement above are done. From here the order would be:

Add the semantic layer (an ML classifier, or an LLM-as-judge on Bedrock) specifically to attack the overlap region the offline ensemble could not separate, and re-measure whether it buys real detection without pushing false positives back up. That is the hypothesis the current numbers set up.
Harden output scanning and tool-call gating, and benchmark against indirect-injection scenarios (poisoned RAG content, malicious tool results), since that is the underserved part.
Publish it as an AWS reference architecture with the numbers attached, honest about where it sits relative to NeuralTrust, Lakera, and the OSS projects, rather than claiming to replace them.
Close the loop: feed false positives back to retune thresholds and grow the attack-vector store, and treat that feedback data as the real moat over time.

That is a build I would stand behind, because every claim in it is either cited or measured. The gap it fills is specific: an open, agent-aware, AWS-native reference design that treats the defense trilemma as a dial instead of a thing to lie about.

If you are working on LLM security, I'd genuinely like to hear which layer you think earns its keep, and whether tool-call gating or output scanning has mattered more in your own systems.

Sources

OWASP Top 10 for LLM Applications (2025, 2026) — prompt injection ranked #1 across editions. https://genai.owasp.org/llm-top-10/
arXiv 2604.06436 — "Why Prompt Injection Defense Wrappers Fail?" (the defense trilemma)
arXiv 2601.15824 — "Introducing the Generative Application Firewall (GAF)"
arXiv 2608.27990 — "CAITLYN: Can LLM Agents Autonomously Synthesize Defenses against Emerging Injection Attacks?"
arXiv 2502.01822 — "Firewalls to Secure Dynamic LLM Agentic Networks"
Check Point acquires Lakera (Sept 2025) — csoonline.com
NeuralTrust Generative Application Firewall; Cloudflare Firewall for AI (vendor sources)
EU AI Act timeline after the 2026 Digital Omnibus (high-risk obligations deferred to Dec 2027 / Aug 2028)

The benchmark figures come from my own offline reference implementation (pure standard-library Python: regex heuristic + character tri-gram cosine detectors + a weighted decision engine), run against a synthetic, template-generated labeled corpus of 192 injection and 93 benign prompts (285 total), with seed attacks held disjoint from the test set. The implementation is not open-sourced; the numbers are operational, not a formal benchmark.

Give Your Document OCR Pipeline a Memory

Sat, 10 Oct 2026 16:20:15 +0000

After OCR, most pipelines throw the layout away and re-discover the same form from scratch every time. textract-field-memory is a small, zero-dependency Python layer that remembers where fields sit, checks they landed in the right place, tells you which layout a document is, and warns you when a form quietly changes. It doesn't do OCR. It makes the OCR you already run a lot more useful.

Where this started

I kept running into the same annoying gap on document pipelines. You send a form through Textract (or Tesseract, or whatever), you get back field names and bounding boxes, you pull the values, and then you throw all the positional information away. The next copy of that exact same form shows up an hour later and the pipeline treats it like it's never seen anything like it before.

That bugged me for a few reasons.

Nothing checks that a field came from the right spot on the page. If "Total" gets read out of the shipping box instead of the amount box, the value still looks like a number, so it sails downstream and nobody blinks.

Nothing notices when a form changes. A vendor nudges their invoice template, fields move a centimeter, and you find out three weeks later when a downstream job starts choking on garbage.

And there's no cheap way to ask "have I seen this layout before?" So every document, familiar or not, takes the same expensive road.

So I wrote a little library that sits after OCR and keeps the memory the pipeline was throwing out.

What it is, and what it definitely isn't

It's a spatial memory layer. You hand it field names plus normalized bounding boxes (x, y, width, height between 0 and 1). It learns roughly where each field tends to appear on a given template, and from then on it can validate positions, identify which template a document belongs to, and watch for drift.

It is not an OCR engine and it never touches pixels. Something upstream has to extract the fields first. This library just remembers where they ended up and reasons about that.

Under the hood there's nothing exotic going on, which I think is a feature. Spatial scoring is intersection-over-union plus the distance between box centers. Name matching is a normalized Levenshtein distance. Template identification is Jaccard overlap of the field-name sets combined with the spatial score. No model, no training step, no network calls. It's pure standard-library Python with zero dependencies, and templates are just JSON files on disk.

A quick look at the API

from field_memory import TemplateMemory
memory = TemplateMemory(store_path="./my_templates")
# 1. Learn — feed documents after OCR extraction
memory.record(document, template_id="employment-form")
# 2. Locate — spatial field lookup with a confidence score
matches = memory.locate(new_doc, "Employee Name")
# → FieldMatch(combined_score=0.95, spatial_score=0.93, within_expected_region=True)
# 3. Identify — which known layout is this?
match = memory.identify_template(new_doc)
# → TemplateMatch(template_id="employment-form", similarity_score=0.87)
# 4. Detect drift — has the layout shifted?
drift = memory.detect_drift(new_doc, "employment-form")
# → DriftReport(is_drifting=False, overall_drift_score=0.02)

Clustering comes for free. If you call record() and leave out the template_id, it either merges the document into a template it recognizes or spins up a new one. Run that for a while and you end up with a map of every distinct layout flowing through your pipeline without ever configuring one by hand.

Why there's no LLM inside

For a layout you've already seen, answering "where's this field?" is basically a dictionary lookup. Sub-millisecond, and free. The same question routed through an LLM is half a second to two seconds and costs you per field. Paying that toll on forms you've processed a thousand times is silly.

So the design leans into being complementary instead of competitive:

def smart_extract(document, field_name):
    matches = memory.locate(document, field_name)
    if matches and matches[0].within_expected_region and matches[0].combined_score >= 0.85:
        return matches[0].key_value, "spatial"   # free, instant
    return call_llm(document, field_name), "llm"  # pay only when needed

Think of it as the cheap first pass that decides whether you even need the expensive one. The 0.85 threshold is just a starting point. Crank it up if a wrong answer is costly and you'd rather fall back to the LLM more often.

Where it actually helps

Domain	What it remembers	What it catches
Invoice processing	Where fields sit per vendor layout	Vendor template changes, wrong-position reads
Insurance claims	Field positions across many form types	Unknown forms that need routing for review
Healthcare intake	Where fields land on clinic forms	Layout revisions before they break downstream
Manufacturing QA	Component positions on boards	Parts that shifted from baseline

The only real requirement is that your data has named things with bounding boxes. That's it.

Some numbers, and the caveats that go with them

I put it through a stress test using real Textract (FORMS) on generated PDFs plus scanned, deliberately degraded versions of the same documents.

Layout identification came out at 100% across five vendors whose invoices look nothing alike, trained on just three documents each. On dense tax returns with 33 fields crammed into a tight three-column grid, roughly 0.03 units apart, it located every field. Drift detection behaved the way I hoped: as I pushed forms further off their baseline, the drift scores climbed smoothly from 0.012 up to 0.047 and tripped the flag once they crossed the threshold. On injected anomalies (shifted, scattered, reversed, wrong-form) it caught four of five. And because it's all local computation, lookups stay under a millisecond.

Now the honest part.

The benchmark runs on synthetic PDFs generated with ReportLab and their scanned variants, not a big real-world labeled corpus. Sample sizes are small, around 20 forms per scenario.

There's no proper precision/recall/F1 yet because I don't have a ground-truth label set built. The numbers above are operational rather than a formal comparison against baselines, and I'd treat them that way.

The one anomaly it missed was the "same layout, just fewer fields" case. The fields that remain are still sitting in the right places, so spatially nothing looks wrong. You could argue that's correct behavior, but it's worth knowing going in.

Handwritten forms are a dead end, because Textract FORMS gives back nothing to learn from. That's an upstream limitation, not something this library can paper over.

I'd rather you hit these on the page than in production.

Try it

Clone it and install from source. This works right now, nothing to sign up for:

git clone https://github.com/aws-samples/sample-textract-field-memory
cd sample-textract-field-memory
pip install -e .
python examples/demo.py   # runs on synthetic data, no AWS needed

demo.py walks through template learning, spatial matching, anomaly detection, and drift analysis on synthetic data, so you can watch the whole thing work without credentials or config.

Repo: github.com/aws-samples/sample-textract-field-memory

It lives under aws-samples but there's no AWS dependency in it. Pure standard-library Python, works completely offline with any OCR source that gives you field names and boxes. MIT-0 licensed.

If you run document pipelines, the thing I'd most like feedback on is turning these stress tests into a real labeled benchmark. And if the "remember it, stop re-discovering it" idea resonates, you've got the whole pitch.

Your LLM Keeps Making the Same Extraction Mistake. Here's How to Make It Learn.

Sat, 10 Oct 2026 16:09:42 +0000

TL;DR — I open-sourced an AWS sample that makes LLM document extraction learn from human corrections. Each QA fix is captured once and reused, so the pipeline gets more accurate and cheaper over time — with no retraining and no redeployment. Repo: github.com/aws-samples/sample-prompt-correction-memory. You can run the whole self-healing loop locally in 30 seconds, no AWS account needed.

The problem: corrections that evaporate

If you've built an LLM extraction pipeline, you know this pain.

Your QA analyst opens today's invoice. The model extracted payment_terms as "quarterly." That's wrong — "quarterly" is the billing frequency; the actual payment term is Net 30. The analyst corrects it. Done.

Tomorrow, an identical invoice from the same vendor arrives. The model extracts... "quarterly" again. Same mistake. The analyst corrects it again.

The correction evaporated. It fixed one document and taught the system nothing.

Multiply that across thousands of documents and dozens of recurring error patterns, and you get the two options most teams settle for:

Pay analysts to re-fix the same patterns forever (QA cost never goes down), or
Fine-tune a model (expensive, slow, needs ML ops, and you redo it every time the patterns shift).

There's a third way that needs neither.

The idea: treat corrections as durable memory

What if every human correction became a permanent, reusable asset?

That's the pattern behind sample-prompt-correction-memory. Instead of throwing corrections away, it stores them and feeds them back into future extractions — so the same mistake is never made twice.

It works as three tiers, and the magic is that traffic gradually shifts from the expensive tier to the free one:

🟢 Tier 1 — Graduated rules ($0, no LLM call)

When the same correction pattern recurs enough times, the system synthesizes a deterministic rule (a regex, a lookup, a normalization) that handles that field with zero LLM cost and sub-millisecond latency. "Net 30" → 30 becomes a rule; it never needs a model again.

🔵 Tier 2 — Cloud LLM extraction (confidence-gated)

Fields with no matching rule go to Amazon Bedrock (Claude Haiku 4.5). Every field comes back with a confidence score. If confidence clears the threshold, accept it and move on.

🟠 Tier 3 — Self-healing (only when needed)

If confidence is below threshold, the system retrieves the most relevant past corrections, formats them as few-shot examples, and re-extracts with a stronger model (Claude Sonnet 4.5). The model learns from the specific mistakes it made before — in-context, no training.

Every QA correction flows into a correction log (DynamoDB) that powers both the self-healing retrieval and the rule graduation. So the more the system is used, the smarter and cheaper it gets.

Why this matters (the practical payoff)

No ML ops. No retraining, no GPUs, no deployment cycles. A correction takes effect on the very next document.
Cost drops as quality rises. Normally you trade one for the other. Here, recurring fixes graduate to $0 rules — so accuracy climbs while per-document cost falls.
QA effort compounds instead of repeating. Fix it once → fixed forever.
It's a QA action, not an ML project. Improving the pipeline no longer requires the ML team. An analyst uploads a correction; the system improves.

The clearest fit is high-repetition, back-office document processing — accounts payable, contracts, compliance filings, supply-chain docs — where the same field errors recur and analyst time is the real expense.

The architecture

Fully serverless, deployed with AWS SAM:

Document ──► S3 ──► EventBridge ──► SQS ──► Lambda (extract)
                                              │
                          ┌───────────────────┼───────────────────┐
                          ▼                    ▼                   ▼
                   Tier 1: Rules       Tier 2: Bedrock      Tier 3: Self-heal
                   (deterministic)     (Claude Haiku 4.5)   (Claude Sonnet 4.5)
                          ▲                                        │
                          │                                        ▼
                          └──────── Correction Log (DynamoDB) ◄────┘
                                    (grows from QA feedback)

QA corrections are uploaded as JSON to an S3 corrections/ prefix; a second Lambda validates and ingests them into the correction log. Everything is encrypted with a customer-managed KMS key, and IAM is scoped to the specific Bedrock model ARNs the sample uses.

See it in 30 seconds (no AWS account)

The quickstart runs the entire self-healing loop with a mocked Bedrock client, so you can watch the mechanism without deploying anything:

git clone https://github.com/aws-samples/sample-prompt-correction-memory
cd sample-prompt-correction-memory
pip install -e ".[dev]"
python examples/quickstart.py

You'll see output like this:

Step 1: Initial extraction (no correction memory)
  Field:      effective_date
  Value:      March 2024        Confidence: 0.55   ❌ NO
Step 2: Self-healing triggers (0.55 < threshold 0.70)
  Retrieving corrections for 'effective_date'...
  Found: "March 2024" → "2024-03-01"
Step 3: Re-extraction with correction memory
  Value:       2024-03-01       Confidence: 0.95   ✓ YES
  Self-Healed: True

Low-confidence extraction → retrieve past correction → re-extract → correct answer. No retraining. No redeployment.

Deploying to AWS

With the AWS CLI, SAM CLI, and Bedrock model access for Claude Haiku 4.5 + Sonnet 4.5:

make deploy    # S3, DynamoDB, Lambda, EventBridge, SQS, KMS
make seed      # load sample corrections into the correction log
make trigger   # upload a sample document → triggers real extraction
make verify    # read back results (fields, confidence, self_healed)
make destroy   # empty buckets + delete the stack

make verify prints each extracted field with its confidence and whether self-healing kicked in — your proof it works end-to-end.

A companion: spatial memory

This sample provides semantic correction memory (what was extracted wrong, and why). Its companion, sample-textract-field-memory, provides spatial memory (where fields appear on a document layout). Together they form a dual memory for document pipelines: use the cheap spatial lookup when you're confident where a field is, and fall back to self-healing extraction when you're not.

Honest limitations

Because it's more useful when it's used well:

It's sample/reference code, not a production library. Harden IAM, add monitoring, and put a human in the loop for high-stakes fields before real use.
It helps most where the same field errors recur. For wildly heterogeneous, one-off documents, there's less pattern to learn.
Graduated rules currently take effect without a human approval step — in production you'd add a review gate before a synthesized rule starts answering on its own.
Don't use it with regulated data (PII/PHI/PCI) as-is; that path needs a proper compliance review.

Try it and tell me

The repo is open source under aws-samples, MIT-0 licensed, with a benchmark suite, an interactive dashboard, and an offline test harness.

👉 github.com/aws-samples/sample-prompt-correction-memory

What repetitive extraction error is your team still fixing by hand? I'd love to hear which patterns you'd want a system like this to learn.

This is an open-source AWS sample intended for demonstration and non-production use.

Cut LLM Document-Extraction Cost by 85% Without Losing Accuracy

Sat, 10 Oct 2026 16:08:52 +0000

Most production LLM extraction pipelines route every field through a frontier model, every time. It works, and for a while nobody questions the bill. Then volume grows, or someone runs the per-document math, and the real question shows up. How much of this spend is buying accuracy, and how much is just habit.

In practice, most of it is habit. The majority of fields in a typical extraction job do not need a frontier model at all. A large share of the model calls answer questions the document never asked. A surprising fraction of calls re-extract data that has not changed since the last run. None of that is a model problem. It is an architecture problem, and architecture problems are the good kind, because you can fix them without waiting on anyone's roadmap.

I build cost-optimized inference for document-extraction workloads with 100+ fields. This is the practitioner's version of what actually moves the number. The moves below, applied in sequence, take per-document cost down by 85 to 93 percent with negligible accuracy impact. The interesting part is not any single technique. It is why they compound instead of overlap.

TL;DR

Set a trusted baseline with the expensive model first, then prove a cheaper model can match it.

Skip fields a document type cannot contain (adaptive selection).

Route each field to the cheapest model that can do the job (tiered routing + distillation).

Stop re-extracting unchanged fields (field-level caching).

Stop paying for idle (serverless scale-to-zero).

The savings multiply, not add, because each move attacks a different kind of waste.

🧭 Set the baseline with the expensive model first

Before optimizing anything, run the workload on a frontier model and treat those results as the baseline. Early on the goal is not low cost. It is output you trust and a confidence signal you believe. Once that baseline exists, it becomes the yardstick for everything cheaper.

Then, field by field, test whether a smaller and cheaper model can match it. Run the cheap model over the same documents, compare against the frontier baseline, and keep the fields where agreement is high. The fields that match move to the cheaper tier. The fields that do not stay on the frontier model.

This is how the tier percentages below get decided in practice. You are not guessing which fields a lightweight model can handle. You are measuring it against output you already trust.

The same comparison runs later as the quality-floor monitor at the end of this post. It starts here, as the step that earns each downgrade.

✂️ Start by not doing unnecessary work: adaptive field selection

The cheapest extraction call is the one you never make.

A contract schema might define 100+ fields, but a given document, say a simple service agreement, only contains 40 to 50 of them. Extracting the rest produces "not found" responses that cost tokens and return nothing. Classify the document type first, then extract only the fields that can actually appear in it.

This one move eliminates 30 to 40 percent of calls per document. It is the right thing to do first, because every later optimization then operates on a smaller base.

🎯 Right-size the model to the field: tiered routing

Not every field needs the same horsepower. In a typical extraction job:

~65 to 70 percent of fields are structurally simple: dates, amounts, party names, reference numbers, categorical flags. Predictable formats, consistent locations. A lightweight model handles these at a fraction of frontier cost with equivalent accuracy.
~20 percent need moderate reasoning: interpreting indirect phrasing, resolving ambiguous references, synthesizing across sections. A mid-tier model (for example a Haiku or Nova Lite class model) is plenty.
~10 percent genuinely need a frontier model: obligation summaries, risk assessment, termination-condition analysis, cross-document context.

Route each field to the cheapest tier that can do the job. The routing decision is nearly free. It is a field-type to tier lookup table plus a confidence-based fallback, so a tier that returns low confidence escalates to the next one. No ML is needed for routing, because the mapping is stable for a given document corpus. Blended per-document cost drops 60 to 75 percent, and quality holds because the hard fields still get frontier attention. The split above is not assumed. It comes straight out of the baseline comparison from the previous section.

Distillation: earn your way to a cheaper Tier 1

Tiered routing gets better when Tier 1 is a model you distilled for your own documents. Run a frontier model (the teacher) over 200 to 500 representative documents to produce gold-standard extractions, then train a smaller student model to reproduce that behavior on your specific fields. Managed services such as Amazon Bedrock Model Distillation handle the training without ML expertise.

The student model learns your narrow task, the dates, amounts, party names, and term lengths you see every day, and it runs much cheaper and faster with a small accuracy gap (about 2 percent) on the tasks it was trained for. Fields outside its training distribution fall through to the mid-tier or frontier model.

The bootstrap problem is obvious. You do not have 200 documents on day one, so distill progressively. Months one and two run routing-only on general-purpose models (40 to 50 percent savings) while storing every validated result as future training data. In month three, once you have accumulated enough examples, distillation runs and Tier 1 becomes your student model (another 30 to 40 percent off). Each quarter you re-distill with more data, and the share of fields handled by the cheapest tier climbs from about 70 percent toward 85 percent. The cost curve bends down the longer the system runs.

♻️ Stop paying twice: field-level caching

Documents get reprocessed when schemas are updated, prompts improve, or a quality review flags systematic errors. Without caching, reprocessing 200 documents costs exactly what processing them the first time did.

Key each result by the document-section SHA-256, the field name, the prompt version, and the model version. If all four match, return the cached value with zero inference. When you update a prompt for one field type, only that field's cache entries invalidate, and everything else serves from cache. A prompt update typically touches three to five field types, so 85 to 90 percent of fields serve from cache on reprocessing. For a 200-document corpus reprocessed after a four-field prompt change, that is 800 extractions instead of 7,000, an 88 percent reduction on that cycle.

🛑 Stop paying for idle: serverless scale-to-zero

A 200-document-per-month workload is idle most of the time. Documents land in object storage, a lightweight function writes metadata that triggers a stream event, and a containerized extraction agent runs only when work exists. During idle periods the system costs nothing beyond storage. For bursty, low-volume workloads this removes the fixed infrastructure floor that usually makes small deployments uneconomical. KEDA is a clean way to get scale-to-zero and event-driven scaling for this on Kubernetes.

🧮 Why these compound instead of add

A tempting mistake is to add the savings together. Distillation saves 75 percent, adaptive selection 35 percent, caching 30 percent, so the total must be 140 percent. That is impossible. The techniques do not each operate on the original cost. Each operates on the residual the previous stage left behind. They address different sources of waste, so they multiply:

Stage	Technique	Operates on	Residual
1	Adaptive field selection	Total field count	60 to 70 percent of baseline
2	Distillation + tiered routing	Remaining fields	20 to 25 percent of baseline
3	Field-level caching	Reprocessed documents	10 to 15 percent on reprocessing
4	Serverless scale-to-zero	Infrastructure	Zero idle cost
Result	Compound effect	End to end	5 to 15 percent of naive baseline

Roughly (1 - 0.35) x (1 - 0.65) x (1 - 0.25), which lands around 0.07 to 0.15 of the original, an 85 to 93 percent reduction. Each technique matters most at a different stage. Adaptive selection eliminates unnecessary work, distillation reduces the cost of necessary work, caching eliminates repeated work, and serverless eliminates idle overhead. No one of them gets you there alone.

🛡️ Making it safe: the quality-floor monitor

Aggressive cost optimization earns an obvious objection. If 70 percent of fields go to a cheaper model, you need a way to know it is not silently getting worse, or drifting as document patterns change.

Build the check in. Each week, randomly re-run 5 to 10 percent of Tier-1 fields through the frontier model as a shadow extraction and compare. This is the same baseline comparison from the start of the post, now running continuously. Agreement at or above 98 percent means the floor is holding. Below 95 percent raises an alert. Below 90 percent auto-triggers re-distillation on the last three months of data. Shadow validation adds only about 3 to 5 percent to total cost, and it means the system monitors itself, with no dedicated QA team required.

The same monitor also spots promotions. If the mid-tier and frontier models consistently agree on a field currently routed to Tier 2, that field is a candidate to fold into the next distillation cycle, so the system keeps shifting work to the cheapest tier on its own.

💡 The payoff beyond the bill

The reason this is worth doing well is not only the savings on an existing workload. Every percentage point off per-unit cost widens the set of use cases where automation is worth building at all. A workload that only penciled out for one of your five document types at naive pricing might clear the bar for three or four once the architecture is right. Cost optimization is not a late-stage cleanup. Treat it as a first-class design requirement alongside accuracy, latency, and reliability, and more of what you want to automate becomes viable.

Five moves, five kinds of waste:

Expensive-model baseline, then prove the cheap model turns every downgrade into a measured decision.
Adaptive field selection removes unnecessary work.
Tiered routing + distillation reduces the cost of necessary work.
Field-level caching removes repeated work.
Serverless scale-to-zero removes idle overhead.

Apply them in sequence, guard them with a self-checking quality floor, and you get frontier-grade extraction at a small fraction of the naive cost, without betting accuracy on it.

Two patterns I open-sourced

Two of the ideas above are available as small, dependency-light libraries under AWS Samples (MIT-0), if you want a reference to build from:

If you are optimizing an extraction pipeline, I would be curious where your biggest cost sink turned out to be: unnecessary calls, over-powered models, reprocessing, or idle. Drop a comment.

AWS: IAM, Regions, Availability Zones, and the Shared Responsibility Model

Tue, 06 Oct 2026 13:20:33 +0000

AWS: IAM, Regions, Availability Zones, and the Shared Responsibility Model

A Practical Beginner's Guide with Certification Mapping (SAA / MLA-C02 / Professional Foundations)

The Beginner Problem

One of the most common mistakes new AWS users make is launching resources before understanding three foundational concepts:

Who is allowed to do something? → IAM
Where are resources running? → Regions and Availability Zones
Who is responsible for security? → Shared Responsibility Model

A beginner might create an EC2 instance, upload files to Amazon S3, and then realize:

They accidentally used administrator permissions everywhere.
They deployed resources in the wrong Region.
They assumed AWS automatically backs up or secures everything.
They cannot find a resource because they're viewing a different Region.

These mistakes appear simple, but they are exactly the type of concepts tested heavily in:

AWS Certified Solutions Architect – Associate (SAA)
AWS Certified Machine Learning Engineer – Associate (MLA-C02)
AWS Professional-level architecture and security domains

In this hands-on article, you'll build foundational AWS knowledge through practical implementation instead of memorization.

Learning Objectives

By the end of this exercise you should understand:

IAM users, groups, permissions, and policies
Least privilege access
AWS Regions and Availability Zones (AZs)
High availability fundamentals
AWS Shared Responsibility Model
AWS account security best practices
How these concepts fit into real cloud architectures

Where These Concepts Fit in Real Architecture

Before opening the console, it helps to understand where these concepts fit.

A simple AWS application may look like this:

User
   |
   v
IAM Authentication & Authorization
   |
   v
VPC
   |
   +---- EC2 Instance (AZ-A)
   |
   +---- EC2 Instance (AZ-B)
   |
   +---- Amazon S3 Bucket
   |
   +---- RDS Database

Architecture Diagram Description

Create an architecture image containing:

One AWS Region
Two Availability Zones within that Region
One EC2 instance in each Availability Zone
One Amazon S3 bucket shared across the Region
One IAM user accessing AWS services
Labels showing:
- IAM controls access
- AZs improve availability
- Regions determine geographic placement
- AWS and customer share security responsibilities

Understanding IAM

What Is IAM?

AWS Identity and Access Management (IAM) is the AWS service used to control who can access AWS resources and what actions they can perform.

Think of IAM as the security guard for your AWS account.

IAM answers three questions:

Who are you?
What resources can you access?
What actions are you allowed to perform?

Without IAM, every user would effectively have administrator permissions, which creates major security risks.

Real-World Example

Imagine a startup team:

Team Member	Access Needed
Developer	Amazon EC2
Data Analyst	Amazon S3
Security Engineer	IAM
Intern	Read-only access

Giving all of them AdministratorAccess would violate AWS security best practices.

Instead, IAM allows organizations to grant only the permissions required for each role. This principle is known as least privilege.

Understanding Regions and Availability Zones

AWS infrastructure is organized into Regions and Availability Zones.

These concepts are frequently tested in AWS certification exams because they directly affect:

Availability
Resilience
Disaster recovery
Compliance
Cost management

What Is a Region?

A Region is a geographic AWS location.

Examples include:

US East (N. Virginia)
US East (Ohio)
Europe (Frankfurt)
Asia Pacific (Mumbai)

When selecting a Region, consider:

User location
Regulatory requirements
Available AWS services
Latency requirements

Example

If your customers are primarily in Pakistan and India, you may choose:

ap-south-1 (Mumbai)

to reduce latency compared to a North American Region.

What Is an Availability Zone?

An Availability Zone (AZ) is an isolated AWS facility within a Region.

Examples:

us-east-1a
us-east-1b
us-east-1c

Each Availability Zone is designed to operate independently.

If one Availability Zone experiences problems:

Applications running in another AZ may continue operating.
High availability architectures remain functional.

Why Multiple Availability Zones Matter

Single-AZ deployment:

EC2
 |
AZ-A

If AZ-A fails:

Application Unavailable

Multi-AZ deployment:

EC2 (AZ-A)
     |
Load Balancer
     |
EC2 (AZ-B)

If one AZ fails:

Traffic continues through surviving AZ

This is a core AWS high-availability pattern.

Understanding the Shared Responsibility Model

One of the most important AWS concepts is understanding who is responsible for what.

Many beginners incorrectly assume:

“AWS manages security, so I'm automatically protected.”

This is false.

AWS security follows a Shared Responsibility Model.

AWS Responsibilities

AWS is responsible for security OF the cloud, including:

Physical data centers
Physical network infrastructure
Power systems
Cooling systems
Hardware
Virtualization infrastructure

AWS manages and protects the underlying cloud infrastructure.

Customer Responsibilities

Customers are responsible for security IN the cloud, including:

IAM permissions
User management
Data protection
Application security
Operating system patching on EC2
Security Groups
Encryption configuration

Certification Exam Example

Question:

An EC2 instance is compromised because security updates were never installed.

Who is responsible?

Answer: Customer

Reason:

For Amazon EC2, customers are responsible for managing and patching the guest operating system.

Prerequisites

Before beginning:

AWS account
Access to AWS Management Console
MFA enabled on the root account (recommended)
Modern web browser
Basic cloud computing familiarity

Important: AWS updates the console regularly. If your interface differs from the screenshots or instructions shown here, verify the latest workflow using the official AWS documentation.

Hands-On Lab

Part 1: Create an IAM User

Certification Concepts

SAA

Identity and Access Management
Security Best Practices
Least Privilege

MLA-C02

Secure access to machine learning resources

Step 1: Open IAM

Navigate to:

AWS Console
→ IAM
→ Users

Why This Step Matters

Users represent individual human identities.

AWS recommends avoiding daily use of the root account.

Step 2: Create a New User

Select:

Create User

Example username:

student-user

Enable:

Provide user access to AWS Management Console

Why This Option Matters

This creates a separate identity rather than sharing root credentials.

This follows AWS security best practices.

Step 3: Assign Permissions

Choose:

Attach policies directly

Select:

ReadOnlyAccess

Why This Policy?

ReadOnlyAccess allows a user to:

View resources
Explore the account
Learn AWS safely

It prevents:

Resource deletion
Infrastructure modification
Administrative changes

This demonstrates least privilege access.

Step 4: Review and Create

Review:

Username
Console access
Attached permissions

Click:

Create User

Save:

Sign-in URL
Username

You will use them later for testing.

Part 2: Explore AWS Regions

Certification Concepts

SAA

Global Infrastructure
Regional Design
Workload Placement

Step 1: Locate the Region Selector

In the AWS Console, locate the Region selector in the upper-right corner.

Examples:

us-east-1
us-west-2
eu-west-1
ap-south-1

Step 2: Switch Between Regions

Move from one Region to another.

Observe:

EC2 instance lists change
VPC configurations change
Resource inventories change

Key Lesson

Most AWS resources are Regional.

Resources created in one Region do not automatically appear in another Region.

Part 3: Explore Availability Zones

Certification Concepts

SAA

Fault Tolerance
Availability
Resilience Design

Step 1: Open EC2

Navigate to:

EC2
→ Launch Instance

You do not need to launch an instance.

You are only exploring the configuration options.

Step 2: Review Network Placement

Locate:

Availability Zone

Examples:

us-east-1a
us-east-1b
us-east-1c

Why This Matters

Workloads distributed across multiple Availability Zones are more resilient than workloads running in a single AZ.

This principle is used heavily in production architectures.

Part 4: Apply the Shared Responsibility Model

Certification Concepts

SAA

Security Design
Operational Excellence

Professional Certifications

Governance
Risk Management
Compliance

Scenario

You launch:

Amazon EC2

Now ask:

Who secures what?

AWS Secures

Physical servers
Network facilities
Storage hardware
Data center operations

You Secure

Application code
IAM permissions
Operating system updates
Security Groups
Credentials
Data access controls

Validation and Testing

Expected Result:

Login succeeds

Test 2: Attempt Administrative Actions

Attempt:

Create a new IAM policy

Expected Result:

Access Denied

Reason:

ReadOnlyAccess does not permit administrative changes.

Test 3: Switch Regions

Move between Regions.

Expected Result:

Different resources appear

Test 4: Identify Availability Zones

Locate at least two AZs within your selected Region.

Expected Result:

Multiple AZs available

Test 5: Shared Responsibility Review

For an EC2 deployment, identify customer responsibilities.

Expected Answers:

IAM configuration
Operating system patching
Security Group configuration

Troubleshooting

Case 1: Resources Cannot Be Found

Cause

Wrong Region selected.

Fix

Verify the Region selector.

Case 2: Access Denied Errors

Cause

IAM policy lacks required permissions.

Fix

Review attached policies and permissions.

Cause

Console access was not enabled during user creation.

Fix

Edit the user settings and enable console access.

Case 4: Using Root Account for Everything

Cause

Common beginner mistake.

Fix

Create IAM users for daily activities.

Enable MFA on the root account.

Case 5: Assuming AWS Configures Security Automatically

Cause

Misunderstanding Shared Responsibility.

Fix

Review which responsibilities belong to AWS and which belong to customers.

Security Considerations

Follow these security practices:

Enable MFA on privileged accounts.
Avoid using the root user for routine work.
Apply least privilege permissions.
Remove unused users.
Review permissions regularly.
Use IAM roles where possible.
Protect access keys.
Monitor account activity.

Cost and Free Tier Awareness

This exercise primarily uses informational console activities.

However, always remember:

AWS pricing changes over time.
Free Tier eligibility changes.
Different Regions may have different pricing.

Before launching resources:

Review AWS Pricing
Review AWS Free Tier documentation
Confirm current pricing directly in AWS documentation

Never assume a resource is free because a tutorial says so.

Cleanup

Delete Test IAM User

Navigate to:

IAM
→ Users
→ student-user
→ Delete

Verify Deletion

Confirm:

User no longer exists

Delete Additional Resources

If you launched resources while experimenting, remove:

EC2 Instances
EBS Volumes
Elastic IP Addresses
Load Balancers

Review each service manually to ensure cleanup is complete.

Intermediate Extension Challenge

Create:

Developers Group

Attach:

ReadOnlyAccess

Then:

Create multiple users.
Add users to the group.
Test permissions.

Skills Developed

IAM Groups
Permission Management
Least Privilege at Scale

Advanced / Professional Discussion

At the professional architect level, these concepts become organizational design decisions rather than basic AWS features.

Multi-Account Strategy

Many organizations separate:

Development
Testing
Production

into different AWS accounts.

Benefits:

Better security boundaries
Easier auditing
Reduced operational risk

Multi-Region Architectures

Mission-critical applications often deploy across multiple Regions.

Benefits:

Disaster recovery
Geographic redundancy
Improved global user experience

Advanced IAM

Large organizations frequently adopt:

IAM Roles
Federation
AWS IAM Identity Center
Attribute-Based Access Control (ABAC)

to manage access at scale.

Compliance Considerations

Region selection may be driven by:

Data residency requirements
Regulatory obligations
Customer contracts
Disaster recovery requirements

These considerations commonly appear in AWS architecture certification exams.

References
AWS IAM Documentation: https://docs.aws.amazon.com/iam/
Getting Started with IAM: https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started.html
AWS Regions and Availability Zones: https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions-availability-zones.html
AWS Regions Documentation: https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html
AWS Shared Responsibility Model: https://aws.amazon.com/compliance/shared-responsibility-model/

Developer Weekly Briefing — October 2, 2026

Fri, 02 Oct 2026 12:18:45 +0000

A lighter week by volume — but the three security stories that did land are worth understanding carefully. Here's what mattered.

A crafted package name was enough to reach an AI agent's AWS credentials

BeyondTrust's Phantom Labs researcher Sergio Garcia found two command injection vulnerabilities in the Amazon Bedrock AgentCore Python SDK's install_packages() function — and the attack surface was exactly as small as the title suggests.

The SDK let agents install Python packages into a Code Interpreter sandbox by passing names to pip install. The first version validated names against a blocklist of five characters. It left out the newline. A name like "pandas" followed by a newline and a shell command reached Code Interpreter as two lines. The second ran. That's CVE-2026-12530, affecting SDK versions 1.1.3 through 1.6.0.

AWS patched it with a regex allowlist. Garcia put a shell command substitution inside square brackets — the pip extras syntax — and the shell ran it before pip did. That's CVE-2026-16796, affecting versions 1.6.1 through 1.18.0. Both carry CVSS 4.0 scores of 8.4.

Where the Code Interpreter session had an execution role attached, the same commands pulled the role's temporary AWS credentials. A function that only installs packages looks far safer to hand an agent than a shell. That's exactly why careful teams expose it.

The fix that held arrived in version 1.18.1: quote the package name so the shell reads it as text, not a command. The logging detail worth acting on: both CloudTrail data events and AgentCore Observability application logs are off by default. CloudTrail doesn't record the command. The application logs do. Watch for references to the metadata service address 169.254.169.254 in those logs.

Mitch Ashley's operational advice: "Scope every role to what it needs, treat package names and dependency files as untrusted input, and turn on the AWS logging by default." If you're running agents on AgentCore, upgrade to 1.18.1 or later, audit your execution role permissions, and consider a fixed package list rather than one the agent can extend.

GitHub's security autofix agent now remembers what it fixed

GitHub agentic autofix — which handles security alerts by exploring relevant files, proposing a fix, rerunning CodeQL to confirm, and opening a draft PR — now uses Copilot Memory when enabled. Before working on an alert, it checks stored memories for context. After creating a fix, it saves the pattern.

The more interesting part is where those patterns go. They don't stay inside autofix — they inform other Copilot features, including code review and the cloud agent, about how this specific repository handles security issues. A fix that runs through autofix can surface as a flag in code review before a similar mistake merges. That moves the work from cleaning up alerts after the fact to catching them earlier.

The design detail worth noting: repository facts are stored with citations pointing to the code that supports them, and facts are validated against the current branch before use. Any unused fact is deleted after 28 days. For enterprise and organization plans, an administrator has to enable Memory before any of this applies. Both features are still in public preview.

IBM Bob goes self-hosted — reaching the enterprises that wouldn't send code to the cloud

IBM made Bob generally available as SaaS in April and said on-premises was coming. It's here. Bob can now run on-premises, in private and sovereign clouds, and in fully air-gapped environments with no outside network connection.

The target market is clear: banks, insurers, government agencies, and organizations with mainframe modernization work they're unwilling to push through someone else's infrastructure. IBM says more than 80,000 employees now use Bob internally, up from 100 developers when internal use began in June 2025, with surveyed users reporting an average 45% productivity gain. The IBM Z Premium Package also gets self-hosted support and deeper mainframe-specific context.

Mitch Ashley's framing is worth keeping: "Bob is IBM's bid for a beachhead with enterprise developers, and self-hosting puts it where cloud-first coding agents struggle to reach. An agent trusted within those core systems is positioned to become the platform on which the same teams build their own agents. That makes choosing Bob a multi-year commitment."

Self-hosting moves the operational burden to the customer: provisioning GPU capacity, managing model updates, monitoring agent behavior, maintaining audit trails. Air-gapped environments also narrow available models. Worth pricing that operational overhead before committing.

Git 2.56: conflict markers, branch cleanup, and performance at scale

Git 2.56 ships with changes that matter at two levels — safety guardrails for everyday use and performance improvements that compound at scale.

The new git add --resolved only stages currently unmerged paths and scans them for leftover conflict markers before staging. The new git branch --delete-merged cleans up merged branches with pattern matching and a --dry-run preview. Both are exactly the guardrails agent workflows need — Mitch Ashley: "An agent that opens branches, resolves conflicts, and rebases hits every rough edge constantly, and a conflict marker staged into a shared branch becomes verification debt someone must find."

On performance: merge-base lookups on the Linux kernel dropped from 167,441 traversal steps to 3,887. A path-limited diff on a Chromium checkout went from about eight minutes to 0.07 seconds. Path-walk repacking on Fluent UI shrank a 558.5 MB pack to 164.4 MB — 71% smaller. For teams running large repositories, or agents running Git at machine speed, the cumulative savings are significant.

CData's new AI gateway fixes the three reasons enterprise AI stalls. Context — or the lack of it — is the most common reason enterprise AI projects don't reach production. CData's approach puts the context layer at the center rather than treating it as an afterthought. Read more
What developers already know about data center delays. The supply chain reality behind GPU availability and data center timelines, from a developer perspective. Read more
PSI's Emmy: 500,000 designs, one physicist in the loop. How PSI uses AI to rethink data center trade-offs at a scale no human design process could cover. Read more
Developer Spotlight: Mike Dabydeen on Rust, AI, and engineering leadership. A conversation with the developer whose comments on the Coder Legion PocketOS thread generated some of the best technical discussion we've published this year. Read more
Every AI agent has a boss. It's time we named them. From Insights From Analytics — the accountability question that most agentic AI governance frameworks are still avoiding. Read more

The through-line this week: agent execution roles, credential scope, and memory persistence are the new attack surface. A package name reaches AWS credentials. A fix pattern teaches a code reviewer. A coding agent goes air-gapped behind a firewall. All three stories are about where agents sit in the trust model — and how much damage the wrong scope creates when something goes wrong.

See you next Friday.

Developer Weekly Briefing is published every Friday on Coder Legion. Written by Tom Smith.

Why the Enterprise Demands VPC for MCP: Deep Dive into SEOSiri’s Isolated AI Security Blueprint

Thu, 01 Oct 2026 08:36:31 +0000

The adoption of the Model Context Protocol (MCP) has changed how developers orchestrate contextual data between Large Language Models (LLMs) and systems. However, as autonomous AI agents move from local terminal playgrounds into the corporate workspace, a significant structural friction point emerges: Network Security.
Engineering and InfoSec teams have reached a turning point where exposing internal resources over standard public proxies is no longer viable.

This post analyzes the architectural necessity of supporting Virtual Private Clouds (VPC) for MCP architectures, details how the updated enterprise suite handles this natively, and explores why the corporate landscape is rapidly adopting SEOSiri Solutions.

1. The Architectural Problem: Why MCP Needed VPC Protection

The Model Context Protocol establishes a direct interface enabling AI clients to read schemas, execute local development scripts, and query underlying file structures. While highly effective, exposing this capability across public endpoints presents fundamental enterprise challenges:

Data Leakage Exposures: Without isolated boundaries, every database pointer, microservice context layer, and internal schema route traverses standard public internet paths.
Compliance Deadlocks: Regulated sectors (such as FinTech, Healthcare, and Defense) operate under strict internal mandates that strictly prohibit public ingress or egress configurations for core computational stacks.
Attack Surface Multipliers: Traditional public gateways require persistent firewall openings, increasing the vulnerability of internal development infrastructure to lateral scan vulnerabilities.

By introducing native VPC Support, MCP operations are brought directly into standard zero-trust network boundaries, enabling secure AI agent interaction behind isolated perimeters.

2. Deep Dive: How SEOSiri Bridges the Isolated Network Layer

The latest deployment module from the SEOSiri Developer Portal adapts the core Model Context Protocol to operate within fully locked-down cloud network structures:

[ Isolated Core VPC Subnet ]
├── Internal Data Source / DB
├── Proprietary Private Repository
└── SEOSiri Hardened MCP Server ──► [ Intercept / Scrubbing Layer ] ──► [ Local AI Client ]

Key Technical Capabilities Built into the Security Layer

Private Ingress Restrictions: Restricts incoming traffic utilizing granular CIDR-block parsing logic, ensuring only explicitly allowed cluster configurations can trigger context exchanges.
The PII Shield Intercept: Integrated directly within the internal server node, this pipeline monitors outgoing strings, automatically redacting and tokenizing sensitive values before structural arrays reach model reasoning endpoints.
Hybrid Cloud Topologies: Native multi-cloud support allows identical initialization frameworks whether your architecture resides on AWS VPC, Google Cloud VPC, or Microsoft Azure VNets.

3. Operational Advantages: The Core Benefits

Implementing an enterprise-grade VPC strategy for your contextual AI layer yields immediate architectural dividends:

Sub-Millisecond Processing Rates: Running the gateway close to local database partitions reduces transit latency overhead during highly interactive multi-turn agent execution sequences.
Native Log Stream Aggregation: System telemetry routes directly into your internal logging tools (like AWS CloudWatch, Google Cloud Operations Suite, or Datadog), keeping compliance audits fully within your administrative perimeter.
Consistent IAM Authentication Policies: Connects directly into existing enterprise access managers using OpenID Connect (OIDC) or strict OAuth2 verification schemas without requiring secondary secret tokens.

4. Why the Corporate World is Standardizing on SEOSiri MCP

Enterprise procurement and security management groups are shifting away from fragmented open-source wrappers toward verified setups for several critical reasons:

Production-Grade Stability: Rather than fragile point-to-point scripts, it provides an audited platform tailored for high-concurrency enterprise microservices.
Simplified Compliance Mapping: Pre-configured network isolation maps provide simple documentation validation for internal security review teams during software vetting cycles.
Future-Proof Extensibility: The platform offers structured integration options that work seamlessly with modern development tools like Claude Desktop, Cursor AI, and Atlassian Rovo without modifying original backend logic.

🛠️ Deploy Within Your Private Network / VPC Tunnels

Setting up an isolated instance within your internal network or Cloudflare Zero Trust framework requires minimal configuration changes. Configure the required secure environment flags upon runtime initialization:

# Initialize the SEOSiri hardened edge gateway via Cloudflare Zero Trust Tunnel (cloudflared)
# or via private container backend within your isolated VPC subnet:
docker run -d \
  --name seosiri-mcp-vpc-instance \
  --network="vpc-private-backend" \
  --publish 127.0.0.1:8080:8080 \
  -e ALLOWED_CIDR_BLOCKS="10.0.0.0/16" \
  -e ENCRYPTION_KEY_SECRET="$VPC_ENCRYPTION_KEY" \
  -e OAUTH_DISCOVERY_URL="https://developers.seosiri.com/.well-known/openid-configuration" \
  ghcr.io/seosiri-official/stal-mcp:latest

To access comprehensive configuration templates, cluster orchestration manifests, and step-by-step guides, visit the documentation space on the [SEOSiri Developer Portal][1].
For additional insights into this architecture and long-term release updates, check out the official announcement breakdown on the [SEOSiri Live Announcement Page][2].
------------------------------


  [1]: https://developers.seosiri.com/
  [2]: https://www.seosiri.com/2026/10/seosiri-mcp-vpc-support.html

Building Email Aliases with Amazon SES: Lessons from EpicMail.org

Mon, 21 Sep 2026 14:02:27 +0000

You create an email address for a project. A month later, the project ends, but the email keeps coming.

Having a separate address helped you organize the mail. Being able to switch that address off would make it much more useful.

That is the idea behind managed aliases in EpicMail. Instead of giving users another mailbox to maintain, we wanted to give them more control over the addresses feeding the mailbox they already have.

I'm Serge, the founder of EpicMail. We use Amazon SES for our built-in email service. Building aliases around it led to some useful product and engineering lessons, especially about forwarding, identity, and what a successful send actually means.

Decide what an alias promises

Consider these example addresses:

john+project [AT] epicmail.org
john+shopping [AT] epicmail.org
john+newsletters [AT] epicmail.org

They all belong to the same base mailbox, but each can represent a different relationship. A project has an end date. A shopping address might become noisy. Newsletters might be more useful in another account.

We chose aliases that users explicitly create and manage. Once created, an alias can receive mail, appear as the sending address, show activity, be blocked, or forward to another connected account the user owns.

An arbitrary suffix does not automatically become an active EpicMail address. That is a deliberate product choice: the aliases area should show the addresses the user has chosen to use.

There is also a privacy distinction worth making clear. Plus addresses expose the base address. They are useful for organization and control, but they are not anonymous email masks. Someone who sees john+shopping [AT] epicmail.org can also see john.

Know what SES provides

SES gives developers sending and receiving capabilities. The user experience around an alias is something the application still has to define.

For straightforward sending, a verified domain identity can cover addresses beneath that domain without individually verifying each address. AWS documents the details and exceptions in its identity verification guide.

Receiving is a separate setup. It requires a supported receiving region, the appropriate MX records, receipt rules, and permissions for the resources involved. The AWS receipt-rule walkthrough covers those pieces.

Those capabilities make a feature like this possible. They do not decide whether an alias should work automatically, which account should receive a forward, or what a user should see after blocking an address.

Writing down those behaviors before building the screens makes the implementation much easier to reason about.

Test the whole conversation

Receiving a message is only the start. A user also expects to reply without accidentally switching identities.

For an alias feature, a useful acceptance test is a complete conversation:

Send a message to the alias from another mailbox.
Open it and reply using the alias.
Check the address the other person sees.
Reply back and confirm the conversation returns to the intended mailbox.

Do the same with a message sent directly from the alias. It is easy to make the compose selector look correct while overlooking what happens at the other end.

BCC deserves a test too. The visible To and Cc headers do not describe every legitimate delivery. AWS distinguishes envelope recipients from message headers in its receiving-notification documentation.

These are useful tests even if you use a different email provider. The recipient experiences the entire exchange, not just your compose window.

Forwarding changes the inbox experience

Suppose someone forwards an alias to another account they have also connected to EpicMail. If both copies remain in their inboxes, a unified view can show the same message twice.

The forward worked, but the resulting experience is annoying.

We added an auto-archive option alongside alias forwarding. It lets the user keep the original stored in the EpicMail mailbox while clearing it from Inbox after the forward is accepted for sending. Archiving preserves the message without keeping it in the way.

It also makes the setting easier to explain. The user chooses where they want to read the message, while retaining a copy in the original mailbox.

A forwarding feature should answer that practical question, not leave the user to discover the duplicate-copy problem afterward.

Use honest status language

Email has more than one meaning of "sent."

Your sending service can accept a message before the destination server receives it. Even acceptance by a destination server does not guarantee that a person will find it in their inbox or read it.

SES provides separate delivery, bounce, and complaint notifications. Those events help describe different outcomes; they should not be collapsed into an unsupported promise that every accepted message reached a person's inbox.

This affects product language as much as infrastructure. A count of forwarded messages needs a clear meaning. An error needs to tell the user whether they should take action. A success message should describe the stage the application actually knows about.

For our auto-archive option, the boundary is acceptance for sending. It is not a guarantee of final delivery, and a later bounce does not automatically restore an already archived original. That is a useful limitation to state plainly.

Treat suppression as part of deliverability

SES suppression lists are another reason an address can look valid without a message being delivered as expected.

AWS distinguishes bounce and complaint reasons, and its account-level suppression list applies to an AWS account in a particular region. The suppression-list documentation explains the behavior and available controls.

For developers, the lesson is to understand the reason and scope before changing suppression settings. A complaint carries different meaning from a delivery failure. Removing an entry is not a universal fix for deliverability, and changing your own account's list does not control another sender's system.

This is part of the operational knowledge needed around email features, even when the interface itself is small.

Make control easy to find

A useful aliases screen should answer ordinary questions quickly:

What did I create this address for?
Is it still active?
Where does its mail go?
Has it been used?
How do I stop receiving mail through it?

In EpicMail, aliases have an area for management and activity, with search and paging as the list grows. The sending identity is available from the compose window, where that choice belongs.

The aliases share the original mailbox's received-message allowance. Creating more addresses does not create additional mailboxes or additional allowances. Keeping that relationship clear avoids surprises later.

Why we added this to EpicMail

EpicMail brings existing accounts into one place: A unified inbox for Gmail, Outlook, Yahoo, iCloud, IMAP, and POP3.

Managed aliases add another way to organize incoming mail, specifically through the optional paid @epicmail.org inbox. They are included with that inbox without a separate fee per alias, and share its 5,000-message received allowance per usage period. This feature does not create aliases at a user's connected Gmail, Outlook, or other provider account.

There is a walkthrough on the EpicMail email aliases page.

The design question I would leave other developers with is this: when someone no longer wants to use an address, how easily can they take control of it? That answer can make an alias much more useful than a different spelling of the same inbox.

Beyond the Model: Why AWS Spent $8B on Generators & What It Means for Cloud Architecture

Fri, 18 Sep 2026 15:57:55 +0000

Hey Legions,

When we think about scaling AI, we usually talk about model params, CUDA kernels, or vector search efficiency. But there’s a physical limit we’re rapidly hitting: Raw Grid Power.

Amazon Web Services (AWS) just locked in an $8 Billion contract with Generac (plus a 2.6% stock warrant) to supply industrial backup generators for their AI data centers. First $2.4B in hardware lands in 2027-2028.

Here is a quick breakdown of the backend realities driving this deal:

Checkpoint Loss Risk: Training massive models takes weeks across thousands of GPUs. A 100ms grid flicker doesn't just drop a connection; it corrupts active memory states and destroys millions of dollars in compute time.

Continuous Heavy Load: A hyperscale AI data center draws 100 to 500 Megawatts continuously. That is equivalent to powering up to 375,000 homes.

Grid Upgrade Mandates: With legislation like the Ratepayer Protection Act passing, cloud providers are now forced to pay full infrastructure costs to upgrade public grids.

The Hardware Bottleneck: Power availability—not just GPU allocation—is now the primary bottleneck for scaling LLMs and cloud clusters.

As devs, are you seeing infrastructure costs or cloud provider power limits affecting how you architect your backends, or are you mostly relying on managed APIs?

👉 Read my full deep-dive analysis here: [https://www.thefluxread.com/2026/09/amazon-just-spent-8-billion-on.html]

Cloud #DevOps #SystemArchitecture #AWS #AI #Infrastructure

SafeLine WAF vs AWS WAF: Self-Hosted Control vs Managed Cloud Rules

Fri, 11 Sep 2026 01:59:47 +0000

SafeLine WAF vs AWS WAF: Self-Hosted Control vs Managed Cloud Rules

If your app lives on AWS, AWS WAF is the path of least resistance: it slots into
ALB, CloudFront, and API Gateway. SafeLine is a self-hosted alternative that works whether or not
you are on AWS. Here is how to think about the choice.

AWS WAF: integrated, but tied to the platform

AWS WAF gives you managed rule groups, rate-based rules, and tight integration with the rest of
your AWS bill and IAM. The catch is lock-in: it only protects AWS-fronted endpoints, and costs
scale with requests and rule evaluations.

SafeLine: portable and self-hosted

SafeLine runs as a container anywhere — on EC2, in a bare-metal rack, or on another cloud. You
get the same semantic-analysis engine regardless of where your app lives, and the Community
Edition is free. For a team that runs a mixed or non-AWS fleet, that portability is the headline
benefit.

Operational differences

Cost model: AWS bills per rule and per request; SafeLine's self-hosted Community Edition
has no per-request fee.
Control: SafeLine puts the dashboard and policy in your hands; AWS WAF is managed through
the AWS console/CLI.
Detection: SafeLine's semantic engine needs less signature tuning than AWS WAF's rule
groups.

When AWS WAF still wins

If you are all-in on AWS and want managed rules with zero servers to patch, AWS WAF is convenient.
SafeLine wins when you want to avoid per-request cost and keep protection consistent across
environments.

FAQ

Can SafeLine protect an ALB-backed app?

Yes — run SafeLine as a reverse proxy in front of the ALB or as the entry point.

Is there a free tier?

SafeLine's Community Edition is free (10 apps / 800 QPS).

How fast is setup?

A single container deploy completes in minutes with the official install script.

Wrap Up

If you want a self-hosted WAF you fully control, SafeLine is worth a look. It sits in
front of your existing stack as a container, uses a semantic-analysis engine to catch SQL
injection, XSS, and bot abuse with very low false positives, and the Community Edition is
free.

Official docs: https://docs.waf.chaitin.com/en/home
GitHub: https://github.com/chaitin/SafeLine
Live demo: https://demo.waf.chaitin.com:9443/statistics

Deploy it in minutes:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en

Infrastructure as Code with AWS CloudFormation

Wed, 09 Sep 2026 07:23:07 +0000

By the end of this guide, you will not just know what CloudFormation does. You will understand why each feature exists, what problem it solves, what breaks without it, and how features chain together to solve real infrastructure problems. Every concept is introduced with a problem first, not a definition first.

What is CloudFormation and Why It Matters
Logical and Physical Resources
Templates: Portable vs Non-Portable
Template Parameters and Pseudo Parameters
Intrinsic Functions
Mappings
Outputs
Conditions
DependsOn
Wait Conditions and cfn-signal
cfn-init
cfn-hup
Nested Stacks
Cross-Stack References

What is CloudFormation and Why It Matters

The Problem It Solves

Imagine you are building a three-tier web application. You need a VPC, subnets, an EC2 instance, a security group, an RDS database, an S3 bucket, and a load balancer. You click through the AWS Console and get it working. Three weeks later, a colleague asks you to replicate the exact environment for staging. You click through everything again. Two hours later, something is different you missed a security group rule, the subnet CIDR is wrong, and the RDS instance has a different parameter group.

This is the core problem. Manual infrastructure is not repeatable, not auditable, and not scalable.

CloudFormation is AWS's answer. You describe your infrastructure in a template a YAML or JSON file and CloudFormation takes that description and builds the real AWS resources. The template becomes the single source of truth.

What CloudFormation Actually Is

CloudFormation is a declarative infrastructure provisioning engine. You declare what you want, not how to create it. CloudFormation figures out the how, including the order in which resources must be created, the dependencies between them, and what to clean up if something fails.

Why It Matters Beyond Convenience

Repeatability The same template deployed ten times produces ten identical environments.
Version control Your infrastructure lives in Git. Every change is tracked. Every rollback is a git revert.
Accountability Who changed the security group? Check the commit history.
Speed A 40-resource stack that takes two hours to click through manually deploys in minutes.
Disaster recovery When a region fails, you re-deploy the template in another region. Your infrastructure is code, not memory.
Cost Stacks can be deleted entirely after use. Temporary environments cost nothing when they are gone.

CloudFormation is not just a tool. It is a practice the practice of treating infrastructure with the same discipline as application code.

Logical and Physical Resources

The Concept

This is the foundation. If you misunderstand this, everything else will be confusing.

A logical resource is what you write in the template. It is a declaration a name you give to a resource and a description of what you want.

A physical resource is what AWS actually creates when CloudFormation processes your template. It has a real ID an instance ID, a bucket name, a security group ID.

The Relationship

When you write this:

Resources:
  MyWebServer:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t3.micro
      ImageId: ami-0c55b159cbfafe1f0

MyWebServer is the logical resource. The actual EC2 instance that gets created say i-0a1b2c3d4e5f6 is the physical resource.

The logical resource exists only in your template. The physical resource exists in AWS.

Why This Distinction Matters

CloudFormation tracks the mapping between logical and physical resources in something called the stack. This mapping is what enables CloudFormation to:

Update a resource when you change the template
Replace a resource when a change requires replacement
Delete all physical resources when you delete the stack

If you manually delete a physical resource that CloudFormation manages, the stack loses track of it. On the next update or delete, CloudFormation will fail or behave unpredictably because the physical resource it expects to find is gone.

Rule: Never manually modify a physical resource that is managed by a CloudFormation stack. Make the change in the template instead.

What Happens During Stack Creation

CloudFormation reads your template.
It builds a dependency graph of all logical resources.
It creates physical resources in the correct order.
It maps each logical resource to its new physical resource.
The stack reaches CREATE_COMPLETE.

If any resource creation fails, CloudFormation rolls back it deletes everything it already created and the stack reaches ROLLBACK_COMPLETE. Nothing is left half-built.

Templates: Portable vs Non-Portable

The Problem with Non-Portable Templates

Here is a simple template:

AWSTemplateFormatVersion: "2010-09-09"
Description: Simple EC2 instance

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t3.micro
      ImageId: ami-0c55b159cbfafe1f0
      SubnetId: subnet-0abc1234

This works. Once. In one region. For one person.

The ImageId is an AMI ID. AMI IDs are region-specific. ami-0c55b159cbfafe1f0 in us-east-1 is not the same image in eu-west-1 in fact, it probably does not exist there at all. The SubnetId is hardcoded to a specific subnet in a specific AWS account.

If you or anyone else tries to deploy this template in a different region or a different account, it will fail.

This is a non-portable template. It has hardcoded values that only work in one specific context.

What Would Happen If You Ran It

In your account, same region: Works.
In your account, different region: Fails. AMI ID does not exist.
In a colleague's account: Fails. Subnet ID does not exist.
In a CI/CD pipeline targeting staging: Fails. Both IDs are wrong.

The Solution: Portable Templates

A portable template contains no hardcoded environment-specific values. Instead, it accepts inputs, references dynamic values, and uses lookup mechanisms to resolve environment-specific details at deploy time.

The tools that enable portability are:

Parameters Inputs you provide at deploy time
Pseudo Parameters Values AWS provides automatically (account ID, region, etc.)
Intrinsic Functions Functions that resolve values dynamically
Mappings Lookup tables built into the template

The portable version of the above template:

AWSTemplateFormatVersion: "2010-09-09"
Description: Portable EC2 instance

Parameters:
  SubnetId:
    Type: AWS::EC2::Subnet::Id
    Description: The subnet to launch the instance into

Mappings:
  RegionAMIMap:
    us-east-1:
      AMI: ami-0c55b159cbfafe1f0
    eu-west-1:
      AMI: ami-0d71ea30463e0ff49

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t3.micro
      ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]
      SubnetId: !Ref SubnetId

Now this template works in any region that is in the map, and in any account. The subnet is provided at deploy time. The AMI is looked up based on which region you are deploying to.

This is the difference between a template you write once and a template you write once and use everywhere.

Template Parameters and Pseudo Parameters

Template Parameters

Parameters make your template an interface. Instead of hardcoding values, you expose inputs that the person deploying the template or an automation system provides at deploy time.

Parameters:
  EnvironmentName:
    Type: String
    Default: development
    AllowedValues:
      - development
      - staging
      - production
    Description: The environment this stack is being deployed to

  InstanceType:
    Type: String
    Default: t3.micro
    AllowedValues:
      - t3.micro
      - t3.small
      - t3.medium
    Description: EC2 instance type

  DBPassword:
    Type: String
    NoEcho: true
    MinLength: 8
    MaxLength: 32
    Description: Database password will not be displayed

Key parameter types:

Type	What It Validates
`String`	Any string
`Number`	Numeric value
`AWS::EC2::Subnet::Id`	Must be a valid subnet ID in your account
`AWS::EC2::KeyPair::KeyName`	Must be a valid key pair name
`AWS::SSM::Parameter::Value<String>`	Pulls value from Systems Manager Parameter Store

The NoEcho: true property is important. It prevents the value from being displayed in the Console or CLI output. Use it for passwords and secrets. Note: it is not encryption. The value is still stored in the stack. Do not use it for genuinely sensitive production secrets use Secrets Manager or SSM SecureString instead.

To reference a parameter inside the template:

Properties:
  InstanceType: !Ref InstanceType

!Ref on a parameter returns its value. That is it. Simple.

Pseudo Parameters

Pseudo parameters are values that AWS provides automatically. You do not declare them. They are always available.

Pseudo Parameter	What It Returns
`AWS::AccountId`	Your 12-digit AWS account ID
`AWS::Region`	The region being deployed to, e.g. `eu-west-1`
`AWS::StackName`	The name of the current stack
`AWS::StackId`	The full ARN of the stack
`AWS::NoValue`	Used to conditionally remove a property
`AWS::Partition`	`aws`, `aws-cn`, or `aws-us-gov`
`AWS::URLSuffix`	`amazonaws.com` or region-specific suffix

Usage example:

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub "myapp-${AWS::AccountId}-${AWS::Region}-logs"

This creates a bucket name like myapp-123456789012-eu-west-1-logs. Because account IDs and regions are globally unique, this bucket name will never collide with anyone else's, and the same template deployed to multiple regions will always produce a unique bucket name.

AWS::NoValue is used when you want to conditionally exclude a property:

Properties:
  DBSnapshotIdentifier: !If [IsProduction, !Ref SnapshotId, !Ref AWS::NoValue]

If the condition IsProduction is false, the property is completely removed from the resource definition. AWS sees it as if you never wrote it.

Intrinsic Functions

What They Are

Intrinsic functions are built-in functions that CloudFormation evaluates when it processes your template. They let you compute values, reference other resources, join strings, look up mappings, and make decisions all at deploy time.

You cannot use them in every part of a template. They work in the Properties section of resources, in Outputs, and in Metadata. They do not work in the Parameters section.

The Core Functions

!Ref

Returns the value of a parameter or the default identifier of a resource.

# On a parameter: returns the parameter value
InstanceType: !Ref InstanceTypeParam

# On a resource: returns the resource's primary identifier
SubnetId: !Ref MySubnet   # Returns the Subnet ID of MySubnet

What !Ref returns depends on the resource type. For an EC2 instance it returns the instance ID. For an S3 bucket it returns the bucket name. For a security group it returns the group ID. Always check the CloudFormation documentation for what !Ref returns for each resource type.

!GetAtt

Returns a specific attribute of a resource not just the primary identifier.

# Get the ARN of a Lambda function
FunctionArn: !GetAtt MyLambdaFunction.Arn

# Get the DNS name of a load balancer
LoadBalancerDNS: !GetAtt MyALB.DNSName

# Get the ARN of an IAM role
RoleArn: !GetAtt MyRole.Arn

!GetAtt is how you wire resources together. You create a load balancer and then pass its DNS name to your Route 53 record. You create an IAM role and pass its ARN to a Lambda function.

!Sub

Substitutes variables into a string. The most readable way to build dynamic strings.

# Simple substitution with pseudo parameters
BucketName: !Sub "myapp-${AWS::AccountId}-${AWS::Region}"

# Substitution with logical resource references
Description: !Sub "This is the ${EnvironmentName} environment stack"

# Substitution with explicit variable map
Command: !Sub
  - "aws s3 cp s3://${BucketName}/config.json /etc/myapp/config.json"
  - BucketName: !Ref MyBucket

!Sub is cleaner than !Join for most string-building tasks. Use it by default and only reach for !Join when you are working with lists.

!Join

Joins a list of values with a delimiter.

# Join with no delimiter
PolicyArn: !Join ["", ["arn:aws:iam::", !Ref AWS::AccountId, ":root"]]

# Join with comma delimiter
AllowedOrigins: !Join [",", [!Ref Domain1, !Ref Domain2]]

!Select

Returns a single value from a list by index.

# Get the first availability zone in the region
AvailabilityZone: !Select [0, !GetAZs ""]

# Get the second
AvailabilityZone: !Select [1, !GetAZs ""]

!GetAZs returns all availability zones in a region. !Select picks one. This combination is used constantly when creating subnets across AZs.

!FindInMap

Looks up a value in a Mapping (covered in the next section).

ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]

!If

Returns one of two values based on a condition.

InstanceType: !If [IsProduction, m5.xlarge, t3.micro]

!And, !Or, !Not, !Equals

Logical operators used when defining conditions.

Conditions:
  IsProductionAndEU: !And
    - !Condition IsProduction
    - !Equals [!Ref AWS::Region, "eu-west-1"]

!ImportValue

Imports a value exported by another stack. This is how Cross-Stack References work — covered in detail in section 14.

VpcId: !ImportValue SharedInfra-VpcId

How Functions Compose

The power comes from nesting. Functions can be composed:

SecurityGroupId: !Select
  - 0
  - !Split [",", !ImportValue SharedInfra-SecurityGroupIds]

Here: import a comma-separated string of security group IDs from another stack, split it into a list, then select the first one. Three functions, one clean result.

This is not just syntax. It is the way CloudFormation templates become self-describing infrastructure documents that adapt to context.

Mappings

What They Are

Mappings are lookup tables built into your template. They let you define a set of key-value pairs and then look up a value at deploy time based on known context like which region you are deploying to, or which environment was selected.

Structure

Mappings:
  RegionAMIMap:
    us-east-1:
      AMI: ami-0c55b159cbfafe1f0
      BastionAMI: ami-0a887e401f7654935
    eu-west-1:
      AMI: ami-0d71ea30463e0ff49
      BastionAMI: ami-08d658f84a6d84a80
    ap-southeast-1:
      AMI: ami-01f7527546b557442
      BastionAMI: ami-0c5199d385b432989

  EnvironmentConfig:
    development:
      InstanceType: t3.micro
      MultiAZ: false
      DeletionProtection: false
    production:
      InstanceType: m5.large
      MultiAZ: true
      DeletionProtection: true

Usage

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]
      InstanceType: !FindInMap [EnvironmentConfig, !Ref EnvironmentName, InstanceType]

!FindInMap takes three arguments: the map name, the top-level key, and the second-level key. It returns the value at that intersection.

Limitations

Mappings are static. They are baked into the template at write time. You cannot populate them dynamically at deploy time, and you cannot look up values in external systems like SSM Parameter Store.

This means:

When an AMI is updated, you must update the mapping manually and redeploy.
You cannot have a different mapping value per account without writing account IDs into the template.

For dynamic lookups, the solution is SSM Parameter Store with AWS::SSM::Parameter::Value<String> parameter types, or Custom Resources (covered in section 19). The AWS-managed AMI parameter store path (/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64) is the standard way to always get the latest Amazon Linux AMI without hardcoding AMI IDs at all.

Outputs

What They Are

Outputs are values that CloudFormation makes available after a stack is created or updated. They serve two purposes:

Visibility Display useful information about what was created (endpoint URLs, resource IDs, ARNs).
Cross-Stack References Export a value so other stacks can import it.

Structure

Outputs:
  WebServerPublicIP:
    Description: Public IP address of the web server
    Value: !GetAtt MyWebServer.PublicIp

  LoadBalancerDNS:
    Description: DNS name for the application load balancer
    Value: !GetAtt MyALB.DNSName

  VpcId:
    Description: VPC ID for use by other stacks
    Value: !Ref MyVPC
    Export:
      Name: !Sub "${AWS::StackName}-VpcId"

When to Use Export

Only add Export when you intend for another stack to reference the value. Not every output needs to be exported. Exports create a dependency you cannot delete the exporting stack while any other stack is consuming its exports.

What Would Happen Without Outputs

Without outputs, you would need to go into the AWS Console or run CLI commands to find the DNS name of your load balancer, the ID of your VPC, or the ARN of your IAM role. Outputs surface these values automatically after deployment, making them available to operators, pipelines, and other stacks.

In a CI/CD pipeline:

# After CloudFormation deploy, grab the load balancer URL
ALB_URL=$(aws cloudformation describe-stacks \
  --stack-name my-app \
  --query "Stacks[0].Outputs[?OutputKey=='LoadBalancerDNS'].OutputValue" \
  --output text)

# Use it to run integration tests
curl -f "http://$ALB_URL/health"

Outputs make your stack queryable. This is essential for automated pipelines.

Conditions

The Problem

You want one template that can deploy to both development and production, but the production environment needs a Multi-AZ RDS instance and an additional NAT Gateway, while development needs neither. Without conditions, you need two templates. With conditions, you need one.

Structure

Conditions are defined in the Conditions section and referenced in resources.

Parameters:
  EnvironmentName:
    Type: String
    AllowedValues: [development, production]

Conditions:
  IsProduction: !Equals [!Ref EnvironmentName, production]
  IsNotProduction: !Not [!Condition IsProduction]

Resources:
  PrimaryDatabase:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceClass: !If [IsProduction, db.m5.large, db.t3.micro]
      MultiAZ: !If [IsProduction, true, false]
      DeletionProtection: !If [IsProduction, true, false]

  NATGateway:
    Type: AWS::EC2::NatGateway
    Condition: IsProduction
    Properties:
      SubnetId: !Ref PublicSubnet
      AllocationId: !GetAtt ElasticIP.AllocationId

The Condition: IsProduction on NATGateway means: only create this resource if IsProduction is true. The !If inside PrimaryDatabase means: use different property values depending on the condition.

Condition Operators

Conditions:
  IsProduction: !Equals [!Ref Env, production]
  IsUS: !Equals [!Ref AWS::Region, us-east-1]
  
  # Both must be true
  IsProductionUS: !And
    - !Condition IsProduction
    - !Condition IsUS

  # Either must be true
  IsProductionOrUS: !Or
    - !Condition IsProduction
    - !Condition IsUS

  # Invert
  IsNotProduction: !Not [!Condition IsProduction]

Limitation

Conditions are evaluated at deploy time. They cannot change during stack execution. You cannot create a condition that says "if resource X was successfully created, then create resource Y" that is what DependsOn and WaitConditions handle.

DependsOn

The Problem

CloudFormation builds resources in parallel where possible. It determines the order automatically by following !Ref and !GetAtt references if resource B references resource A, CloudFormation knows to create A first.

But sometimes resource B does not reference resource A in its properties, yet it still needs A to exist before it can be created or function correctly. CloudFormation does not know about this implicit dependency, so it might try to create both simultaneously, and B fails because A is not ready.

The Solution

DependsOn explicitly tells CloudFormation: do not start creating this resource until that resource is complete.

Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16

  InternetGateway:
    Type: AWS::EC2::InternetGateway

  VPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref MyVPC
      InternetGatewayId: !Ref InternetGateway

  PublicSubnet:
    Type: AWS::EC2::Subnet
    DependsOn: VPCGatewayAttachment
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: 10.0.1.0/24

Here, PublicSubnet references MyVPC via !Ref, so CloudFormation knows it must wait for the VPC. But the subnet does not reference VPCGatewayAttachment in its properties. Without DependsOn, CloudFormation might create the subnet before the gateway is attached, and routing will not work correctly.

A More Common Use Case — RDS and EC2

Resources:
  Database:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceClass: db.t3.micro
      Engine: mysql

  AppServer:
    Type: AWS::EC2::Instance
    DependsOn: Database
    Properties:
      UserData: !Base64
        !Sub |
          #!/bin/bash
          echo "DB_HOST=${Database.Endpoint.Address}" >> /etc/app/config

The AppServer does reference Database via !Sub, which creates an implicit dependency. But DependsOn is also good practice here you want the database to be fully ready before the application server boots and tries to connect.

What Happens Without It

Without DependsOn where it is needed, resources are created in the wrong order. The stack may still reach CREATE_COMPLETE but your application may fail at runtime because the dependency was not ready when the dependent resource was configured.

Limitation

DependsOn only tells CloudFormation to wait until the resource creation is complete. It does not tell CloudFormation to wait until the resource is ready to serve traffic or until a process inside the resource has finished running. For that, you need Wait Conditions and cfn-signal.

Wait Conditions and cfn-signal

The Problem That DependsOn Cannot Solve

CloudFormation considers an EC2 instance "created" the moment the API call to launch it succeeds. From CloudFormation's perspective, the resource is done. But the instance has not finished booting. The operating system is still starting. Your bootstrap script the one that installs your application, configures the web server, and starts your process is still running.

If a second resource depends on the application being ready, DependsOn is not enough. CloudFormation will proceed the moment the EC2 API reports success, not when your application is actually ready.

The Solution: cfn-signal and WaitConditions

cfn-signal is a script that runs inside your EC2 instance and sends a signal back to CloudFormation: "I am done and I succeeded" or "I am done and I failed."

A WaitCondition is a CloudFormation resource that pauses the stack and waits for a specific number of signals before proceeding.

Together, they let you tell CloudFormation: "Wait until the bootstrap process inside the instance has finished before you continue creating other resources."

How It Works. Step by Step

You create an EC2 instance with a UserData script that runs your bootstrap logic.
At the end of the script, you call cfn-signal to send a success or failure signal.
CloudFormation sees the signal and either continues or fails the stack.

The Code

Resources:
  WaitHandle:
    Type: AWS::CloudFormation::WaitConditionHandle

  WebServer:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]
      InstanceType: t3.micro
      UserData:
        !Base64
          !Sub |
            #!/bin/bash -xe
            # Install and configure application
            yum update -y
            yum install -y httpd
            systemctl start httpd
            systemctl enable httpd
            echo "<h1>Hello from ${AWS::StackName}</h1>" > /var/www/html/index.html
            
            # Signal CloudFormation that bootstrap is complete
            /opt/aws/bin/cfn-signal -e $? \
              --stack ${AWS::StackName} \
              --resource WebServerWaitCondition \
              --region ${AWS::Region}

  WebServerWaitCondition:
    Type: AWS::CloudFormation::WaitCondition
    DependsOn: WebServer
    Properties:
      Handle: !Ref WaitHandle
      Timeout: 600
      Count: 1

-e $? passes the exit code of the last command. If the script ran successfully, $? is 0 CloudFormation receives a success signal. If the script failed, $? is non-zero CloudFormation receives a failure signal, fails the wait condition, and rolls back the stack.

What Happens Without cfn-signal

Without signalling, CloudFormation marks the EC2 instance as CREATE_COMPLETE the moment the API call succeeds typically within 10-15 seconds. Any resource that depends on the instance being fully configured will attempt to use it before it is ready. Load balancer health checks will fail. Downstream resources will misconfigure. Your application stack reaches CREATE_COMPLETE in a broken state.

This is the most common cause of "CloudFormation says it worked but the application is not working" problems.

Timeout

The Timeout is in seconds. 600 means CloudFormation will wait up to 10 minutes for the signal. If no signal arrives, the stack times out and rolls back. Size this based on how long your bootstrap realistically takes, with a comfortable buffer.

Limitation

cfn-signal handles the moment of creation well. But what about ongoing configuration? What if you need to update the instance configuration after the stack is deployed, or re-apply configuration if it drifts? For that, you need cfn-init.

CloudFormation Init (cfn-init)

The Problem with UserData

UserData is a blunt instrument. It is a script that runs once at instance launch and that is it. If you update the CloudFormation template and the instance is not replaced, UserData does not re-run. If the configuration drifts someone manually changes a file on the instance there is no way for CloudFormation to detect or correct it.

Also, UserData is imperative: you write step-by-step instructions. The result depends entirely on the starting state of the machine. If any step fails partway through, you have a half-configured instance with no clean way to recover.

The Solution: cfn-init

cfn-init is a declarative configuration engine built into the CloudFormation helper tools. Instead of writing scripts that say "run these commands," you declare the desired state: "these packages should be installed, these files should exist with this content, these services should be running."

Configuration is written in the Metadata section of the resource using the AWS::CloudFormation::Init key.

Structure

Resources:
  WebServer:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          packages:
            yum:
              httpd: []
              php: []

          files:
            /var/www/html/index.php:
              content: !Sub |
                <?php
                echo "<h1>Environment: ${EnvironmentName}</h1>";
                echo "<p>Stack: ${AWS::StackName}</p>";
                ?>
              mode: "000644"
              owner: apache
              group: apache

            /etc/httpd/conf.d/myapp.conf:
              content: |
                <VirtualHost *:80>
                    DocumentRoot /var/www/html
                    DirectoryIndex index.php
                </VirtualHost>
              mode: "000644"
              owner: root
              group: root

          services:
            sysvinit:
              httpd:
                enabled: true
                ensureRunning: true
                files:
                  - /etc/httpd/conf.d/myapp.conf
                packages:
                  yum:
                    - httpd

    Properties:
      ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]
      InstanceType: t3.micro
      UserData:
        !Base64
          !Sub |
            #!/bin/bash -xe
            # Run cfn-init to apply the configuration
            /opt/aws/bin/cfn-init -v \
              --stack ${AWS::StackName} \
              --resource WebServer \
              --region ${AWS::Region}
            
            # Signal success or failure
            /opt/aws/bin/cfn-signal -e $? \
              --stack ${AWS::StackName} \
              --resource WebServerWaitCondition \
              --region ${AWS::Region}

  WebServerWaitCondition:
    Type: AWS::CloudFormation::WaitCondition
    DependsOn: WebServer
    Properties:
      Handle: !Ref WaitHandle
      Timeout: 600
      Count: 1

  WaitHandle:
    Type: AWS::CloudFormation::WaitConditionHandle

The Four cfn-init Keys

Key	What It Does
`packages`	Installs system packages via yum, apt, rpm, or other package managers
`files`	Creates files with specific content, permissions, and ownership
`commands`	Runs shell commands in a specific order with optional test conditions
`services`	Ensures services are started, enabled, and restarted when dependencies change

configSets Ordering Multiple Configurations

When you have complex configuration that needs to run in phases, use configSets:

Metadata:
  AWS::CloudFormation::Init:
    configSets:
      full_install:
        - install_cfn
        - install_base
        - install_app
        - configure_app

    install_cfn:
      files:
        /etc/cfn/cfn-hup.conf:
          content: !Sub |
            [main]
            stack=${AWS::StackId}
            region=${AWS::Region}
          mode: "000400"
          owner: root
          group: root

    install_base:
      packages:
        yum:
          httpd: []
          php: []
          php-mysqlnd: []

    install_app:
      files:
        /var/www/html/index.php:
          content: !Sub |
            <?php phpinfo(); ?>
          mode: "000644"
          owner: apache
          group: apache

    configure_app:
      services:
        sysvinit:
          httpd:
            enabled: true
            ensureRunning: true

Then in UserData, reference the configSet:

/opt/aws/bin/cfn-init -v \
  --stack ${AWS::StackName} \
  --resource WebServer \
  --configsets full_install \
  --region ${AWS::Region}

What Happens Without cfn-init

Without cfn-init, you rely entirely on UserData scripts. These are harder to maintain, harder to debug, and run only once at launch. Configuration drift is invisible, and updating configuration requires replacing the instance. cfn-init makes your instance configuration as declarative and auditable as your infrastructure definition.

Limitation

cfn-init applies configuration at launch. It does not continuously monitor or re-apply configuration when the template changes, unless the instance is replaced. For tracking template changes and re-applying configuration without replacing instances, you need cfn-hup.

cfn-hup

The Problem cfn-init Alone Cannot Solve

You update your CloudFormation template specifically, you change the content of a configuration file managed by cfn-init. You run aws cloudformation update-stack. CloudFormation processes the update.

If the EC2 instance is not being replaced (because the change does not require replacement only a metadata change), CloudFormation will not re-run cfn-init. The instance keeps running with the old configuration. The update completes, the stack reaches UPDATE_COMPLETE, and your configuration is silently out of date.

The Solution: cfn-hup

cfn-hup is a daemon that runs on the instance and polls the CloudFormation stack for changes to the resource's metadata. When it detects a change, it re-runs cfn-init to apply the updated configuration.

This gives you the ability to update instance configuration through CloudFormation without replacing the instance.

How to Set It Up

cfn-hup requires two configuration files on the instance, typically created via cfn-init itself:

Metadata:
  AWS::CloudFormation::Init:
    configSets:
      full_install:
        - install_cfn
        - install_app

    install_cfn:
      files:
        /etc/cfn/cfn-hup.conf:
          content: !Sub |
            [main]
            stack=${AWS::StackId}
            region=${AWS::Region}
            interval=5
          mode: "000400"
          owner: root
          group: root

        /etc/cfn/hooks.d/cfn-auto-reloader.conf:
          content: !Sub |
            [cfn-auto-reloader-hook]
            triggers=post.update
            path=Resources.WebServer.Metadata.AWS::CloudFormation::Init
            action=/opt/aws/bin/cfn-init -v \
              --stack ${AWS::StackName} \
              --resource WebServer \
              --configsets full_install \
              --region ${AWS::Region}
            runas=root
          mode: "000400"
          owner: root
          group: root

      services:
        sysvinit:
          cfn-hup:
            enabled: true
            ensureRunning: true
            files:
              - /etc/cfn/cfn-hup.conf
              - /etc/cfn/hooks.d/cfn-auto-reloader.conf

How It Works

cfn-hup.conf tells cfn-hup which stack to watch and how often to poll (every 5 minutes in this example).

cfn-auto-reloader.conf tells cfn-hup what to do when it detects a change: watch the Metadata.AWS::CloudFormation::Init path of the WebServer resource, and when it changes, re-run cfn-init with the full configSet.

When you update the template — change a file, add a package, modify a service — cfn-hup detects the metadata change within the poll interval and re-applies the full configuration to the running instance.

The Complete Pattern

cfn-init, cfn-signal, and cfn-hup form a complete configuration management pattern:

cfn-init applies configuration at launch
cfn-signal tells CloudFormation that configuration is complete
cfn-hup keeps configuration in sync with the template over time

Together, they turn your EC2 instances into configuration-as-code managed infrastructure that stays synchronized with your CloudFormation templates without requiring replacement.

Nested Stacks

The Problem with Single Large Stacks

A CloudFormation stack has a hard limit of 500 resources. But the real problem appears well before that limit.

A template with 80+ resources becomes difficult to read, difficult to test, and difficult to reason about. Changes to one part of the template require updating and deploying the entire thing. Teams working on different parts of the infrastructure step on each other. Re-using patterns across projects is impossible because everything is in one monolithic file.

This is the same problem that led software engineers to adopt functions, modules, and packages. The solution is the same: decomposition.

The Solution — Nested Stacks

A nested stack is a CloudFormation resource of type AWS::CloudFormation::Stack. It references another CloudFormation template stored in S3 and deploys it as a child stack. The parent stack manages the lifecycle of its child stacks.

Resources:
  NetworkStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/mybucket/templates/network.yaml
      Parameters:
        EnvironmentName: !Ref EnvironmentName
        VpcCidr: 10.0.0.0/16
      TimeoutInMinutes: 20

  AppStack:
    Type: AWS::CloudFormation::Stack
    DependsOn: NetworkStack
    Properties:
      TemplateURL: https://s3.amazonaws.com/mybucket/templates/app.yaml
      Parameters:
        EnvironmentName: !Ref EnvironmentName
        VpcId: !GetAtt NetworkStack.Outputs.VpcId
        SubnetIds: !GetAtt NetworkStack.Outputs.SubnetIds
      TimeoutInMinutes: 30

  DatabaseStack:
    Type: AWS::CloudFormation::Stack
    DependsOn: NetworkStack
    Properties:
      TemplateURL: https://s3.amazonaws.com/mybucket/templates/database.yaml
      Parameters:
        SubnetIds: !GetAtt NetworkStack.Outputs.SubnetIds
        DBPassword: !Ref DBPassword

The parent template is thin it orchestrates the child stacks and passes data between them via outputs and parameters. Each child template is a focused, independently testable unit.

Passing Data Between Nested Stacks

Notice !GetAtt NetworkStack.Outputs.VpcId. This is how you pass data from one nested stack to another:

The network.yaml template has an Outputs section that exports VpcId.
The parent references it via !GetAtt NetworkStack.Outputs.VpcId.
The parent passes it as a parameter to AppStack.

# network.yaml Outputs section
Outputs:
  VpcId:
    Value: !Ref MyVPC
  SubnetIds:
    Value: !Join [",", [!Ref SubnetA, !Ref SubnetB]]

What Would Happen Without Nested Stacks

All resources live in one template. It grows. Team members modify the same file. Changes to the network layer require touching the application template. Testing one component requires deploying everything. The template becomes the infrastructure equivalent of a 5,000-line monolithic application technically functional but practically unmaintainable.

Limitation

Nested stacks solve the organization and size problem within a single deployment. They do not solve the problem of sharing infrastructure across multiple independent teams or projects. If Team A creates a VPC that Team B also needs, nested stacks are not the right tool Cross-Stack References are.

Cross-Stack References

The Problem Nested Stacks Do Not Solve

Nested stacks keep related resources together. But what about shared infrastructure that multiple independent stacks need to reference?

Your networking team creates a VPC stack: VPC, subnets, route tables, NAT gateways. This is the foundation. Three separate application teams each deploy their own stacks that need to run inside that VPC.

With nested stacks, the networking stack would need to be the parent of all three application stacks. That creates an artificial coupling the networking team owns the deployment of all application stacks. This does not reflect how real teams work.

The right model is: the networking stack exists independently, exports its values, and each application stack imports what it needs.

The Solution: Exports and ImportValue

A stack can export named values. Any other stack in the same region and account can import those values.

Exporting stack (networking stack):

Naming Convention

Export names must be unique within a region and account. The convention StackName-ResourceName is standard. You can also use !Sub with pseudo parameters for uniqueness:

Export:
  Name: !Sub "${AWS::StackName}-VpcId"

Putting It All Together.

Every feature in CloudFormation exists because there was a real problem that could not be solved without it. Here is how they connect:

None of these features are optional in a serious production environment. Each one closes a gap that, without it, requires manual intervention, custom scripting, or accepted risk.

Written by Onyedikachi Obidiegwu | Cloud Security Engineer

Act Security's Amphi Shrinks a 30,000-Character AWS Policy Down to 7,000

Thu, 03 Sep 2026 18:00:00 +0000

David Kerber pulled up a policy with 30,000 characters in it — right at the edge of what AWS will let you attach to a role — and ran it through a tool called IAM Shrink. A few seconds later it was 25,000 characters. He ran it again: 23,000. One more pass, and it dropped to 7,000. Same permissions, a quarter of the size, no additional attack surface.

That's the kind of problem engineers hit constantly and rarely talk about: AWS IAM policies quickly become unreadable and oversized, and fixing them by hand means collapsing action lists into wildcards without accidentally granting permissions you didn't intend. Kerber built a tool for it because he got tired of doing it manually during consulting work. He's now a Cloud Software Architect at Act Security, which acquired his open-source project, Cloud Copilot, last month and folded it into a new free toolkit called Amphi.

Every tool in Amphi follows the same philosophy: do one thing, do it well, run it in the browser or from the command line, and never send the policy data anywhere. IAM Expand takes a wildcard string like s3:*et*ob* and shows exactly which actions it matches as you type it. It can also invert a policy — take an allow, not action statement and spit back the actual list of what's allowed, pulled from AWS documentation that refreshes daily. IAM Convert turns a policy into Terraform, which Kerber built for the same reason as IAM Shrink: he was hand-converting policies too often. Chain the two together in a CI pipeline — shrink, then convert — and your Git history still shows what you actually intended to grant, even after the policy's been compressed for size.

The policy tester is the one built for how developers already think. Paste in a policy, write expectations against it — "this call should be denied," "this call should be allowed" — and the simulator tells you which statement matched and why. Edit the policy live and the results update as you type. Kerber described it as writing red-green tests for a permission set rather than guessing how it behaves.

The most complex tool handles service control policies and resource control policies — organization-wide rules that can govern every request across hundreds of AWS accounts at once. Kerber's blunt about how well understood these are in practice: "Nobody really understands them. They just think they do — including me." His answer is to run every condition in the policy against every other condition and generate a full truth table. One example he pulled up produced 108 unique permutations, each one showing exactly which combination of conditions leads to an allow or a deny.

Then there's the pair built for scale: IAM Collect pulls every policy type from every account you point it at — Kerber said some users run the free version across 7,000 to 8,000 AWS accounts at a time — and IAM Lens queries that data to determine who can actually do what. Ask it who can access a specific S3 bucket, and it runs roughly 20,000 simulations to return every principal with access, along with which objects they can access. Ask the reverse — what can this one role do — and it merges every attached policy, guardrail, and resource policy into a single answer. Kerber's framing for why that matters: "If an auditor asks you to prove who has access to a bucket with driver's license photos in it, this does that. And it scales. And it's free."

I asked what reachability mapping is actually telling an engineer. His answer was more direct than most vendor pitches: it's showing risk. "The first question is who can do it. The second question is, should they? The delta between those two is risk — and it's not a matter of if that gets exploited. It's a matter of when."

Not everything got a confident answer, and that's worth noting on its own. Asked where Amphi overlaps with Permit.io or Sysdig, Kerber said flatly he doesn't know either tool well enough to compare. He couldn't name a customer. And while Act's commercial platform already covers Azure and GCP alongside AWS, he wouldn't commit to whether the open-source tools ever will — "I don't know one way or another whether we'll pursue that."

The commercial boundary is at least clear. The open-source project stays AGPL: free to run inside your own company, off-limits if you're building a product to resell. Act's paid platform is the bigger, multi-cloud version — AWS, Azure, GCP, identity, network — and Kerber says acquiring Cloud Copilot rather than building it in-house came down to time. The tool was already validated at enterprise scale, the edge cases were already worked out, and in security, speed matters more than pride of authorship. He also says open-source contributions have picked up since the acquisition, not slowed down — Cloud Copilot went from a one-person side project to something with real backing.

His closing point is the one to sit with: the industry's been on a least-privilege journey for years, but the urgency just went up. "An attacker's going to get in, and they're going to exploit every access path you haven't closed." Free tooling that shows you exactly where those paths are is a reasonable bet regardless of which platform eventually wins.

S3 vs Supabase Storage vs Cloudinary (2026 Guide)

Mon, 31 Aug 2026 03:19:01 +0000

You're building a Next.js app that needs to store files — user avatars, product photos, PDFs, whatever — and you hit the same wall every developer hits: where do the actual files go?

For most apps, storing large user-uploaded files directly in Postgres isn't the best fit. Object storage is designed for this workload and keeps your database focused on relational data. So you're left picking between three names that show up in almost every tutorial: AWS S3, Supabase Storage, and Cloudinary. They all "store files," but they solve genuinely different problems, and picking the wrong one means either fighting your tools for months or paying for features you'll never touch.

By the end of this guide, you'll know exactly what each one is actually built for, how to wire up a basic upload with real code for all three, what they cost, and which one fits your specific project — not just whichever one showed up first in your search results.

Quick Answer

Choose AWS S3 if you want inexpensive, highly flexible object storage with a huge ecosystem and full AWS control.
Choose Supabase Storage if you're already using Supabase for your database and auth — it plugs directly into the same Row Level Security model you already know.
Choose Cloudinary if images and videos are a core part of your product and you want transformations (resizing, cropping, format conversion, transcoding) handled automatically, with almost no backend code.

If none of those single-sentence answers settled it, keep reading — the "why" behind each one is where the real decision lives.

Why This Comparison Is Tricky

Most "S3 vs X" posts compare feature checklists side by side and call it a day. The problem is these three tools aren't really competing for the same job:

S3 is raw object storage. It stores bytes under a key. It has no idea what an "image" is — resizing, format conversion, and CDN delivery are all things you have to bolt on separately (usually CloudFront).
Supabase Storage is an object-storage service with an S3-compatible API, wrapped in Supabase's API, authentication, and Postgres RLS model.
Cloudinary is a media pipeline first, storage second. Its actual product is on-the-fly image and video transformation — resize, crop, convert to WebP/AVIF, transcode video, remove backgrounds — all driven by URL parameters, with storage as the thing that makes that possible.

Keep that distinction in mind through the rest of this guide: you're not just picking "where files live," you're picking how much of the surrounding work (delivery, optimization, access control) you want to own yourself.

At a Glance

	AWS S3	Supabase Storage	Cloudinary
What it actually is	Raw object storage	Object storage + Supabase Auth/RLS integration	Media transformation pipeline
Access control model	IAM policies + bucket policies	Row Level Security (SQL policies)	Signed uploads / upload presets
Built-in image transforms	None — add an image-processing/optimization layer separately	Available in Paid Plans	Extensive (resize, crop, AI background removal, format negotiation)
Video handling	None — you'd wire up a separate transcoding service	Not a core feature	First-class — transcoding, adaptive streaming, a built-in video player
CDN included	No — add CloudFront separately	Yes, via the Supabase CDN	Yes, built in
Best paired with	Any stack, especially if you already use AWS	A Supabase-backed app (Postgres + Auth)	Any stack where images/video are core UX
Free tier	12-month intro tier only, otherwise pay-as-you-go	~1 GB storage, 5 GB egress (check current limits)	25 credits/month (1 credit = 1 GB storage, 1 GB bandwidth, or 1K transformations)<br/> credits are shared across these resources
Pricing model	Storage + requests + data transfer, billed separately (see below)	Flat plan tiers + usage overages	Credit-based, fixed tiers

💡 Tip: Pricing on all three changes often enough that any number printed in a blog post is a snapshot, not a promise. Treat the figures above as ballpark, and always check each provider's live pricing page before committing budget to one.

AWS S3: Raw, Cheap, and You Own Everything

Amazon S3 (Simple Storage Service) is the original cloud object store — files go in under a "key" (basically a path string), and you get them back out the same way. No opinions about what the file is.

Where S3 wins:

One of the cheapest raw storage per GB at real scale, especially once you're past a few hundred GB
Full control — bucket policies, lifecycle rules (auto-delete or archive old files), versioning, cross-region replication
The de facto standard — every CDN, backup tool, and CMS speaks S3's API, including S3-compatible alternatives like Cloudflare R2 and Backblaze B2

Where S3 struggles:

Zero built-in image processing. Want a thumbnail? You'll need a separate image-processing layer, such as a Lambda-based workflow, an image optimization service, or a framework-level optimizer.
No CDN out of the box — S3 alone serves from one region. You need CloudFront (or another CDN) in front of it for real-world performance.
More setup: IAM users, bucket policies, CORS config, and (for direct browser uploads) presigned URLs are all things you configure by hand.

S3 Pricing Is More Than "Per GB Stored"

This matters enough for a "which is cheapest" comparison that it deserves its own callout: S3's bill is not just storage. You're billed separately for several things at once:

Storage — per GB-month, and it varies by storage class (Standard, Infrequent Access, Glacier, etc.)
Requests — PUT/COPY/POST/LIST requests cost more per-request than GET requests, and both are metered
Data transfer out — serving files to the internet (not through a CDN) is billed per GB and is usually the line item that surprises people
Retrieval fees — if you move infrequently-accessed files to a cheaper storage class, pulling them back out costs extra
Lifecycle transitions — moving objects between storage classes automatically (e.g. Standard → Glacier after 90 days) also incurs a small per-object request cost

A storage-only comparison (S3 vs. Supabase's per-GB rate vs. Cloudinary's credits) misses this. An app that serves thumbnails directly from S3 with heavy traffic and no CDN in front can rack up request and transfer charges that dwarf the storage bill itself — which is exactly why pairing S3 with CloudFront isn't just a performance decision, it's often a cost one too (CloudFront's cached data transfer is typically cheaper than S3's direct transfer-out rate).

A Minimal S3 Upload in Next.js 16

This uses the presigned URL pattern — your server never touches the file bytes, it just asks S3 for permission and hands the browser a temporary signed URL to upload directly.

npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

// lib/s3-client.ts
import { S3Client } from "@aws-sdk/client-s3";

export const s3Client = new S3Client({
  region: process.env.AWS_REGION!,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
});

// app/api/presigned-url/route.ts
import { NextRequest, NextResponse } from "next/server";
import { PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { s3Client } from "@/lib/s3-client";
import { randomUUID } from "crypto";
import { auth } from "@/lib/auth"; // your auth solution of choice

const ALLOWED_TYPES = ["image/jpeg", "image/png", "image/webp"];
const MAX_FILE_SIZE = 5 * 1024 * 1024; // 5MB

export async function POST(request: NextRequest) {
  // 1. Authentication — reject anonymous requests before doing anything else
  const session = await auth();
  if (!session?.user) {
    return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
  }

  const { fileName, fileType, fileSize } = await request.json();

  // 2. MIME allowlist — reject anything you don't explicitly support
  if (!ALLOWED_TYPES.includes(fileType)) {
    return NextResponse.json({ error: "Unsupported file type" }, { status: 400 });
  }

  // 3. Max file size — a best-effort check against the size the client reports.
  // A presigned PUT can't fully enforce this on its own; see the note below.
  if (typeof fileSize !== "number" || fileSize > MAX_FILE_SIZE) {
    return NextResponse.json({ error: "File too large" }, { status: 400 });
  }

  // 4. Filename sanitization — never trust the client's filename for anything
  // beyond picking a safe extension. Strip everything else.
  const safeExtension =
    fileName?.split(".").pop()?.replace(/[^a-z0-9]/gi, "").slice(0, 10) || "bin";

  // 5. Server-generated key — the client never chooses where its file lands.
  // Scoping by the authenticated user's id is also your authorization boundary:
  // even if someone tampers with the request, they can only ever get a
  // presigned URL scoped to their own prefix.
  const key = `uploads/${session.user.id}/${randomUUID()}.${safeExtension}`;

  const command = new PutObjectCommand({
    Bucket: process.env.S3_BUCKET_NAME!,
    Key: key,
    ContentType: fileType,
  });

  const uploadUrl = await getSignedUrl(s3Client, command, { expiresIn: 60 });

  return NextResponse.json({ uploadUrl, key });
}

What's happening here: your server never sees the file bytes — it just signs a temporary permission slip — but it does gate who gets that slip (authentication), what they're allowed to upload (MIME allowlist), roughly how big it can be, and exactly where it lands (a server-generated key under the caller's own user id, never a client-supplied path). The browser then does a plain fetch(uploadUrl, { method: "PUT", body: file }) straight to S3, keeping large files off your server entirely — which matters a lot on serverless platforms with body-size limits.

⚠️ This example is still a demonstration, not a drop-in production upload endpoint. A few things it deliberately leaves out, worth knowing about before you ship:

Authoritative size enforcement — the fileSize check above only validates what the client claims its file is. A presigned PutObjectCommand doesn't enforce a byte limit by itself. For a hard limit, use a presigned POST with a content-length-range condition instead of a presigned PUT, or verify the object's real size after upload (via an S3 event notification or a scheduled cleanup job) and delete anything oversized.

Checksum/content validation — for anything security-sensitive, pass a ChecksumSHA256 to PutObjectCommand so S3 rejects an upload whose bytes don't match what the client claimed, and consider scanning uploaded files (e.g. with a virus-scanning Lambda trigger) before treating them as safe to serve.

Rate limiting — nothing above stops one authenticated user from requesting thousands of presigned URLs per minute. Put a rate limiter (Upstash Ratelimit, a simple sliding-window check in your database, or your edge/CDN's built-in limiting) in front of this route in production.

⚠️ Common Mistake: Making the bucket public "to make things easier." None of the common upload/display patterns need a public bucket — presigned URLs grant scoped, temporary access without exposing the bucket itself. Leave Block all public access on unless you have a specific, narrow reason to open one prefix.

Supabase Storage: S3 Underneath, Postgres on Top

Supabase Storage provides an S3-compatible object-storage interface and integrates it with Supabase Auth and Postgres RLS — every bucket can be protected by Row Level Security policies, the same SQL-based authorization system you're already using for your database tables if you're on Supabase.

Where Supabase Storage wins:

If you're already using Supabase Auth + Postgres, storage access rules live in the same mental model — a policy like "a user can only read their own files" is one create policy statement, not a separate IAM configuration
Simple, batteries-included API (upload, download, createSignedUrl) that works identically from the browser or the server
Offers on-the-fly image transformations included via a query-param API, so you don't always need a separate CDN layer for simple resizing, although availability and usage depend on your plan

Where Supabase Storage struggles:

Currently supports automatic WebP optimization but its transformation system is considerably narrower than Cloudinary's
Free-tier storage and egress are modest — fine for an MVP, tight for a media-heavy app
You're tied to Supabase's infrastructure choices; less low-level control than raw S3

A Minimal Supabase Storage Upload

npm install @supabase/supabase-js @supabase/ssr

// lib/supabase/client.ts
import { createBrowserClient } from "@supabase/ssr";

export function createClient() {
  return createBrowserClient(
    process.env.NEXT_PUBLIC_SUPABASE_URL!,
    process.env.NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY!
  );
}

// components/avatar-uploader.tsx
"use client";

import { createClient } from "@/lib/supabase/client";

export function AvatarUploader() {
  const supabase = createClient();

  const handleUpload = async (e: React.ChangeEvent<HTMLInputElement>) => {
    const file = e.target.files?.[0];
    if (!file) return;

    // Get the current user from the session — never take a userId as a
    // prop or form value. The path below is only as trustworthy as this.
    const {
      data: { user },
    } = await supabase.auth.getUser();

    if (!user) return;

    const filePath = `${user.id}/${Date.now()}-${file.name}`;

    const { error } = await supabase.storage
      .from("avatars")
      .upload(filePath, file, { upsert: false });

    if (error) {
      console.error("Upload failed:", error.message);
      return;
    }

    // For a private bucket, generate a temporary signed URL to display it
    const { data: signed } = await supabase.storage
      .from("avatars")
      .createSignedUrl(filePath, 60 * 60); // 1 hour

    console.log("Signed URL:", signed?.signedUrl);
  };

  return <input type="file" accept="image/*" onChange={handleUpload} />;
}

What's happening here: .storage.from("avatars").upload(...) uploads directly through the Supabase client — no separate presigned-URL round trip needed, because auth and authorization already ride along with the client's session. createSignedUrl mirrors S3's presigned-URL pattern, but scoped through the bucket's own RLS policies instead of IAM.

-- Only let a user manage files inside their own folder
create policy "Users can upload their own avatar"
on storage.objects
for insert
to authenticated
with check (
  bucket_id = 'avatars'
  and (select auth.uid())::text = (storage.foldername(name))[1]
);

What's happening here: storage.foldername(name) splits the object path (e.g. user-id/photo.jpg) into segments, so this policy checks that the folder name matches the uploader's own auth.uid(). The authorization model uses Postgres RLS, just like your database tables.

⚠️ Important — don't mistake a client-supplied path for real security. In the component above, user.id comes from supabase.auth.getUser(), which reads the server-verified session — that part is trustworthy. But the resulting filePath string is still just something the client constructs and sends along with the upload request. What actually makes this safe is the RLS policy, which independently re-checks (select auth.uid()) against the path on Supabase's side, every time, regardless of what the client sends.

This distinction matters: if you ever accept a user id as a raw prop, a hidden form field, or anything else that isn't derived from the verified session — or if RLS on the bucket is ever disabled or misconfigured — nothing stops a client from writing someone-elses-id/file.jpg into the path and having the request succeed. The policy is the actual security boundary here, not the fact that the code "usually" builds the path from the right value.

💡 Tip: Writing RLS policies by hand for every bucket pattern (owner-only, team-shared, admin override) gets repetitive fast. If you're already on Supabase, a Supabase RLS Policy Generator can produce the strict, performance-optimized version of these policies — including the (select auth.uid()) wrapper used above — for your specific access pattern.

Cloudinary: A Media Pipeline for Images and Video

Cloudinary's whole pitch is different from the other two: you're not really buying storage, you're buying transformation-on-delivery — and it's easy to think of Cloudinary as "an image tool," but that undersells it. Video is a first-class part of the product too: transcoding, adaptive bitrate streaming, and a dedicated video player component (CldVideoPlayer) ship alongside the image tooling, not as an afterthought. Upload one asset once, and every size, crop, and format you'll ever need — image or video — gets generated from URL parameters, with no separate resize job and no manual WebP conversion.

Where Cloudinary wins:

Transformations are wildly capable across both media types: resize, crop, auto-format (automatically selects an appropriate optimized format for the requesting browser), AI background removal, text/image overlays for images; transcoding and adaptive streaming for video
The next-cloudinary SDK gives you drop-in components — <CldImage> wraps next/image, <CldVideoPlayer> handles video — with almost no configuration
Built-in CDN delivery for both, so there's no separate CloudFront-equivalent to wire up

Where Cloudinary struggles:

Free tier (25 credits/month, where 1 credit ≈ 1 GB storage, 1 GB bandwidth, or 1,000 transformations) disappears fast on anything with real traffic — video in particular eats through credits quickly since files are larger
Pricing is credit-based across three different resource types at once, which makes forecasting cost harder than a flat per-GB rate
It's optimized for media-heavy applications; if most of your workload is arbitrary documents and backups, general-purpose object storage is usually a more natural fit

A Minimal Cloudinary Upload + Display in Next.js 16

npm install next-cloudinary cloudinary

// app/api/sign-cloudinary-params/route.ts
import { v2 as cloudinary } from "cloudinary";
import { NextResponse } from "next/server";
import { auth } from "@/lib/auth"; // your auth solution of choice

cloudinary.config({
  cloud_name: process.env.NEXT_PUBLIC_CLOUDINARY_CLOUD_NAME,
  api_key: process.env.CLOUDINARY_API_KEY,
  api_secret: process.env.CLOUDINARY_API_SECRET,
});

// The widget generates its own paramsToSign and sends them here — never
// sign that object blindly. Only a small, known set of keys is allowed
// through; anything else gets the whole request rejected.
const ALLOWED_PARAM_KEYS = new Set(["timestamp", "folder", "upload_preset", "source"]);

export async function POST(request: Request) {
  const session = await auth();
  if (!session?.user) {
    return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
  }

  const { paramsToSign } = await request.json();
  const keys = Object.keys(paramsToSign ?? {});
  const unexpectedKeys = keys.filter((key) => !ALLOWED_PARAM_KEYS.has(key));

  if (unexpectedKeys.length > 0) {
    return NextResponse.json(
      { error: `Unexpected upload parameters: ${unexpectedKeys.join(", ")}` },
      { status: 400 }
    );
  }

  const signature = cloudinary.utils.api_sign_request(
    paramsToSign,
    process.env.CLOUDINARY_API_SECRET!
  );

  return NextResponse.json({ signature });
}

What's happening here: the widget builds paramsToSign on its own based on how it's configured, and sends that exact object to your endpoint to be signed — so you can't simply substitute your own parameters the way you might for a normal API request, or the signature won't match what the widget actually uploads with. What you can do, and what this endpoint does, is refuse to sign anything that isn't on a small allowlist of expected keys, and require a logged-in session before generating a signature at all. This stops the endpoint from being used to sign arbitrary transformation instructions, moderation overrides, or unexpected metadata that a modified client request might try to sneak in. Once signed, the upload still goes directly from the browser to Cloudinary — your CLOUDINARY_API_SECRET never reaches the client.

💡 Tip: Signed uploads narrow who can request a signature, but the file itself — its folder, allowed formats, and max size — is still best constrained by an upload preset configured in the Cloudinary dashboard (Settings → Upload → Upload presets), since a preset's restrictions are enforced by Cloudinary itself and can't be bypassed by tampering with the request the way a loosely-validated paramsToSign object could be.

// components/cloudinary-uploader.tsx
"use client";

import { useState } from "react";
import { CldUploadWidget, CldImage } from "next-cloudinary";

export function CloudinaryUploader() {
  const [publicId, setPublicId] = useState<string | null>(null);

  return (
    <div>
      <CldUploadWidget
        signatureEndpoint="/api/sign-cloudinary-params"
        options={{ folder: "uploads" }}
        onSuccess={(result) => {
          if (result.info && typeof result.info === "object") {
            setPublicId(result.info.public_id);
          }
        }}
      >
        {({ open }) => (
          <button onClick={() => open()}>Upload an image</button>
        )}
      </CldUploadWidget>

      {publicId && (
        <CldImage
          src={publicId}
          width={600}
          height={400}
          crop="fill"
          alt="Uploaded image"
        />
      )}
    </div>
  );
}

What's happening here: <CldImage> renders the uploaded asset with a crop="fill" transformation applied automatically, on top of next/image's own optimization. Swap this component for <CldVideoPlayer> and the same signed-upload flow works for video uploads, using the same signing endpoint.

⚠️ Common Mistake: Using an unsigned upload preset (no signatureEndpoint, just a public preset name) for anything beyond a quick prototype. Anyone who knows the preset name can initiate uploads subject to the restrictions configured on that preset — fine for a personal demo, risky for anything public-facing.

Media Optimization: Who Actually Handles It?

This is usually the deciding factor in practice, so it's worth calling out directly:

S3 does nothing for you, for images or video. You either accept unoptimized originals, or build your own pipeline (Lambda@Edge, a CloudFront Function, a dedicated transcoding service, or routing S3 through Cloudinary/imgix as a second layer).
Supabase Storage offers a transformation API (resize + quality via query params) that works with a custom Next.js image loader — available only in Paid Plans, and images only; there's no equivalent video pipeline.
Cloudinary is purpose-built for exactly this, for both media types. Automatic format negotiation (AVIF/WebP per browser), on-the-fly cropping, <CldImage>'s tight next/image integration, and adaptive-bitrate video streaming mean you write the least code here.

If your app's core UX depends on fast, well-optimized media at scale — a marketplace, a photo-heavy blog, a portfolio builder, or a product with video uploads (course platforms, social apps, UGC video) — that alone can justify Cloudinary's cost. If media is incidental — a profile picture here, a document there — S3 or Supabase Storage is plenty.

Security Models Compared

	How access is controlled
S3	IAM policies (who can call the API) + bucket policies (what's publicly readable) + presigned URLs for temporary, scoped access
Supabase Storage	Row Level Security policies written in SQL, evaluated per-request against the logged-in user — the same system protecting your database tables
Cloudinary	Signed uploads (server signs the request, ideally after validating it) or unsigned upload presets (public, rate-limited by preset config)

The practical difference: S3 and Cloudinary both support signed requests for scoped upload/access workflows, while Supabase emphasizes identity-aware RLS policies. Supabase Storage centers on who is asking, evaluated through Postgres RLS policies. If your app already has rich, row-level permission logic in Postgres, Supabase Storage lets you reuse that thinking instead of maintaining a parallel IAM policy language — but in all three, a client-constructed path or parameter is never the actual security boundary on its own; the server-side check (RLS, IAM, or a whitelist) is.

Can You Mix Them?

Yes, and plenty of real apps do. A common pattern:

S3 (or Cloudflare R2, an S3-compatible alternative) for cheap, high-volume storage of originals — documents, raw uploads, backups
Cloudinary in front of just the media subset (images and video), either by uploading directly to Cloudinary or using its remote fetch feature to transform images that already live in S3
Supabase Storage for anything tightly coupled to your Supabase-authenticated user data, where RLS access rules matter more than transformation power

There's no rule that says "pick exactly one." Reach for the tool that fits that specific type of file, not a single storage layer for your entire app by default.

Decision Matrix

Your situation	Pick
Already deep in AWS infrastructure	S3
Need the cheapest storage at real scale, and can account for request/transfer costs	S3 / R2 depending on egress profile
Already using Supabase for auth + database	Supabase Storage
Want row-level access rules without a separate IAM setup	Supabase Storage
Images/video are core to your product's UX	Cloudinary
Want automatic format optimization with near-zero setup	Cloudinary
Storing mostly non-image files (PDFs, CSVs, backups)	S3
Small MVP, want the fastest path to "it works"	Supabase Storage if the app already uses Supabase;<br/>Cloudinary if images or video are central.

Frequently Asked Questions

Is Supabase Storage actually built on S3?

Supabase Storage is S3-compatible and exposes an S3 protocol endpoint, so tools that speak the S3 API can interact with it. The important distinction is that Supabase adds its own API, authentication integration, and Postgres RLS model.

Do I need Cloudinary if I'm already using next/image?

Not necessarily. next/image already handles resizing, format conversion, and lazy loading for images you serve from S3 or Supabase Storage — it just needs the source domain allowlisted in remotePatterns. Reach for Cloudinary specifically when you need transformations next/image doesn't do itself, like AI background removal, video transcoding, or on-the-fly cropping driven by user input.

Which one is cheapest for a small side project?

For a small MVP, Supabase Storage is attractive if you're already using Supabase, while Cloudinary is attractive when image or video transformations are central. S3 can also be inexpensive, but remember it's billed across storage, requests, and data transfer separately — factor in all three before assuming it's the cheapest option for your specific traffic pattern.

Is a presigned S3 upload URL alone enough security?

No. A presigned URL controls whether an upload to a specific key is allowed within its expiry window — it doesn't validate file type, size, or content on its own beyond what you sign into the request. Pair it with server-side checks (auth, MIME allowlist, size limits) before issuing the URL, as shown in this guide's example, and treat anything beyond that (checksums, virus scanning, rate limiting) as genuinely needed for production, not optional polish.

Can I switch between these later without a full rewrite?

Keeping storage access behind a small abstraction makes migration easier. The files themselves may be easy to migrate; the harder part is usually recreating URLs, access-control rules, transformations, and metadata.

Does Cloudinary replace the need for a CDN?

Yes — Cloudinary delivers every asset through its own CDN by default, so you don't need to add CloudFront or another CDN in front of it the way you would with raw S3.

What about Cloudflare R2 as another alternative?

R2's S3-compatible API makes many application patterns—such as presigned URLs and S3 SDK usage—very similar to S3, although its pricing and access-control model differ.

Wrapping Up

None of these three is a universally "better" choice — they're solving different halves of the same problem. S3 gives you the cheapest, most flexible raw storage and expects you to build the rest, including the parts of its own billing model (requests, transfer, retrieval) that are easy to overlook. Supabase Storage trades some of that flexibility for tight integration with the auth and database system you're probably already using — just remember that a client-supplied path is never the actual security boundary; the RLS policy is. Cloudinary trades storage cost for a transformation pipeline — for images and video — that would otherwise take real engineering time to build yourself.

Start by asking what you're actually storing and who needs to see it — that answer usually points to one of these three faster than a feature-by-feature comparison ever will.

Continue Learning

Free Developer Tools

Supabase RLS Policy Generator — generate strict, pattern-specific Row Level Security policies for your storage buckets, including the owner-only pattern shown in this guide.

Developer Stories: Jaafar on Building Reliable Systems for Arabic Newsrooms

Mon, 24 Aug 2026 01:25:35 +0000

Developer Stories: Jaafar on Building Reliable Systems for Arabic Newsrooms

Developer Stories is a CoderLegion series highlighting the real journeys, challenges, lessons, and experiences of developers building software in the real world.

For this edition, we spoke with Jaafar, founder of ManTek Technologies in Dubai, about his journey from studying IT engineering and AI at Damascus University to working with AWS, WordPress, and high-traffic publishing systems.

Rather than focusing on a traditional career timeline, this conversation explores something developers often learn only through experience: the difference between systems that appear to work and systems that can actually be trusted.

From silent failures and Arabic-language challenges to scaling WordPress and building NUZ, Jaafar shares lessons learned from working with production systems where mistakes can become visible to thousands—or millions—of users.

From Damascus University to ManTek

Jaafar studied IT engineering with a focus on AI, which he describes as involving a lot of statistics and not much of what he eventually ended up doing professionally.

His path into technology and media took him to the UAE, where he joined Al Khaleej Newspaper.

Publishing has a rhythm you do not appreciate from outside: it never pauses, the deadlines are real, and when something is slow or wrong it is wrong in public.

That experience led to work with a television channel in Dubai Media City, where the systems became considerably deeper and the scale considerably larger.

AWS was already part of the technology stack when he arrived.

Rather than learning cloud technology through tutorials or isolated projects, he learned it while working on systems carrying real traffic.

AWS was already the stack when I arrived there, so I learned it the useful way, on a system that was already carrying real traffic.

For Jaafar, that became one of the biggest additions to his career.

WordPress came later, and interestingly, the reason wasn't primarily technical.

Management had decided to move away from depending on in-house systems.

Bespoke platforms can be expensive to build and maintain, while also creating a significant learning curve whenever a new developer joins the team.

WordPress offered something different: a platform developers already knew and could extend.

I think that reasoning is underrated.

Eventually, ManTek Technologies grew out of Jaafar having enough experience to be useful on his own, combined with a backlog of ideas he had been carrying around for years.

At some point, building those ideas became more interesting than leaving them unbuilt.

When Systems Fail Quietly

One of the recurring themes in Jaafar's work is silent failure.

A system that crashes is usually obvious.

A system that confidently gives you the wrong answer can be much more dangerous.

Because a failing check tells you it failed. A blind one hands you a clean, confident, wrong answer and lets you act on it.

One example came while auditing structured data on his own website.

A profile URL appeared to be returning a "Page not found" response. After checking it in a real browser and running automated checks for common bot-blocking behaviour, everything appeared normal.

So he removed the URL from twenty-two pages and began preparing the commit message.

Then he discovered the problem.

The profile actually existed.

The platform was simply serving logged-out visitors a page saying that it did not.

Nothing had timed out. Nothing had crashed. The check had not reported an error.

It had accurately reported the page it had been given—even though that page itself was misleading.

The instrument had not errored. It had truthfully reported on the page it was given, which happened to be a page about not finding things.

Once he recognised the pattern, he found five more examples in the same week.

There was an API field answering a different question than the one he intended to ask. A list endpoint disagreed with its own item endpoint. An analytics beacon could not be detected with curl for two independent reasons.

There were also editors that accepted a save and appeared to complete the operation while nothing actually reached disk.

That last failure was particularly painful.

A fully reviewed Arabic translation—forty-eight changed lines—existed nowhere except inside a browser tab.

The important lesson wasn't that these problems were technically difficult.

They weren't.

The danger is not that these are hard to fix. Every one was trivial to fix. The danger is that nothing asks you to look.

The 404 That Wasn't

Building for Arabic

One of the biggest challenges developers can underestimate when building for Arabic is that many assumptions in software were never explicitly written down.

That means when those assumptions break, the system may not necessarily tell you.

Jaafar highlights several examples.

Arabic URLs and Byte Limits

WordPress caps a new URL slug at 200 bytes in core through utf8_uri_encode( $title, 200 ).

For English, that budget roughly corresponds to 200 characters.

Arabic is different.

Because Arabic characters can require approximately six bytes after percent-encoding, the same limit can translate to roughly thirty-two characters.

A normal Arabic headline can exceed that easily.

The result can be a URL silently truncated in the middle of a word.

Even more concerning, this can affect URLs that have already been published.

Search

Arabic search introduces another class of problems.

WordPress search uses LIKE '%term%' over the post table, comparing raw bytes.

But Arabic contains optional diacritics, multiple written forms of letters such as alef and hamza, and characters such as tatweel, which can stretch a word without changing its meaning to a reader.

On a real archive containing 250,000 articles, Jaafar compared a search for:

مُحَمَّد

The results were striking:

Search across 250,000 articles	Results	Time
WordPress	1	4.8 s
With normalisation and an index	28,238	0.55 s

One result instead of more than twenty-eight thousand.

And again, there was no obvious error.

The editors experiencing the problem didn't necessarily report that search was broken.

They reported that it was slow.

Slow is the part you can feel.

Rendering Arabic

The challenges don't stop at encoding and search.

Jaafar also encountered an issue while generating images containing Arabic text.

He kept anchoring the text to the wrong edge.

The relevant attribute was text-anchor, where start does not necessarily mean "left". It means the start of the reading direction—which, for Arabic, is the right.

The library was behaving correctly.

His intuition was wrong.

The general lesson: if you build for a language you do not read, your tests will share your assumptions. Test with real content from real users.

WordPress Arabic Search

Scaling WordPress

Jaafar has worked with genuinely large Arabic newsroom traffic, and his lessons about scaling WordPress are less about simply throwing more resources at the application.

He identifies three major lessons.

The spike is rarely the interesting problem

A 20x traffic surge on a news story can sound terrifying.

But much of that traffic consists of identical anonymous requests that can be handled by caching.

The more interesting problems are the requests that cannot be cached.

Search.

Logged-in editorial traffic.

And the write path that still needs to publish content while everything else is happening.

Jaafar also makes an important clarification about the often-mentioned 50,000 concurrent-user figure:

It is load-tested, not an observed peak.

The test demonstrates that the cacheable path can hold up.

It says much less about the newsroom itself—and the newsroom is the part that keeps you awake.

Background jobs fail quietly

WordPress cron is another potential weak point.

Because cron is triggered by traffic, putting WordPress behind a load balancer can result in cron running on every server simultaneously—or on none of them.

Scheduled posts can fail to publish without necessarily producing an obvious alarm.

"Nothing happened" is not an event.

This is why one of Jaafar's open-source plugins elects a single cron runner across a fleet and raises an alert when it stalls.

Move work away from WordPress

Media processing is a good example.

Serving and resizing images directly from the application means the application is doing work that doesn't necessarily belong there.

Moving that work to object storage and an edge layer means the origin server becomes less sensitive to traffic spikes.

After five years of working with large-scale Arabic newsroom traffic, Jaafar sees a consistent pattern:

Uptime comes from having less of the system in the request path, not from tuning what is in it.

The Importance of Observability

For Jaafar, observability is slightly different from simply having logging and monitoring.

A system can have both and still fail silently.

The question he cares about is:

Can a check distinguish "this is fine" from "I cannot see this"?

Many systems cannot.

An uptime monitor hitting a homepage can tell you that the homepage renders.

It cannot tell you whether scheduled posts were published, whether email actually left the system, or whether search returned the correct records.

He experienced this distinction on his own website when two analytics counters for the same site and period disagreed by roughly two orders of magnitude.

Neither system was broken.

Neither was lying.

They were simply answering different questions.

One counted every request reaching the edge, including bots.

The other counted browsers that actually executed the measurement script.

Only one represented the number of people reading the site.

Using the wrong number would have overstated readership by approximately eighty times.

Before you trust a number, know what it counts. That is observability too, and it is the part nobody writes runbooks for.

His approach is therefore to alert on outcomes users would actually notice rather than simply monitoring whichever signal is easiest to collect.

Failures should be loud by design.

One of his plugins, for example, refuses to send mail when it is misconfigured rather than silently falling through to a transport that appears healthy but delivers nothing.

Refusing is friendlier than pretending.

Open Source and Real-World Problems

Jaafar has now released five open-source projects, each connected to the real-world problem that led to it.

For him, the connection between the diagnosis and the fix is important.

The fixes themselves are often small.

For example, wp-arabic-slug-schema-guard addresses the byte-budget problem around Arabic WordPress slugs.

But the valuable part isn't just the code.

It is understanding why the problem exists.

WordPress truncates at 200 bytes.

Arabic can consume approximately six bytes per character.

And published slugs can potentially be rewritten underneath an existing site.

A fix without that context is a snippet nobody trusts. The context without a fix is a complaint. Publishing both together is the only version that is actually useful to the next person.

The plugins are deliberately implemented as must-use plugins rather than ordinary plugins.

The reason is that infrastructure such as cron election and mail transport should not be something that can simply be switched off from an administrator screen.

There was even a lesson in the packaging of the plugins themselves.

Several plugins declared:

"type": "wordpress-muplugin"

so that Composer could install them into the correct directory.

But two of them did so without requiring composer/installers, the package responsible for interpreting that type.

In projects that already included the dependency, everything worked.

Elsewhere, Composer reported a successful installation but placed the plugin in vendor/, where WordPress never loaded it.

The installation succeeded.

The plugin simply did nothing.

Which was, ironically, exactly the kind of silent failure the plugins were designed to prevent.

What I learned from releasing them: the environment is the bug, almost every time.

Across six rounds of adversarial testing on one of the projects, every bug discovered involved either a check that could not fail or a silent wrong answer.

And none of those bugs were found simply by reading the code.

ManTek Technologies on GitHub

Building NUZ

Jaafar is currently developing NUZ, a cloud-native publishing platform aimed at high-traffic newsrooms.

He is careful to clarify its current status:

NUZ is pre-release. I am building the core, and there are no external partners on it yet.

That distinction matters.

The project is not being presented as a platform already running with external partners. It is still being built.

The problem NUZ is intended to address is the way many publishing platforms approach scale.

A newsroom might begin with a CMS designed for a blog.

Then caching is added.

Then a CDN.

Then a media pipeline.

Then a search layer.

Each addition introduces another component and another potential failure mode.

Eventually, the newsroom is maintaining an architecture that nobody originally designed as a whole.

NUZ starts from a different assumption:

The traffic is spiky, the content is Arabic, and the editorial team publishes continuously while all of that is true.

Engineering in the Real World

When working with production systems, clever solutions are not always the best solutions.

Jaafar learned this through experience.

One early fix involved copying a core function, changing one line, and shipping it.

It worked.

Then the underlying platform changed that function three times across three releases.

His copy quietly diverged from the original implementation.

Eventually, the bug that returned looked nothing like the problem he had originally fixed.

The better solution was to use a documented filter and replace only the transport.

No fork.

No duplicated implementation.

No code that needed to be kept in sync with the platform.

When the platform gives you a seam, take the seam. Never fork.

The broader lesson is simple:

Prefer the boring integration point that the platform promises to keep working over the clever one that happens to work today.

Advice for Developers

When asked what habit developers should develop to become better at building reliable systems, Jaafar offered a surprisingly simple answer.

Before you accept that a check found nothing, point it at something you know is there.

The idea is straightforward.

If you are checking whether a value appears in a file, first test the check against a file where you know the value exists.

If you are verifying that a record is absent, first query a record you know exists.

The goal is to establish that the check itself is capable of finding what it is supposed to find.

Because sometimes the most dangerous result isn't an error.

It is a perfectly clean "nothing found."

Final Thoughts

Jaafar's journey is a good example of how real engineering experience is often built in places that tutorials don't cover.

It is built in production systems.

In traffic spikes.

In unexpected platform behaviour.

In multilingual content.

In failed background jobs.

In analytics numbers that answer the wrong question.

And especially in the quiet failures that look like success.

His approach is ultimately less about making systems look impressive and more about making them honest.

A system should tell you when it doesn't know.

A check should prove that it can actually see what it claims to see.

And when a platform provides a supported way to extend it, use that instead of building something that will quietly drift away from the original system.

For developers working with real users and real production systems, those lessons can be worth far more than another tutorial.

About Developer Stories

Developer Stories is a CoderLegion series featuring developers from our community and exploring the real journeys behind the code—the challenges they have faced, lessons they have learned, projects they are building, and experiences that can help other developers grow.

If you have a developer journey, engineering lesson, project, or experience you'd like to share with the CoderLegion community, we'd love to hear from you.

Setting Up Your AWS Builder ID & Connecting to Skill Builder

Sun, 23 Aug 2026 09:10:40 +0000

An AWS Builder ID is your personal profile used to access AWS developer and learning tools like the AWS Builder Center, Amazon Q Developer, and AWS Skill Builder without a paid corporate AWS account.

Phase 1: How to Create an AWS Builder ID

Step 1: Create Your AWS Builder ID

Go to the Sign-In Page: Head over to the AWS Builder Center
Click Sign In / Register: Select the option to sign in or create an AWS Builder ID.
Choose Your Sign-Up Method:

Option A (Fastest): Choose Continue with Google, Apple, or GitHub to link an existing provider.
Option B (Email): Enter your personal email address and click Next.

4. Verify Your Email: Check your inbox for a verification code sent by AWS. Enter the code on the screen and click Verify. (Do this if it gives u an error)

Step 2: Complete Your Profile Setup

Complete your profile by choosing ur alias(username); it will be used as ur builder ID too, and choose a location.

Select "Yes, I am a student" when prompted to configure your student-specific profile path.

Fill out your remaining personal details, upload a profile photo, and claim your custom Builder Center alias!

Step 3 (Optional but highly recommended): Verify Your Student Status & Unlock Pro Features 💥

Want to get the most out of your cloud journey? You can verify your student status using SheerID to gain access to Pro features and rewards for a full year!

How to do it: Navigate to the student verification section within your Builder Center profile or click on the one on your home page.

What you'll provide: Basic information like your school name and school-issued email or documentation.

The Reward: Unlocks massive perks like an annual Skill Builder subscription, giving you hands-on labs, 900+ digital courses, and certification preparation tools completely free!

1. Learn

Topics: Explore specialized technical domains and modern cloud architectures (such as Generative AI, DevOps, Security, and Serverless).
Students: Dedicated pathways, starter content, and foundational resources structured specifically for academic learners.

2. Build

Capabilities: Guides, documentation, and architecture patterns detailing how to implement specific AWS services.
Toolbox: Developer tools, Software Development Kits (SDKs), Command Line Interface (CLI) integrations, and deployment kits.
Workshops: Hands-on, step-by-step tutorials designed to help you build real, production-style projects.

3. Connect

Events: Information on virtual and local webinars, hackathons, and global AWS community meetups.
Spaces: Topic-based discussion forums (like the Student Builders or AI & ML Scholar spaces) where you can share code, ask questions, and collaborate.

4. Community

Heroes: Recognized global experts who share deep AWS knowledge and open-source projects.
Community Builders: Technical enthusiasts who publish content, write guides, and mentor peers.
User Groups: Regional, self-starter cloud user networks meeting worldwide.
Student Builder Groups: Campus-based student clubs designed to connect peers, host events, and build practical cloud projects together.

5. Wishlist & Explore AWS

Wishlist: A feedback platform where builders can submit feature requests, suggest tool upgrades, and vote on ideas to directly influence the AWS product roadmap.
Explore AWS / Skill Builder: Quick links to support articles, open-source projects, startup resources, and the AWS Skill Builder training ecosystem.

💡 Want the Full complete guide?

Read the complete, interactive guide complete with all screenshots, quick tips, and extra community resources directly on my official Hashnode blog:
👉 Read the Full Article on Hashnode

AWS Is Betting $500M That Students Will Learn Cloud By Building. The Payoff Is Still Unproven.

Fri, 21 Aug 2026 00:04:25 +0000

Every few years, a major cloud provider makes a play for the next generation of developers. AWS tried it before with AWS Educate, a dedicated academic program that eventually shut down. This week, the company is trying again, with more money and a different theory of what went wrong the first time.

AWS announced a commitment of more than $500 million in resources aimed at university students worldwide, delivered through a new Student Rewards program inside AWS Builder Center. Verified students can earn up to $579 in value: a year of premium AWS Skill Builder access (a $449 value covering 900-plus courses and hands-on labs), $30 in AWS Credits, and a foundational certification exam voucher worth $100. I asked Rick Suttles, Senior Manager, Product & Engineering for Builder Experience at AWS, to walk through what's actually new here, and where the open questions still sit.

What AWS says it learned from Educate

The most telling answer in the whole exchange is the one about the program AWS already tried and abandoned. "Educate was a separate, siloed experience," Suttles said. Student Rewards, by contrast, lives inside AWS Builder Center, the same community working developers already use. Students build alongside professionals, using the same tools, in the same space, rather than in a walled-off academic sandbox. AWS also cut the application process students previously had to clear. Now it's straightforward: verify enrollment, start building.

Whether that structural change actually produces better outcomes than Educate did is impossible to know yet. But it's a specific, testable claim, not a case of AWS simply doing the same thing again with a bigger number attached.

The credential doesn't expire, but AWS won't promise a job

Suttles was clear that whatever a student earns during the 12 months, certifications and completed projects, stays theirs after the program access ends. That's a real, concrete commitment. What AWS wouldn't commit to is what happens after: renewal paths beyond the first year are still being worked out, and Suttles said the company will share more "as we learn how students are using the resources."

The bigger gap is on outcomes. Asked directly whether AWS has data on how many students who complete the certification path actually land cloud or AI roles, Suttles didn't have numbers to share, since this cohort is just getting underway. Asked what number AWS would point to a year from now as proof the program worked, he named "students building things" as the leading indicator, with certifications as a secondary signal, on the theory that career outcomes follow from real project work rather than credentials alone. It's a reasonable bet, but it's a bet. There's no employer commitment on record treating this specific credential as a guaranteed hiring signal beyond the general observation that AWS certifications are already recognized in the industry.

Worth being straightforward about that gap: this is a brand-new program, and AWS not having placement data yet is expected, not damning. But it does mean the actual return on time invested is still an open question for any student deciding whether to commit to it.

Global access, with the exclusions spelled out plainly

On equity, the answers were more concrete. Eligibility isn't tiered by institution type. Community colleges and technical schools get the same access as research universities, and no prior AWS relationship is required. AWS also pointed to more than 1,000 existing Student Builder Groups spanning over 60 countries, many at schools outside the research-university tier, with the option for students to start a new group if their campus doesn't have one.

Internationally, Suttles said the reward value is identical everywhere verification is available, which currently excludes only sanctioned regions, with China, Hong Kong, and Taiwan verification coming soon. That's a cleaner answer than most global tech benefit programs manage to give, where regional pricing or partial access is often buried in fine print.

The single-vendor question, answered directly

The most obvious criticism of any vendor-run student program is that it trains people to use one company's tools rather than teaching transferable skills. Suttles didn't dodge it: his argument is that cloud fundamentals, distributed architecture, infrastructure as code, AI services, transfer across environments regardless of which vendor's console a student learned them in first. That's a defensible position, though it's also exactly what every cloud vendor running a program like this would say. Worth judging for yourself once you see what the coursework actually covers.

On that front, the AI-specific material isn't just cloud fundamentals with a new label. Skill Builder includes dedicated courses on generative AI, prompt engineering, and model training, alongside services like Amazon Bedrock and Amazon SageMaker, on top of the standard networking, security, compute, and storage curriculum.

A wrinkle worth noting: the content requirement

Part of how students unlock the full reward value is by publishing technical content, project write-ups and tutorials, on Builder Center and engaging with the broader community. Suttles framed this as reinforcing a student's own learning while building a public portfolio and a network of fellow builders. It's also, functionally, a way for AWS to generate a steady stream of community-authored technical content at essentially no cost, which isn't a criticism so much as worth naming plainly for what it is.

What this actually means if you're deciding whether it's worth a student's time

The structural changes from Educate are real and specific enough to take seriously: integration into a live professional community, no application gate, global parity, no institutional gatekeeping. The open question is whether "students building things" as a success metric actually translates into the career outcomes AWS is implicitly promising. Right now, that's a bet AWS is asking students to take alongside them, backed by real resources but not yet by real placement data.

Measuring whether semantic chunking actually helps, on one fixed corpus

Tue, 18 Aug 2026 16:08:46 +0000

Everyone repeats that semantic chunking beats fixed-size chunking for RAG. I wanted to know if that was true for my corpus, or just a thing people say. So I built the comparison into EvalBench: one corpus, one query set, three chunking strategies, and a single precision number I could read off a leaderboard.

The result I care about here is not a benchmark score. It is the setup. Getting a fair comparison between chunkers is harder than writing the chunkers, because almost everything that makes retrieval look good or bad has nothing to do with how you split the text.

The trap: comparing chunkers by comparing pipelines

If you swap the chunker and also swap the embedder, or the corpus, or the top-k, you have not measured the chunker. You have measured a different pipeline. Most "semantic wins" posts do exactly this, and you cannot tell how much of the win is the split versus everything else that moved with it.

So in EvalBench the only thing that varies is the split. A RAG run is identified by an embedder::chunk_strategy string, parsed apart before anything runs:

def parse_pipeline_model(model: str) -> tuple[str, ChunkStrategy]:
    """Parse the exact ``embedder::chunk_strategy`` row identifier."""
    ...
    if strategy not in _ALLOWED_STRATEGIES:
        raise ValueError(f"unknown chunk strategy {strategy!r}")

Same embedder, same 200-document corpus, same 15 queries across finance, legal, medical, physics, and software. The strategy is the one knob. Everything downstream is shared code.

The three strategies

There are three, not two. The interesting comparison is not fixed versus semantic, it is fixed versus recursive versus semantic, because recursive is what most people actually ship.

fixed_512 is the dumb one. Whitespace-tokenize the whole document, cut it into 512-token windows with a 64-token overlap, done. It respects nothing about the text.

def _window_token_groups(tokens, window=512, overlap=64):
    groups, start = [], 0
    while start < len(tokens):
        end = min(start + window, len(tokens))
        groups.append(list(tokens[start:end]))
        if end == len(tokens):
            break
        start = end - overlap
    return groups

recursive splits on structure first. Try paragraph breaks, then single newlines, then sentence boundaries, and only fall back to a hard token cut when a piece is still too big. Then it repacks those units into bounded chunks. This is the LangChain-style splitter, and it is a real baseline, not a strawman.

semantic is the one people get excited about. Embed every sentence, walk them in order, and start a new chunk when the meaning drifts. Drift is measured as cosine similarity between adjacent sentences dropping below a threshold:

low_similarity = (
    len(current_sentences) >= 3
    and cosine_similarity(vectors[index - 1], vectors[index]) < 0.65
)
too_large = len(current_tokens) + len(sentence_tokens) > 512
if current_tokens and (too_large or low_similarity):
    flush()

Two things in that snippet matter and neither is obvious from the marketing. The >= 3 guard means a chunk needs at least three sentences before a similarity drop is allowed to break it, so one weird sentence early on does not shatter a chunk into fragments. And the too_large check means semantic still has a hard 512-token ceiling. So it is not really "split on meaning." It splits on meaning until it hits the same 512-token window fixed_512 uses, then cuts. That ceiling is what keeps the comparison fair. If semantic could emit arbitrarily long chunks, it would win on precision for a boring reason: bigger chunks catch more gold documents.

The metric has to be simple enough to trust

I score every strategy on context precision. Take the top 10 ranked chunks, count how many come from a document in the gold set, divide:

retrieved_chunks = ranked_chunks[:10]
context_precision = (
    sum(chunk.doc_id in gold_doc_ids for chunk in retrieved_chunks)
    / len(retrieved_chunks)
    if retrieved_chunks
    else 0.0
)

That is deliberately crude. It counts a chunk as relevant if its source document is relevant, not if the chunk text itself answers the query. A stricter metric would grade chunk content, but then I would be measuring my grader as much as my chunker, and I already have a separate faithfulness judge for content quality. For a chunker comparison, "did we pull chunks from the right documents" is the question, and document-level precision answers it without dragging an LLM into the measurement.

Every metric is clamped to [0, 1] with a hard failure if it lands outside, so a broken chunker that emits duplicate or malformed chunks throws instead of quietly publishing a nonsense score:

def _clamp_metric(value):
    tolerance = 1e-12
    if not math.isfinite(value) or value < -tolerance or value > 1.0 + tolerance:
        raise ValueError(...)
    return float(min(1.0, max(0.0, value)))

What broke, and what I would do differently

The gold labels are per document, not per chunk. That is the real limitation here. If a relevant document is 40 chunks long and only 2 of those chunks actually answer the query, my precision metric happily credits all 40 as relevant when they surface. So this setup answers "does this strategy retrieve from the right documents," and it is blind to whether it retrieves the right passage inside them. For a first pass on a 200-document corpus that is a reasonable trade, but it means a chunker that fragments a relevant document into many pieces can look better than one that pulls a single tight passage, purely because more of its chunks share a gold doc_id. Chunk-level labels would fix this and are the obvious next step.

The other thing I would change is the sample size before trusting any ranking. 15 queries, 3 per domain, is enough to see a large effect and nowhere near enough to call a small one. EvalBench already publishes Wilson intervals next to every mean for exactly this reason, so the leaderboard shows the uncertainty instead of hiding it. If two strategies land within each other's intervals at n=15, the correct read is "no measured difference," not "semantic wins by a hair." I would rather report a wide interval than a confident wrong number.

Takeaway

If you want to know whether semantic chunking helps, hold the embedder, corpus, queries, and token ceiling fixed, and vary only the split. Then read the precision with its confidence interval attached. A comparison where three things moved at once tells you nothing about the one you changed.

Deploying a Microservices E-Commerce Application on AWS Using Shell Scripting

Tue, 04 Aug 2026 10:45:47 +0000

This article was originally published on Hashnode and is being republished here.

It covers deploying a microservices-based e-commerce application on AWS using Shell Scripting, including server setup, service configuration, and application deployment.

Over the past few days, I learned Shell Scripting and decided to apply everything I had learned by building a complete project from scratch to gain real hands-on experience.

My goal wasn't just to deploy the RoboShop e-commerce application on AWS using Shell Scripting. I also wanted to understand what happens behind the scenes—how services start, how they communicate with each other, how to automate repetitive deployment tasks, and most importantly, how to troubleshoot issues when things go wrong.

Throughout this project, I learned how to debug deployment issues by analyzing HTTP status codes, checking Nginx logs, application logs, and database logs, identifying whether the problem was in the upstream or downstream service, and writing proper deployment logs to make troubleshooting easier.

More than the automation itself, the debugging experience gave me strong hands-on knowledge of application deployment, troubleshooting, and real-world infrastructure automation.

What is the RoboShop E-Commerce Project?
RoboShop is a microservices-based e-commerce application consisting of around 10 services, including Frontend, Catalogue, User, Cart, Shipping, Payment, MongoDB, MySQL, Redis, RabbitMQ, and Dispatch. Each service has its own responsibility and communicates with other services to complete the order flow.

This project is a great way to understand how multiple services are deployed, connected, and managed in a real-world environment.

Application Components
The RoboShop application is built using multiple technologies, with each service having a specific responsibility.

Service and Technology
Mongodb (NoSQL Database)
Catalogue (Node.js)
Redis (Caching Server)
User (Node.js)
Cart (Node.js)
Mysql (SQL Database)
Shipping (Java)
Rabbitmq (Message Broker)
Payment (Python)
Dispatch (Go (Golang))
Frontend (Nginx (Reverse Proxy))

How Requests Flow
The Frontend uses Nginx as a reverse proxy. Instead of clients communicating directly with backend services, every request first reaches Nginx. Based on the requested URL, Nginx forwards the request to the appropriate backend service.

location /api/catalogue/ {
    proxy_pass http://catalogue:8080/;
}

location /api/user/ {
    proxy_pass http://user:8080/;
}

location /api/cart/ {
    proxy_pass http://cart:8080/;
}

location /api/shipping/ {
    proxy_pass http://shipping:8080/;
}

location /api/payment/ {
    proxy_pass http://payment:8080/;
}

Using a reverse proxy provides several benefits:

Clients communicate only with Nginx, not directly with backend services.

The actual backend server details remain hidden.

Nginx routes requests to the correct service based on the URL.

It also makes it easier to manage load balancing, SSL termination, and routing from a single entry point.

Deployment Approach
Although the services are built using different technologies such as Node.js, Java, and Python, the deployment process is almost the same for every service.

The deployment flow includes:

Installing the required runtime
Creating the application user and directory
Downloading and extracting the application
Installing application dependencies
Configuring a systemd service
Loading the database schema (where required)
Starting and enabling the service

Since this workflow was common across almost every service, I moved all the reusable functions into a single common.sh file and reused it across all deployment scripts. This reduced duplicate code and made the scripts much easier to maintain.

What I Automated
During this project, I automated the following tasks:

✅ Validated every command using Linux exit codes (0 = Success, non-zero = Failure) to stop execution immediately if a step failed.

✅ Implemented idempotency, allowing the scripts to be executed multiple times safely.

✅ Moved all reusable deployment logic into a common common.sh file.

✅ Automated application deployment for all services, including runtime installation, dependency installation, and systemd configuration.

✅ Automated EC2 instance creation using AWS CLI.

✅ Automated Route53 DNS record creation using AWS CLI.

✅ Generated detailed log files for every deployment step to simplify troubleshooting.

✅ Performed root user validation before executing privileged operations.

Instead of repeating the same steps for every server, I automated the entire deployment using Shell Scripting. This made deployments consistent, repeatable, and much faster.

Debugging Challenges
This was the most interesting part of the project and where I learned the most. Writing the scripts was only one part, Troubleshooting the issues taught me how to debug deployments by following logs and tracing requests step by step instead of guessing.

After deploying the application, I expected everything to work. However, when I opened the application, the catalogue products were not loading.

Here's how I approached the issue:

Checked the browser's HTTP status code and found a 504 Gateway Timeout.

Looked at the Nginx access.log on the frontend and found that
=> Nginx forwarded the request to the upstream (backend) service.

=>The upstream service did not return a response within the configured timeout. (waiting for something to get from database)

Verified that the Catalogue service was running using systemctl status

Checked the Catalogue service logs using: journalctl -u catalogue -f

The logs showed that the Catalogue service couldn't connect to MongoDB.

Logged into the MongoDB server and found that the MongoDB service was stopped.

Started the MongoDB service, and the Catalogue service immediately began working.

💡What I learned:
This issue taught me not to jump to conclusions or restart services blindly. Instead, I learned to follow the request flow—from the browser, to Nginx, to the application logs, and finally to the database—to identify the actual root cause.

This debugging process gave me much more confidence in troubleshooting real deployment issues and understanding how different services interact in a microservices environment.

🚀 What's Next?
In the next part of this project, I'll automate the application configuration using Ansible, a configuration management tool. This will help eliminate manual server configuration and make deployments even more consistent and scalable.

🎉 Final Thoughts
This project helped me turn my Shell Scripting knowledge into practical experience. Along the way, I learned a lot by intentionally stopping services, analyzing logs, and troubleshooting the resulting issues.I gained valuable hands-on experience in automation, deployment, and debugging. I'll continue building more projects to strengthen my practical skills and gain even more real-world experience.

If you're interested in exploring the implementation, feel free to check out the complete project source code from below.
https://github.com/pavankumar0816/shell-roboshop-common

I am including a few screenshots of the deployed application:

What happens when your migration agent dies at file 7 of 12

Fri, 31 Jul 2026 16:57:25 +0000

A migration agent spends real money and real minutes per run. It clones a repo, installs its deps, plans a file order, then rewrites files one at a time, running the target's test suite after each one. On a mid-sized repo that is a lot of Bedrock calls and a lot of wall-clock time.

So the question I had to answer early: what happens when the process dies at file 7 of 12?

If the answer is "start over," the agent is a toy. You have already paid for six migrated files and a passing test suite on each, and a spot reclamation or a redeploy throws all of it away. Worse, the API side runs on Lambda, which has a hard 15-minute ceiling. A real migration blows past that. So the long-horizon work cannot live where the request comes in, and the thing doing the work has to survive being killed.

Here is how RepoModernizer handles it, and the bug that resume introduced that I did not see coming.

The split: API on Lambda, worker on Fargate

The API is a FastAPI app behind API Gateway, running on Lambda through Mangum. It takes the request, writes nothing important, and drops a message on SQS. That is it. It never runs the migration.

An SQS consumer Lambda reads the message and calls ecs.run_task to launch a one-shot Fargate task. The Fargate task is the worker. It runs the LangGraph pipeline to completion (or until it dies) and then exits. Nothing stays warm. Idle cost is zero because there is nothing idle.

# consumer_handler.py — Lambda that turns an SQS message into a Fargate run
_ecs.run_task(
    cluster=os.environ["ECS_CLUSTER"],
    taskDefinition=os.environ["ECS_TASK_DEFINITION"],
    launchType="FARGATE",
    overrides={
        "containerOverrides": [{
            "name": "worker",
            "environment": [
                {"name": str(k).upper(), "value": str(v)}
                for k, v in body.items() if v is not None
            ],
        }]
    },
)

The whole reason the worker is on Fargate and not Lambda is that 15-minute cap. Lambda is fine for the request. It is not fine for a job that might run half an hour.

Two kinds of durable, in two different places

The worker has two things it cannot afford to lose when it dies: the repo it is editing, and the graph state that says how far it got. These live in two different stores, on purpose.

The repo lives on EFS, mounted into the Fargate task at /mnt/workspace. The task clones the target repo there, makes a branch, and every diff it applies lands on that filesystem. When a new task picks up the same job, the files are already sitting there in the state the last run left them.

The graph state lives in DynamoDB. Every node the LangGraph pipeline finishes gets written as a checkpoint. That is what lets a fresh process know it already ingested, already installed deps, already migrated files 0 through 6, and should resume at file 7.

Splitting them matters. DynamoDB holds the plan, the cursor, and the per-file status. EFS holds the actual bytes. Neither one alone is enough to resume. You need the state to know where you were and the filesystem to have the work you already did.

The checkpointer is a real LangGraph saver, not a side table

LangGraph has a checkpointer interface, BaseCheckpointSaver. The out-of-the-box options are an in-memory saver (useless across processes) and integrations you may or may not want to run. I wrote one backed by DynamoDB directly, because DynamoDB was already in the stack and I wanted single-table access patterns I controlled.

The core is put and get_tuple. put serializes the checkpoint with LangGraph's own JsonPlusSerializer and writes it under a per-task partition key:

def put(self, config, checkpoint, metadata, new_versions):
    thread_id = config["configurable"]["thread_id"]
    checkpoint_id = checkpoint["id"]
    parent_id = config["configurable"].get("checkpoint_id")
    self._put_blob(
        f"TASK#{thread_id}", f"CKPT#{checkpoint_id}", (checkpoint, metadata),
        extra={"parent_checkpoint_id": parent_id or ""},
    )
    return {"configurable": {
        "thread_id": thread_id,
        "checkpoint_ns": config["configurable"].get("checkpoint_ns", ""),
        "checkpoint_id": checkpoint_id,
    }}

get_tuple does the reverse. Given a task id and no specific checkpoint, it queries the latest one:

resp = self._table.query(
    KeyConditionExpression=Key("PK").eq(f"TASK#{thread_id}")
        & Key("SK").begins_with("CKPT#"),
    ScanIndexForward=False, Limit=1,
)

ScanIndexForward=False, Limit=1 gives the most recent checkpoint for that task. On cold start the worker builds the graph with this checkpointer attached, and LangGraph replays from the last saved node. Every checkpoint carries a 14-day TTL so old task state ages out of the table on its own.

The graph build is where the checkpointer gets wired in:

graph = build_graph(deps, checkpointer=checkpointer)
config = {"configurable": {"thread_id": task_id}}

thread_id is the task id. That single value ties a resuming process back to everything the previous process wrote.

What "resume" actually does, and what it does not

I want to be precise here, because it is easy to oversell this.

Resume works at the node level, not the byte level inside a file. If the worker dies while migrating file 7, it does not continue from the middle of file 7. On the next run, the loop restarts the migrate_file node for the current cursor from the top: it re-reads the file off EFS, asks the model for fresh content, applies the diff, and reruns tests. Files 0 through 6 are already marked done in the checkpointed state, so it does not touch them. It picks up at the current cursor and moves forward.

So "resumes mid-migration" is true in the sense that matters (you do not redo six files of completed, test-passing work), and false in the sense that a single interrupted file gets redone from scratch. That is the honest version.

The bug: resume, done twice, opens the same PR six times

The pipeline has a human-in-the-loop gate. When the planner scores a file's risk above a threshold, the migrate_file node calls LangGraph's interrupt() with the diff, and the run pauses. A human approves or rejects. Approval comes back in as a separate action that resumes the graph with Command(resume=...).

Here is the thing I missed. That migrate_file node re-executes from its top on every resume. Re-reads the model, re-applies, re-runs tests. That is fine when there is genuinely a pending interrupt to satisfy. It is not fine when the resume action fires twice.

And it fires twice more often than you would think. SQS can deliver a message more than once. A user can click approve again while a slow Fargate cold start has not yet updated the checkpoint, so the UI still shows a pending decision. Either way, a second resume lands for a task whose interrupt was already cleared by the first one.

Calling invoke() again does not no-op. It runs the node fresh, produces a new diff, applies it, and drives the run all the way to finalize again, which opens a pull request. I found this live: one approve click produced six to nine identical commits on the PR.

The first fix was to check whether there is actually a pending interrupt before resuming:

snapshot = graph.get_state(config)
has_pending_interrupt = any(t.interrupts for t in snapshot.tasks)

That helped. It did not fix it. Two approve calls sent about a second apart still produced one extra commit, not zero. The reason is that get_state then invoke is check-then-act. Two callers landing in the same instant both read has_pending_interrupt = True before either one resumes. Classic race.

The real fix is atomicity, and DynamoDB is the only thing in this system that offers it. I added a try_claim that does a conditional put: the write only succeeds if the item does not already exist.

def try_claim(self, task_id: str, key: str) -> bool:
    try:
        self._table.put_item(
            Item={"PK": f"TASK#{task_id}", "SK": f"CLAIM#{key}",
                  "ttl": int(time.time()) + _TTL_SECONDS},
            ConditionExpression="attribute_not_exists(PK)",
        )
        return True
    except ClientError as exc:
        if exc.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

The key is the checkpoint id being resumed. Only the first caller to claim that specific checkpoint proceeds to invoke. The second gets False and returns the existing state without re-running anything.

claimed = checkpointer.try_claim(task_id, f"resume:{checkpoint_id}")
if has_pending_interrupt and claimed:
    result = graph.invoke(Command(resume={...}), config=config)
else:
    result = snapshot.values

There is a second guard on the far side too, at finalize: before opening a PR, check whether this task already has a PR URL recorded, and bail if it does. Belt and suspenders, because the cost of a duplicate PR is annoying and public.

The other thing that broke: git, EFS, and root

Two smaller landmines, both worth knowing if you mount EFS into a container that runs git.

The EFS access point pins every file to uid 1000 no matter who writes it. The container runs as root. Git 2.35.2 and up refuses to operate on a repo it sees as owned by a different user (that is the CVE-2022-24765 dubious-ownership check), so the first git command after the clone blows up. Fix is one line at worker startup:

subprocess.run(["git", "config", "--global", "--add", "safe.directory", "*"], check=True)

The second one showed up only on JS repos. The diff step normalizes the target file to end in a newline before diffing. git apply matches context against real bytes, and difflib never emits the "No newline at end of file" marker, so a file without a trailing newline fails to apply cleanly. Python source almost always ends in a newline, so my Python fixtures never caught it. Hand-written JS and JSON often do not, and it broke against a real JS repo.

What I would do differently

The duplicate-commit bug is the interesting one, and the lesson is not "add a claim." The lesson is that I built idempotency in the wrong layer first. I reached for a state check (has_pending_interrupt) when the problem was concurrency, and a state check cannot solve concurrency. Only a conditional write can. If I were doing it again I would put the atomic claim in from the start, at every resume boundary, and treat the state check as a cheap early-out rather than the guarantee.

The other thing: migrate_file re-running from the top on resume is convenient but wasteful. It re-pays for a Bedrock call on a file it may have already generated. A cleaner design caches the generated content against the checkpoint so a resume that is really just a duplicate delivery costs nothing. I have not built that yet.

Durability is not one feature. It is durable state, durable storage, and idempotent resume, and the last one is the one that bit me.

Can't Hire a Security Team? A Practical Guide to Reducing Your AWS Exposure

Fri, 31 Jul 2026 12:58:29 +0000

Many SaaS startups or small companies that utilize cloud technologies face a dilemma: whether to invest in cybersecurity and compliance now or delay until the company begins proving itself.

More often than not, the company may lack a full-time employee or team who specialize in these types of questions due to financial factors. This is why SMBs usually delegate security to the sysadmin and compliance to engineering managers. Unfortunately, despite the fact that security assessments and compliance checks may seem simple and basic, they still require dedicated professionals to implement properly due to the depth of knowledge and experience required.

According to research, many SMBs assume "We're Too Small to Be Targeted".

As of 2026, one of the top concerns that SMBs face is cybersecurity incidents. A hurdle that SMBs face is that hiring a security team or a consulting company can cost anywhere from a few thousand US dollars up to hundreds of thousands per year, sometimes wiping out a startup's revenue during early development.

A simple, practical approach to this problem is to adopt a security-by-design approach in your processes and workflows. This may seem a bit annoying at the beginning for your teams, but it will save you a lot of headaches in the long term and reduce the amount of rework required later on when you finally hire a consulting company to address any significant gaps or risks.

Web Applications Security

The web application is one of the top attack surfaces since this is what faces users or external parties via the cloud. There are many frameworks and standards, but Microsoft's SDLC is one of the most well-known processes for securing web applications. If you're a SaaS startup, then it's highly recommended to first secure the front line before beginning to talk about cloud security. Other standards worth checking out are the OWASP Top 10 for web applications.

AWS Security Checklist

Now, once you have got the front line secured, the next logical step is to secure the infrastructure in the cloud.

Enforce MFA on privileged users.
Stick to the principle of least privilege.
Enable GuardDuty and monitor for anomalies.
Enable CloudTrail to monitor and audit API operations as well as sensitive operations.
If you use S3, make sure buckets are private by default, block public access, and enable encryption wherever possible.
Assign specific roles with the least privileges needed to services or users.

The above is just a simple checklist that covers many of the essentials without talking to a consultant yet.

When to Ask for Help

It is most effective to get professional advice from cloud consultants during the design phase and before going live. This is the most efficient strategy, as having a proper design will lead to much better outcomes and save your company a lot of money in refactoring.

It is important to note that getting advice before going live with a new feature or ahead of an audit could save you from a lot of stress and trouble since you proactively remediate issues or problems before clients, auditors, or other parties find them.

Avoid Big Names

It is very tempting to contact well-known, reputable consulting companies for help as they usually are superior in expertise and are normally well equipped. However, this does come at a significant cost since big names entail overhead.

You can get similar help at much more reasonable prices from less reputable companies. You could save between 50% and 70% in expenses if you get help from small consultants for relatively similar results.

Many smaller consulting firms actually have experts who worked with big companies and know how they operate; the only difference is that you're paying for outcomes instead of brand.

Conclusion

By taking a hybrid approach, you can avoid a lot of cyber risks while still getting outcomes at a reasonable price. There is no need to sacrifice security when you can get it alongside your product efforts.

One domain broke. I refused to rerun the other five.

Wed, 29 Jul 2026 22:37:12 +0000

TerraformAgent turns a plain-English infrastructure request into validated Terraform. Six nodes: an orchestrator plans the domains, a researcher pulls AWS docs, a set of domain subagents write code in parallel, an aggregator stitches it together, a reviewer inspects it with multiple LLMs, and an evaluator runs the real terraform fmt and terraform validate before anything ships.

The subagents run in parallel, one per domain. Networking writes network.tf, compute writes compute.tf, security writes iam.tf and kms.tf, and so on. On a typical request, four or five of them fire at once.

Then the evaluator finds a problem in exactly one of them. A wildcard IAM action, an invalid reference in network.tf, a validate failure that names a single file.

The easy move is to loop the whole pipeline back to the start and regenerate everything. That works. It also throws away four passing domains to fix one broken one, and every domain is a separate LLM call. On a five-domain request, a naive full rerun costs five calls to fix a one-call problem, and it does that on every retry.

I didn't want to pay for that. So the retry only reruns the domains that actually failed.

The pipeline, briefly

The graph is linear up to the evaluator, then it branches. After the evaluator runs, one conditional edge decides whether to finish or route back into the domain subagents. When it routes back, the subagents don't rerun everything. They rerun only the domains that got flagged.

The key decision: attribute the failure to a domain

The whole thing hinges on one question. When the evaluator or reviewer reports a problem, which domain owns it?

I answer that by string-matching filenames. Each domain owns a known set of files:

DOMAIN_FILES: Dict[str, List[str]] = {
    "networking": ["network.tf"],
    "compute": ["compute.tf"],
    "storage": ["storage.tf"],
    "database": ["database.tf"],
    "security": ["iam.tf", "kms.tf"],
}

When something fails, compute_domains_to_retry scans the combined text of the review feedback, the fmt output, and the validate output for any of those filenames. Every filename it finds maps back to the domain that owns it, and only those domains get rerun.

def compute_domains_to_retry(
    review_feedback: str, eval_results: Dict[str, Any], active_domains: List[str]
) -> List[str]:
    is_blocking = "BLOCKING_ERROR" in review_feedback
    fmt_failed = eval_results.get("terraform_fmt_pass") is False
    validate_failed = eval_results.get("terraform_validate_pass") is False

    if not (is_blocking or fmt_failed or validate_failed):
        return []

    combined_text = "\n".join([
        review_feedback,
        eval_results.get("terraform_fmt_output", ""),
        eval_results.get("terraform_validate_output", ""),
    ])

    known_files = [f for files in DOMAIN_FILES.values() for f in files]
    flagged_files = [f for f in known_files if f in combined_text]
    domains_to_retry = sorted({file_to_domain(f) for f in flagged_files if file_to_domain(f)})

    if not domains_to_retry:
        domains_to_retry = sorted(active_domains)

    return domains_to_retry

If validate fails with Error in network.tf: invalid reference, that returns ["networking"] and nothing else reruns. If the reviewer writes BLOCKING_ERROR: iam.tf has a wildcard action, that returns ["security"]. If both files show up, both domains come back.

There's a deliberate fallback in the last three lines. If something failed but no filename was attributable, it reruns every active domain rather than skip the retry. I would rather pay for a full rerun than silently pass broken code because I couldn't parse a blame target.

Wiring it into the graph

LangGraph does the routing with a conditional edge. The evaluator writes domains_to_retry into state, and a router function reads it:

def should_retry_or_finish(state: dict) -> str:
    if state.get("domains_to_retry"):
        return "domain_subagents"
    return "end"

workflow.add_conditional_edges(
    "evaluator",
    should_retry_or_finish,
    {"domain_subagents": "domain_subagents", "end": END},
)

When the subagents node runs on a retry, it reads that same list and only spins up workers for the flagged domains. Everything else keeps its previous output untouched:

active_domains = state.get("active_domains", [])
domains_to_retry = state.get("domains_to_retry", [])
domains_this_pass = domains_to_retry if domains_to_retry else active_domains

existing_outputs = dict(state.get("domain_outputs", {}))

with ThreadPoolExecutor(max_workers=max(len(domains_this_pass), 1)) as pool:
    futures = {
        pool.submit(_run_domain_subagent, domain, state, feedback): domain
        for domain in domains_this_pass
    }
    for future in as_completed(futures):
        domain = futures[future]
        existing_outputs[domain] = future.result()

existing_outputs starts as a copy of the previous pass, so the passing domains carry forward and only the flagged ones get overwritten. First pass runs everything, retry pass runs the subset.

The cap, and why it's there

A retry loop with no ceiling is a way to burn money on a domain the model can't fix. The evaluator increments a counter and stops routing back after three:

retry_count = state.get("retry_count", 0)
if domains_to_retry and retry_count < 3:
    next_status = AgentStatus.EVALUATING
    retry_count += 1
else:
    next_status = AgentStatus.COMPLETED
    domains_to_retry = []

Three is a guess, not a tuned number. It's enough for the model to recover from a bad reference or a wildcard it can see in the feedback, and low enough that a domain it fundamentally can't fix doesn't spin forever.

What I'd do differently

The attribution is string matching, and that's the weak point. It only works when the failure names a file that maps to a domain. A cross-file error, a validate message that doesn't mention a filename, or a provider-level error with no file in it all fall through to the "rerun everything" branch. When that happens, selective retry quietly becomes a full rerun and I've paid for the routing logic without getting the savings.

I have real test coverage on the happy paths. test_validate_failure_with_file_in_output confirms a network.tf error returns only ["networking"], and test_blocking_with_no_attributable_file_retries_all_active confirms the fallback reruns everything. What I don't have is a way to attribute a failure that doesn't name a file. The honest next step is to make the subagents emit structured findings tagged with their domain, so attribution reads a field instead of grepping a string. That would shrink the fallback-to-everything case, which is the only case where this design earns nothing.

Takeaway

Retry the thing that failed, not everything around it, and accept that the whole saving depends on being able to name what failed.

EKS Auto Mode: What It Actually Changes (and What It Doesn’t)

Mon, 27 Jul 2026 10:00:00 +0000

What EKS Auto Mode is

EKS Auto Mode, generally available on December 1, 2024, shifts more Kubernetes infrastructure responsibility from you to AWS. You still run an EKS cluster in your AWS account, and your workloads still use the Kubernetes API, but AWS takes over much of the compute, storage, networking, node lifecycle, and core add-on management that platform teams usually wire together themselves.

For compute, Auto Mode uses Karpenter-style provisioning under the hood. When pods are unschedulable, Auto Mode provisions nodes that fit the workload’s requirements: instance family, size, architecture, capacity type, and availability zone. When capacity is no longer useful, it can consolidate and terminate nodes.

The important framing is this: Auto Mode is not “EKS without nodes.” It is EKS where AWS manages the node lifecycle more aggressively. You own the workloads, their scheduling requirements, their disruption behavior, and the operational consequences of those choices. AWS owns more of the infrastructure plumbing.

What it replaces

Before Auto Mode, running EKS in production usually meant choosing and operating several layers yourself:

Managed node groups: You chose instance types, defined scaling ranges, managed AMI updates, handled node draining, and configured Cluster Autoscaler or another scaling mechanism.

Self-managed Karpenter: More flexible than managed node groups, but you owned the Karpenter controller, IAM, NodePools, EC2NodeClasses, disruption settings, upgrades, and failure modes.

Fargate: AWS-managed compute per pod, with no node management, but no DaemonSets, a narrower workload compatibility envelope, and a different cost model.

EKS Auto Mode replaces a large part of that platform assembly with a managed model: declare workload intent and high-level compute constraints; AWS provisions and manages the EC2 instances behind it.

How it works in practice

You create or update an EKS cluster with Auto Mode enabled. The default setup can use AWS-managed built-in node pools. If you need more control, you create a NodeClass for Auto Mode infrastructure settings and a Karpenter NodePool for workload-facing scheduling constraints.

# NodeClass: EKS Auto Mode infrastructure settings for managed EC2 nodes.
apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: private-compute
spec:
  subnetSelectorTerms:
    - tags:
        kubernetes.io/role/internal-elb: "1"
  securityGroupSelectorTerms:
    - tags:
        aws:eks:cluster-name: prod-eks
  ephemeralStorage:
    size: "100Gi"

# NodePool: workload-facing constraints for nodes that Auto Mode may provision.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: private-compute
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: eks.amazonaws.com/instance-category
          operator: In
          values: ["c", "m", "r"]
  limits:
    cpu: "1000"
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

Those API groups are the Auto Mode-specific split documented by AWS: apiVersion: eks.amazonaws.com/v1, kind: NodeClass for Auto Mode node infrastructure, and apiVersion: karpenter.sh/v1, kind: NodePool for scheduling and capacity constraints.

The NodeClass is where you express AWS infrastructure placement and node-level defaults. The NodePool is where you express what kind of capacity is acceptable for workloads. Do not copy a self-managed Karpenter EC2NodeClass into Auto Mode; Auto Mode uses its own NodeClass API.

Auto Mode provisions nodes when pods are pending, consolidates when nodes are underutilized, and replaces nodes during maintenance or scale-down. AMI and node lifecycle updates are handled by AWS. AWS says Auto Mode AMIs are generally released weekly with CVE and security fixes, and Auto Mode nodes have a maximum lifetime of 21 days, which you can reduce. Your application still has to tolerate the disruption: a bad PodDisruptionBudget, strict affinity rule, or singleton stateful workload can still block or degrade a replacement.

Built-in components are managed differently than in a classic EKS build. AWS lists pod networking, service networking, cluster DNS, autoscaling, block storage, load balancer controller, Pod Identity agent, and node monitoring agent as Auto Mode capabilities. With Auto Mode compute, common add-ons such as Amazon VPC CNI, kube-proxy, CoreDNS, Amazon EBS CSI Driver, and EKS Pod Identity Agent become redundant for Auto Mode nodes, and the relevant controllers can run on AWS-owned infrastructure rather than as visible pods in your account. You can still install AWS Load Balancer Controller in an Auto Mode cluster when you need both models during migration, but AWS does not support directly migrating existing load balancers from AWS Load Balancer Controller to Auto Mode; use IngressClass or loadBalancerClass boundaries and plan blue-green migration. Treat this as a change in ownership, not as a reason to skip validation.

The workload support matrix is broader than Fargate, but not identical to self-managed nodes:

Capability	Auto Mode status
EC2 Spot	Supported through `karpenter.sh/capacity-type` requirements such as `spot`, `on-demand`, and `reserved`
Graviton / arm64	Supported through `kubernetes.io/arch: arm64` and supported Graviton instance families
GPU / accelerators	Supported for documented accelerated families; Auto Mode manages NVIDIA, Trainium, and Inferentia drivers/device plugins for supported instance types
Windows nodes	Not supported
DaemonSets	Supported as Kubernetes DaemonSets, but host-level assumptions must be validated against locked-down managed nodes

What you gain

Reduced operational surface. Node group management, AMI lifecycle, Cluster Autoscaler tuning, Karpenter controller upgrades, and a chunk of add-on wiring move out of your day-to-day scope.

Better provisioning shape by default. Dynamic provisioning is usually a better fit than fixed node group shapes. You get nodes that more closely match actual pod requirements instead of trying to pre-plan a small set of instance types.

Automatic node patching. AWS manages the node image and replacement flow. That reduces toil, but it also means your workloads need disruption policies that let AWS replace nodes safely.

Faster cluster bootstrapping. A new Auto Mode cluster can get to a usable production baseline faster than a hand-assembled EKS cluster with node groups, autoscaling, networking add-ons, storage drivers, and load balancer controllers.

Native Spot integration. Auto Mode can use Spot capacity through NodePool requirements, but you still need workload-level interruption tolerance: replicas, budgets, graceful shutdown, and queue semantics where relevant.

What you give up

Node-level access. Auto Mode nodes are intentionally locked down compared with traditional self-managed nodes. If your incident response process assumes SSH, SSM, manual package inspection, or ad hoc host changes, it needs to change.

Custom AMIs. You cannot treat the node image as your own artifact. AWS determines the operating system and AMI for Auto Mode managed instances; you cannot directly access the instance or install software on it. If your organization requires internally built, hardened, or certified AMIs, Auto Mode is likely blocked.

Unrestricted host agents. Kubernetes DaemonSets are supported, but they are the sharp edge. Some node agents work; others do not. Anything that assumes privileged host access, custom kernel modules, hostPath writes, IMDS access without hostNetwork, or low-level runtime integration needs a proof of compatibility.

Less tuning surface. You give up direct control over kubelet flags, container runtime configuration, bootstrap scripts, and arbitrary node setup. That is the point of the product, but it is also the boundary.

Different cost visibility. Managed node groups make capacity easier to reason about because you chose it up front. Auto Mode changes capacity dynamically, so cost control moves toward budgets, labels, reports, and workload-level resource hygiene.

EKS Auto Mode vs managed node groups vs Fargate

	Auto Mode	Managed Node Groups	Fargate
Node management	AWS manages node lifecycle	Shared: AWS manages the group primitive, you manage capacity shape and many updates	AWS-managed per-pod compute
AMI updates	Automatic through AWS-managed node replacement	You schedule and operate rolling updates	N/A
Instance selection	Dynamic through NodePools	You choose instance types and scaling ranges	Not exposed
Custom AMIs	No; AWS determines the AMI	Yes	No
DaemonSets	Supported, but validate host-access assumptions	Yes	No
SSH / node access	Restricted	Usually available if you enable it	No
Spot support	Yes, through capacity-type requirements	Yes, with node group or Karpenter design	No; Amazon EKS does not support Fargate Spot
Cost model	EKS control plane + EC2 + EKS Auto Mode fee	EKS control plane + EC2	EKS control plane + Fargate pod pricing
Operational burden	Low for nodes, medium for workload compatibility	Medium	Low for nodes, medium for compatibility
Right for	Default candidate for teams that do not need node customization	Regulated/custom node environments and mature platform teams	Workloads that fit Fargate’s restrictions and want per-pod isolation
Hidden constraints / gotchas	AWS controls node image and lifecycle; DaemonSets, privileged pods, hostPath, PDBs, topology rules, and node agents can block migration	You still own AMI drift, autoscaler tuning, disruption handling, and capacity fragmentation	No DaemonSets, limited host-level integrations, different networking/storage constraints, and less flexibility for mixed workload shapes

The Auto Mode pricing

Do not model Auto Mode as “EC2 plus a generic percentage” unless you have pulled the actual rate for your region and instance mix. The official structure is:

Total EKS Auto Mode cluster cost =
  EKS control plane cost
  + normal EC2 cost for instances launched and managed by Auto Mode
  + EKS Auto Mode management fee on those managed EC2 instances
  + normal surrounding AWS costs: EBS, load balancers, data transfer, CloudWatch, etc.

EKS Auto Mode management fee =
  sum of Auto Mode-managed instance runtime
  x the regional EKS Auto Mode management rate for each EC2 instance type

The EKS Auto Mode fee is applied to the EC2 instances that Auto Mode launches and manages. It is billed in addition to the normal EC2 charge and in addition to the EKS control plane charge. AWS bills the Auto Mode fee per second with a one-minute minimum, and the charge is independent of whether the underlying EC2 capacity is On-Demand, Spot, covered by Reserved Instances, or covered by Compute Savings Plans.

Do not treat the fee as a contractual flat percentage. The official pricing page describes it as a management fee that varies by EC2 instance type, and AWS pricing data is regional. The public pricing example for US West (Oregon) shows c6a.2xlarge at $0.306/hour for EC2 plus $0.03672/hour for Auto Mode, c6a.4xlarge at $0.612/hour plus $0.07344/hour, m5a.2xlarge at $0.344/hour plus $0.04128/hour, and m5a.xlarge at $0.172/hour plus $0.02064/hour. Those examples equal 12% of the listed On-Demand EC2 rate, but the safe formula for real planning is: sum(instance-hours by instance type and region x published Auto Mode management rate).

The practical cost question is not “is there a premium?” There is. The useful question is whether the premium is lower than the engineering time, incident risk, and opportunity cost of operating node lifecycle yourself.

For small teams, the answer may be yes even if the raw bill increases. For high-scale, cost-sensitive platforms, the answer needs real data: compare current EC2 waste, bin-packing efficiency, Spot usage, interruption rate, and platform maintenance time against an Auto Mode pilot.

When to use EKS Auto Mode

Use Auto Mode if:

You run EKS on AWS and do not have a hard requirement to manage nodes yourself
You do not require custom AMIs or custom node bootstrap logic
You want to reduce the operations surface for node lifecycle management
You want Karpenter-like provisioning without operating Karpenter yourself
Your workloads are mostly stateless or disruption-tolerant
Your observability, security, and storage agents are compatible with Auto Mode

Stick with managed node groups if:

Your organization requires internally certified or hardened AMIs
You need specific kernel configuration, kubelet flags, bootstrap scripts, or host packages
You depend on privileged DaemonSets or host-level security tooling that Auto Mode cannot support
You are in a regulated environment where the node image supply chain must be owned internally
Your platform team already operates Karpenter well and values the extra control

Use Fargate if:

You specifically want per-pod compute isolation
Your workload does not need DaemonSets or host-level integrations
You accept Fargate’s scheduling, networking, storage, and observability constraints
You want to avoid managing EC2 capacity entirely for a narrow class of workloads

Migration from managed node groups

Migrating an existing cluster to Auto Mode is supported, but it is not a one-command operational migration. AWS supports enabling Auto Mode on existing clusters, but you must update the cluster IAM role permissions and trust policy, enable compute, block storage, and load balancing capabilities together, and meet required add-on versions when those add-ons are installed. AWS also calls out unsupported direct migrations for EBS volumes from the standard EBS CSI provisioner to the Auto Mode EBS CSI provisioner, existing load balancers from AWS Load Balancer Controller to Auto Mode, and clusters using alternative CNIs or other unsupported networking configurations. A conservative path looks like this:

Enable Auto Mode on a non-production cluster running Kubernetes 1.29 or greater.
Inventory workloads by scheduling assumptions: node selectors, affinities, tolerations, topology spread, PDBs, privileged mode, hostPath, local storage, and DaemonSet dependencies.
Create or select the relevant Auto Mode NodeClass and NodePool resources.
Move a low-risk namespace first by changing selectors, tolerations, or labels so pods land on Auto Mode nodes.
Watch scheduling, replacement, load balancer behavior, persistent volume provisioning, logging, metrics, and security events.
T

WordPress cron behind a load balancer: fixing 'Missed schedule'

Wed, 22 Jul 2026 17:43:41 +0000

A newsroom schedules its lead story for six in the morning. At 6:05 it is not on the homepage. Nobody did anything wrong: the post was written, scheduled, saved, and its publish time has come and gone. There is no error, no failed job, nothing in the log. The post just sits in the admin list with a small grey label beside it, Missed schedule, as though it had quietly decided not to happen.

This is the worst kind of bug, the completely silent kind. And on WordPress spread across more than one server, it is barely a bug at all. It is what the default setup does under load, and the fix nearly everyone reaches for makes it worse, in a way that is genuinely hard to see.

wp-cron is not cron

The name lies. wp-cron is not a system scheduler. On a default install, WordPress checks for due jobs on a normal page load: near the end of a request it fires a non-blocking loopback call to wp-cron.php, which runs whatever is due. No visitors, nothing runs. A quiet site can sit for hours with nothing firing; a busy one runs its cron on the back of a random reader's page view.

On one server that is merely untidy. Put it behind a load balancer and it starts to fail in ways you cannot see. The loopback goes to your own domain, which resolves to the load balancer, which hands it to some instance, not necessarily the one that fired it. If an instance cannot reach the public hostname, the loopback silently does nothing. You now have a scheduler whose reliability depends on reader traffic and internal networking, for jobs like publishing the morning's lead story.

The fix everyone gives you, and the one it hides

Search this and every answer is the same two lines: switch the loopback off, and drive cron yourself.

define( 'DISABLE_WP_CRON', true );

* * * * * wget -q -O - https://www.example.com/wp-cron.php?doing_wp_cron >/dev/null 2>&1

We ran exactly that. Then, because a crontab lives on one machine and our machines come and go, we moved the trigger to a Lambda that hit the site every minute, so nothing depended on a single box staying alive. It looked correct. The request went out every minute, on schedule, forever.

And posts kept coming up Missed schedule.

Both rows want the same thing: the due events run on time. The top row triggers cron with an HTTP request, so it inherits everything sitting in front of the origin, a cache that can answer the same fixed URL without ever running PHP, a load balancer that can hand it to a draining node, and a web request that dies on a stuck lock or a mid-run recycle. The bottom row runs cron in-process with WP-CLI, so none of that is in the path.

The mistake is buried in that one-liner, and it is the same mistake whether a crontab or a Lambda fires it: we were still triggering cron with an HTTP request. Everything wrong with that survives the move to Lambda, because it has nothing to do with who makes the request and everything to do with the request itself.

Follow the path it takes. It hits your public URL, so it meets your CDN first. Our trigger was one fixed URL, wp-cron.php?doing_wp_cron, the same string every minute, which is precisely what a cache exists to answer without troubling the origin. So a good share of those tidy every-minute hits were served from cache as an empty 200, and PHP never ran. The dashboards were green. Cron was not running.

Past the cache it meets the load balancer, which sends it to whichever instance it likes, including one that is draining. If it does reach a live node, wp-cron.php runs as an ordinary web request, under the web server's timeout, holding a lock other requests respect for sixty seconds. Let one heavy job overrun that lock, or let the instance recycle mid-run, and cron wedges or dies with the queue half done. The publish_future_post event for the 6am story is somewhere in that unfinished tail.

None of it surfaces as an error. The trigger fires, something returns 200, and the post sits at Missed schedule until an editor asks why.

Stop hitting a URL

Here is the turn that fixed it, and it is smaller than it sounds: the reliable way to run cron makes no HTTP request at all.

wp cron event run --due-now

WP-CLI boots WordPress in a normal PHP process on the machine and runs the due events straight against the database. No page, so no CDN to cache it. No loopback, so no load balancer to misroute it. It runs in the CLI, which has no web timeout, so a slow job finishes instead of being cut off. Every fragile link in the HTTP path is simply not in this one.

That is the whole distance between "keep hitting the server so cron stays alive" and "run cron". Hitting a page keeps it triggered. It does not keep it finishing, on time, and finishing is the only thing a scheduled post cares about.

Which leaves one question, the one that turns this from a one-liner into a fleet problem: if cron now runs in-process on a machine, which machine?

Which machine?

Cron now runs in-process, which means it runs on whichever machine the crontab sits on. In a fleet, that innocent question is the whole problem, because there are only two obvious answers and both are wrong.

Pin it to one machine and you are back where you started. That machine is cattle: a deploy replaces it, a scale-in retires it, a health check recycles it, and your one crontab leaves with it. Cron stops, silently, and the 6am story misses again. A single runner is a single point of failure wearing a tidier hat.

So put it on every machine, bake it into the image, and let every node run cron. Now the opposite breaks. wp cron event run on each node executes the due events locally, against the shared database, in the same minute. The event that publishes the lead story fires on all of them. Readers get the newsletter three times, the post publishes and re-publishes, the feed importer runs in triplicate. The transient lock core uses for the loopback does not save you here, because each CLI process is its own world and, unless your object cache is shared, they cannot even see each other's lock.

Run everywhere, execute once

The answer is to keep cron on every node, so there is no single box to lose, and put one shared lock in front of the run, so only one node does the work each minute.

Every instance runs the trigger, so there is no single box to lose, but each run first tries to take a shared lock (a MySQL advisory lock, or a Redis key), and only the winner executes the due events. The runner records each successful pass, so a total stall is loud instead of silent.

Every node's crontab calls the same command every minute. The first thing it does is ask for a lock. Exactly one node gets it; the rest are turned away in a millisecond and exit having done nothing. The winner runs the due events and releases the lock. Next minute the race is run again, and it does not matter who wins.

For the lock you already have the one shared thing you need: the database. Every instance talks to the same MySQL, so a MySQL advisory lock is a coordinator with nothing new to deploy:

SELECT GET_LOCK('wp_fleet_cron', 0);

Ask for it with a zero-second timeout, run cron only if you got it, release it when you are done. There is a second reason to prefer this over the transient lock WordPress uses: an advisory lock lives on the database connection, so if the runner is killed mid-job, the connection closes and the lock releases itself. The stuck-lock failure that wedges the transient one for sixty seconds cannot happen here. If you run Redis across the fleet, a SET key value NX PX key does the same job; the point is that the lock is shared, not which store holds it.

Make the silence loud

One thing is still missing, and it is the thing that started all this. Elected and locked, cron runs correctly, right up until the day it does not: a bad deploy, a lock a broken database will not grant, a crontab that quietly stopped being installed. A scheduler that fails silently is the whole reason we are here.

So the runner writes down every successful pass, the time and the node that did it. That one record turns the unanswerable question, "is cron running?", into a check anything can make:

wp fleet-cron status || send-me-an-alert

It exits non-zero the moment the last run is older than it should be, so an outage pages you before an editor finds it. The missed 6am story stops being a mystery someone stumbles on and becomes an alert you already handled. It is the same rule that runs through our WordPress mail plugin: a failure you cannot see is worse than one that shouts.

The plugin

It is one file. Drop it into wp-content/mu-plugins/, add one line to wp-config.php, and run one command from system cron on every node:

define( 'DISABLE_WP_CRON', true );

* * * * * cd /var/www/html && wp fleet-cron run --quiet

Then hand the health check to your monitoring, so a stall reaches you rather than an editor:

wp fleet-cron status || send-me-an-alert

Or install it with Composer:

composer require mantekio/wp-fleet-cron

One honest word on where it is in its life. It is new, and it ships as 0.9.0. It has been exercised end to end against a real database and WP-CLI: a single runner elected out of five firing at once, a lock that frees itself when the holder is killed mid-job, the watchdog going loud the moment a run goes stale. What it has not yet had is months in a live multi-node fleet, so it is out in the open under the GPL, on GitHub and Packagist, being hardened in daylight rather than behind a curtain. The repository is github.com/mantekio/wp-fleet-cron. Same loop as the pieces before it: the article explains the thinking, the repository is the working plugin, the package is the one-line install, and each points at the other two.

Known limits

Being clear about what a tool does not do is part of the tool.

The lock assumes a single primary database. That is the normal case, and it is the one thing every node in a fleet already shares. On a multi-primary topology, or behind a connection pooler that hands one session to different callers, swap in a Redis key (SET key value NX PX); the plugin leaves a seam for it.
It assumes you can run system cron on the nodes. If your host forbids that, keep the loopback, but the watchdog still tells you the instant cron stops.
A long job serialises cron behind it. The lock is held for the length of a run, so a job that overruns the minute makes the next tick skip rather than stack a second copy on top. That is deliberate, you never want two, but a genuinely heavy job wants its own queue, not a seat in the cron tick.
Some managed hosts already run cron correctly for a fleet. Check what yours does before adding anything.

Publishing a post on time is the most solved problem in WordPress, on one server. Across a fleet, on the morning a scheduled lead has to be live at six, it turns out to be a small distributed-systems problem in a CMS costume, and the fixes are the dull, correct ones: run cron where nothing can cache or misroute it, let exactly one node act, and make the silence loud the instant it falls. None of it is clever. All of it is the distance between a scheduler you hope is running and one you know is.

That is the shape of running WordPress on AWS for a newsroom: the real problems hide inside the boring infrastructure, and the work is making the boring parts unbreakable. If a scheduled post has ever quietly failed to appear, and you would rather it never did again, let's talk.

The variables my subagents referenced and nobody declared

Wed, 22 Jul 2026 16:19:53 +0000

terraform validate failed on nearly every multi-file run with the same error class:

Error: Reference to undeclared input variable

  on iam.tf line 14, in resource "aws_iam_policy" "app_access":
  14:     Environment = var.environment

An input variable with the name "environment" has not been declared.
This variable can be declared with a variable "environment" {} block.

Same error on var.project_name. Same error on var.region. Same error on var.account_id. Four names, referenced across storage.tf and iam.tf and database.tf, declared nowhere.

I had not written any of those references. An LLM had, because I told it to.

The pipeline, briefly

TerraformAgent is a 6-node LangGraph pipeline that turns a natural-language infrastructure request into validated Terraform: orchestrator → researcher → domain_subagents → aggregator → reviewer → evaluator, with a conditional edge from evaluator back to domain_subagents for targeted retries. The orchestrator picks domains deterministically, keyword-matching the task and mapping services through a static table, with security always included. Each domain subagent generates its own files in parallel, the aggregator merges them into one directory, and the evaluator runs real terraform fmt and terraform validate against a provider cache baked into the container image.

The line that did it

Every domain subagent prompt carried this requirements block:

    Requirements:
    - Use proper HCL2 syntax
    - Add descriptive comments for each resource
    - Follow AWS best practices (tags, encryption, least privilege)
    - No hardcoded secrets — use variables or data sources
    - Reference other domains' resources by the conventional names given in the Research Context

Line three is the problem. "Follow AWS best practices (tags, encryption, least privilege)."

The subagents complied. Tagging a resource properly means an Environment tag and a Project tag, and any competent Terraform author reaches for var.environment and var.project_name to fill them. IAM policies need an account ID and a region in the ARN, so var.account_id and var.region showed up too. The model produced exactly the code a human would have written.

A human would also have written the variable blocks. The model could not, because of where it sits in the pipeline.

Why the architecture hid it

A single-file generator never hits this. It declares the variable it uses, in the file it is writing, in the same pass.

Generation here is split across parallel subagents that each own one domain's files:

DOMAIN_FILES = {
    "storage": ["storage.tf"],
    "database": ["database.tf"],
    "security": ["iam.tf"],
    ...
}

The storage subagent owns storage.tf. The security subagent owns iam.tf. A variable "environment" block belongs to neither. It belongs to the merged directory, and no subagent knows the merged directory exists.

So every subagent's output was individually valid HCL. The aggregated directory was not. Splitting generation across agents created a shared namespace that no agent owned, and the failure lived entirely in that gap.

Fix, part one: something above the subagents declares it

There was already a precedent I had not recognized as a category. versions.tf had this exact shape: a file no domain owns, generated once by the orchestrator, merged in by the aggregator.

def render_versions_tf() -> str:
    """Render the shared versions.tf content from the known provider list.

    Generated once by the Orchestrator and merged in by the Aggregator, so
    no domain subagent needs to emit its own required_providers block.
    """

Variables belong in the same bucket. So render_variables_tf(), following that pattern:

def render_variables_tf() -> str:
    """Render the shared variables.tf content every domain subagent is told
    to use for tagging (var.environment, var.project_name) plus the
    aws_caller_identity/aws_region data sources for account ID and region.

    Domain subagent prompts are instructed to follow AWS tagging best
    practices, which reliably produces references to these exact names —
    but no domain subagent declares them. Generated once by the
    Orchestrator and merged in by the Aggregator (same pattern as
    versions.tf), so `terraform validate` always finds them declared.
    """
    return """variable "environment" {
  description = "Deployment environment name (e.g. dev, staging, prod)"
  type        = string
  default     = "dev"
}

variable "project_name" {
  description = "Project name used for resource naming and tagging"
  type        = string
  default     = "terraform-agent"
}

data "aws_caller_identity" "current" {}

data "aws_region" "current" {}
"""

Both variables carry defaults. That is not cosmetic. The evaluator runs terraform validate with no tfvars file, so a variable without a default would trade one failure for another.

The orchestrator generates it alongside versions.tf:

def orchestrator_node(state: dict) -> dict:
    task = state.get("task", "")
    services = detect_aws_services(task)
    active_domains = sorted({service_to_domain(s) for s in services} | {"security"})
    return {
        "active_domains": active_domains,
        "shared_versions_file": render_versions_tf(),
        "shared_variables_file": render_variables_tf(),
        "status": AgentStatus.RESEARCHING,
        "current_node": "researcher",
    }

And the aggregator picks it up. Before:

merged = {
    "versions.tf": state.get("shared_versions_file", ""),
}
for domain_files in state.get("domain_outputs", {}).values():
    merged.update(domain_files)

After:

merged = {
    "versions.tf": state.get("shared_versions_file", ""),
    "variables.tf": state.get("shared_variables_file", ""),
}
for domain_files in state.get("domain_outputs", {}).values():
    merged.update(domain_files)

Fix, part two: two of the four symbols stopped existing

My instinct was to declare all four names. That instinct was half wrong.

Account ID and region are not deployment inputs. They are facts about the credentials and provider config the run already has. Terraform will tell you both without being asked:

data "aws_caller_identity" "current" {}

data "aws_region" "current" {}

So var.account_id and var.region did not get declarations. They stopped being referenced. Two of the four undeclared symbols went away by removing the need for the symbol rather than satisfying it.

This is the part I keep coming back to. When something references a name that does not exist, declaring the name is the obvious move and often the worse one. Half this bug was solved by shrinking the namespace instead of filling it.

Fix, part three: the prompt reads the declarations instead of describing them

The first version of this fix hand-wrote the scope into the prompt string:

    A shared variables.tf is already provided (do not redeclare it). It gives you:
    - var.environment (string, default "dev")
    - var.project_name (string, default "terraform-agent")
    - data.aws_caller_identity.current.account_id — use this, NOT var.account_id
    - data.aws_region.current.name — use this, NOT var.region
    Do not reference any other var.* name — anything else will fail terraform validate
    because it will not be declared anywhere.

That worked and it was wrong. Two places now knew the list of shared symbols: render_variables_tf(), which declares them, and this string, which describes them. Nothing kept the two in agreement. Add a variable to the renderer and forget the prompt, and subagents never use it. Remove one from the renderer and leave it in the prompt, and you are back to undeclared references and the identical validate failure.

So the prompt stopped describing the scope and started parsing it. parse_shared_scope() pulls the declared names out of the rendered variables.tf:

_VARIABLE_BLOCK_RE = re.compile(r'variable\s+"([^"]+)"\s*\{([^}]*)\}', re.DOTALL)
_DATA_BLOCK_RE = re.compile(r'data\s+"([^"]+)"\s+"([^"]+)"\s*\{')
_DEFAULT_ATTR_RE = re.compile(r'default\s*=\s*"?([^"\n]*)"?')


def parse_shared_scope(variables_tf: str) -> List[Dict[str, str]]:
    if not variables_tf:
        return []

    entries: List[Dict[str, str]] = []

    for name, body in _VARIABLE_BLOCK_RE.findall(variables_tf):
        default_match = _DEFAULT_ATTR_RE.search(body)
        default_value = default_match.group(1).strip() if default_match else ""
        entries.append({"kind": "variable", "name": name, "default": default_value})

    for data_type, data_name in _DATA_BLOCK_RE.findall(variables_tf):
        entries.append({"kind": "data", "type": data_type, "name": data_name})

    return entries

And render_domain_scope_note() builds the prompt text from that parse:

def render_domain_scope_note(variables_tf: str) -> str:
    scope = parse_shared_scope(variables_tf)
    if not scope:
        return ""

    lines = [
        "A shared variables.tf is already provided (do not redeclare it). "
        "It gives you:"
    ]
    for entry in scope:
        if entry["kind"] == "variable":
            default_note = f' (default "{entry["default"]}")' if entry["default"] else ""
            lines.append(f'    - var.{entry["name"]}{default_note}')
        else:
            attr = _DATA_SOURCE_PRIMARY_ATTRIBUTE.get(entry["type"])
            ref = f'data.{entry["type"]}.{entry["name"]}' + (f".{attr}" if attr else "")
            lines.append(f"    - {ref}")

    declared_var_names = ", ".join(
        f'var.{e["name"]}' for e in scope if e["kind"] == "variable"
    )
    lines.append(
        f"    Do not reference any var.* name other than {declared_var_names or '(none declared)'} "
        "— anything else will fail terraform validate because it will not be declared anywhere."
    )
    return "\n".join(lines)

The subagent prompt is now a single interpolation:

def _domain_prompt(domain: str, state: dict, feedback: str) -> str:
    files = ", ".join(DOMAIN_FILES[domain])
    feedback_str = f"\n[RETRY] Previous Reviewer/Evaluator feedback to fix: {feedback}" if feedback else ""
    scope_note = render_domain_scope_note(state.get("shared_variables_file", ""))
    return f"""You are an expert DevOps Coder responsible for the '{domain}' domain of this infrastructure request.
    ...
    {scope_note}

One thing does not survive the parse. Which attribute you actually want off a data source is provider schema knowledge, not something the HCL declaration states, so that stays a lookup:

_DATA_SOURCE_PRIMARY_ATTRIBUTE: Dict[str, str] = {
    "aws_caller_identity": "account_id",
    "aws_region": "name",
}

A data source with no entry here still gets listed in the scope note, just without the attribute hint. That is the one piece of the contract still maintained by hand, and it degrades quietly rather than producing invalid code.

Stating the consequence matters too. "Do not reference other variables" gets partial compliance. "It will not be declared anywhere and validate will fail" gets better compliance, because the model can reason about the outcome.

The second bug, same evaluator run

The reviewer flagged scope creep in the same pass. Subagents were emitting IAM roles, custom policies, and KMS keys nobody had asked for. A request for a DynamoDB table came back with a customer-managed KMS key and a key policy attached to it.

Same source line. "Least privilege" and "encryption" read as instructions to build IAM resources and encryption resources, not as constraints on the resources actually requested. The subagent has no notion of scope beyond what the prompt gives it, and the prompt gave it a mandate.

The requirements block now carries a boundary:

    - Build ONLY the resources this request needs. Do not add IAM roles, policies,
      KMS keys, or other supporting resources unless the request requires them or
      the specific resource you're creating cannot function without them (e.g. an
      EC2 instance needing a security group). When AWS provides a sensible default
      (e.g. DynamoDB's AWS-owned encryption key), use the default instead of
      creating a custom resource for it.

One vague instruction, two unrelated failure modes. Undeclared references and unrequested resources have nothing in common except the line that caused them.

Verification

I reproduced the error before fixing it. A DynamoDB table plus an IAM policy referencing all four names, written to a real directory, run through real terraform init -backend=false and terraform validate. Fails without variables.tf. Passes with it.

Tests at three levels. Renderer content in test_tools.py:

class TestRenderVariablesTf:
    def test_declares_environment_and_project_name(self):
        content = render_variables_tf()
        assert 'variable "environment"' in content
        assert 'variable "project_name"' in content

    def test_variables_have_defaults(self):
        content = render_variables_tf()
        # Every declared variable must have a default so subagent-generated
        # code that references it validates with no tfvars supplied.
        assert content.count('variable "') == content.count("default")

The drift regression test, which is the one that made part three worth building. Append a variable to the rendered output and assert the prompt mentions it, with no hand-written change anywhere:

    def test_scope_note_derived_from_same_parse_as_variables_tf(self):
        """If a variable is added to render_variables_tf(), the scope note
        must mention it automatically with no hand-written change needed —
        this is the actual regression test for the prompt/renderer drift bug."""
        synthetic_variables_tf = render_variables_tf() + '\nvariable "extra_flag" {\n  default = "x"\n}\n'
        note = render_domain_scope_note(synthetic_variables_tf)
        assert "var.extra_flag" in note

And the wiring, in test_nodes.py, asserting the prompt reflects state and does not carry a stale scope block when state is empty:

class TestDomainPrompt:
    def test_scope_note_reflects_state_variables_file(self):
        state = {"task": "x", "shared_variables_file": render_variables_tf()}
        prompt = _domain_prompt("storage", state, feedback="")
        assert "var.environment" in prompt
        assert "data.aws_caller_identity.current.account_id" in prompt

    def test_missing_variables_file_produces_no_stale_scope_note(self):
        state = {"task": "x"}
        prompt = _domain_prompt("storage", state, feedback="")
        assert "var.environment" not in prompt

That second one matters more than it looks. If shared_variables_file is missing from state, the scope note is empty rather than advertising symbols that were never merged in.

The evaluator caught this, not me

Worth stating plainly. Nobody read the generated code and noticed a missing declaration. The evaluator ran terraform validate and the run failed with a file, a line number, and a symbol name.

A generator without a real validation gate ships confidently wrong output. The HCL was well-formed, commented, and idiomatic. It looked correct in review. An LLM judge asked "is this good Terraform?" would have said yes, because it is good Terraform, apart from being invalid. The reason I can describe this failure to the exact line is that something mechanical ran the actual tool.

When you split code generation across parallel agents, any symbol that crosses an agent boundary has to be produced by something that sits above all of them, and described to them by reading what it produced.

My eval leaderboard published a negative retry count

Mon, 20 Jul 2026 20:47:10 +0000

I was looking at the EvalBench dashboard a few weeks ago and one cell stopped me:

Retries to valid
0.125
95% CI -0.035 – 0.285
n=40

A negative number of retries. You cannot retry a request negative-point-oh-three-five times. The mean was fine. The interval was nonsense, and it was nonsense in a way that pointed straight at how I was computing every number on the page.

EvalBench is a multi-provider LLM eval platform I built. Three suites run against OpenAI and Anthropic models across five domains: structured-output reliability, latency and cost with judge scoring, and RAG retrieval quality. Every suite emits the same MetricRecord shape, and the runner aggregates those records into leaderboard rows.

The interesting part was not the arithmetic. It was that a frontend formatting string had been quietly making a statistical decision for months.

Why I stopped publishing bare means

Here is the structured-output suite, 40 samples per model:

Model	Schema valid	First-attempt valid
anthropic/claude-sonnet-4-5	85%	80%
openai/gpt-4o	72.5%	72.5%

A 12.5 point gap. If I publish just those two numbers, every reader concludes Sonnet is meaningfully better at schema adherence.

Now the same table with intervals:

Model	Schema valid	95% CI
anthropic/claude-sonnet-4-5	85%	70.9% – 92.9%
openai/gpt-4o	72.5%	57.2% – 83.9%

The intervals overlap across roughly eleven points. At n=40 I cannot tell these models apart on this metric. The honest reading is "no detectable difference," not "Sonnet wins by 12.5."

That is the whole reason the interval math exists. A mean with no sample size and no interval is a claim disguised as a measurement.

Three metric types, three different intervals

The mistake I see in eval tooling is applying one interval formula to every column. Proportions, unbounded counts, and tail latencies have different distributions, and a single formula gets at least two of them wrong.

Proportions get Wilson, not normal approximation

Schema validity is a proportion. The textbook normal approximation, p ± 1.96 * sqrt(p(1-p)/n), breaks badly near 0 and 1. At p=1.0 it produces a zero-width interval, which claims perfect certainty from 40 samples.

Wilson does not do that:

def wilson_interval(values: Sequence[float]) -> Estimate:
    """Return the 95% Wilson interval for binary or fractional successes."""
    n = len(values)
    if n == 0:
        return _empty_estimate()

    z = 1.96
    mean = sum(values) / n
    denominator = 1 + z**2 / n
    center = (mean + z**2 / (2 * n)) / denominator
    half = (
        z
        * math.sqrt((mean * (1 - mean) + z**2 / (4 * n)) / n)
        / denominator
    )
    return Estimate(
        mean=mean,
        n=n,
        ci_low=max(0.0, center - half),
        ci_high=min(1.0, center + half),
    )

Four perfect scores gives you [0.510, 1.0], not [1.0, 1.0]. Four failures gives [0.0, 0.490]. The interval stays inside [0,1] and stays wide when n is small.

I pinned those exact values in a test rather than asserting loose properties:

@pytest.mark.parametrize(
    ("values", "mean", "ci_low", "ci_high"),
    [
        ([1.0, 1.0, 1.0, 1.0], 1.0, 0.5100999795960008, 1.0),
        ([0.0, 0.0, 0.0, 0.0], 0.0, 0.0, 0.48990002040399916),
        ([1.0, 0.0, 1.0, 0.0], 0.5, 0.15003570882017148, 0.8499642911798285),
        ([1.0, 0.5], 0.75, 0.19786250921045673, 0.9733234672343529),
    ],
)
def test_wilson_interval_matches_hand_computed_values(
    values: list[float], mean: float, ci_low: float, ci_high: float
) -> None:
    estimate = runner_module.wilson_interval(values)
    assert estimate.ci_low == pytest.approx(ci_low)

Hand-computed constants catch algebra typos that a property assertion like ci_low <= mean will happily let through.

p95 latency gets an order statistic

p95 latency is the one people get most wrong. A normal interval around a 95th percentile assumes the latency distribution is symmetric, and provider latency is not remotely symmetric. It has a long right tail from retries and queueing.

So p95 does not get a parametric interval at all. It gets a nearest-rank estimate with bounds pulled from the sorted sample:

def percentile_interval(values: Sequence[float], q: float) -> Estimate:
    """Return a nearest-rank percentile with a binomial order-statistic CI."""
    n = len(values)
    if n == 0:
        return _empty_estimate()

    ordered = sorted(values)
    estimate_index = min(n - 1, max(0, math.ceil(q * n) - 1))
    estimate = ordered[estimate_index]
    if n == 1:
        return Estimate(mean=estimate, n=n, ci_low=estimate, ci_high=estimate)

    lower_index = min(n - 1, max(0, _binomial_quantile(n, q, 0.025)))
    upper_index = min(n - 1, max(0, _binomial_quantile(n, q, 0.975)))
    return Estimate(
        mean=estimate,
        n=n,
        ci_low=ordered[lower_index],
        ci_high=ordered[upper_index],
    )

The count of observations below the 95th percentile follows a binomial distribution. Take the 2.5th and 97.5th quantiles of that binomial, use them as indices into the sorted latencies, and the bounds are actual observed latencies. No distribution assumption, and the interval cannot land somewhere the data never went.

_binomial_quantile builds the PMF by recurrence from the mode outward rather than calling math.comb, because binomial coefficients at n in the hundreds overflow into float garbage:

mode = min(n, max(0, math.floor((n + 1) * probability)))
masses = [0.0] * (n + 1)
masses[mode] = 1.0

failure_odds = (1.0 - probability) / probability
for successes in range(mode, 0, -1):
    masses[successes - 1] = (
        masses[successes] * successes / (n - successes + 1) * failure_odds
    )

Starting at the mode with mass 1.0 and walking outward keeps every ratio near unity. The masses get normalized by their own sum at the end, so the unnormalized start does not matter.

The bug

Two of those three formulas were right. The routing between them was not.

Here is what aggregate_records used to do:

proportion_metrics = {
    metadata.get("key")
    for metadata in suite.display_metrics
    if metadata.get("format") == "percent"
}

interval = (
    wilson_interval(values)
    if metric_key in proportion_metrics
    else normal_mean_interval(values)
)

Interval selection keyed off the display format string. And retries_to_valid was declared like this:

{
    "key": "retries_to_valid",
    "label": "Retries to valid",
    "format": "number",
    "higher_is_better": False,
}

Format is "number", so it missed proportion_metrics and fell through to normal_mean_interval, which has no clamps:

standard_error = statistics.stdev(values) / math.sqrt(n)
half = 1.96 * standard_error
return Estimate(mean=mean, n=n, ci_low=mean - half, ci_high=mean + half)

Mean 0.125, mostly zeros with a few ones, standard deviation large relative to the mean. mean - half goes below zero and nothing stops it.

normal_mean_interval is correct for what it is. The failure is that I made a statistical decision depend on a presentation field. "percent" is a formatting instruction for the frontend. I overloaded it into "this quantity is a bounded proportion," and the two meanings drifted apart the moment I added a metric that was a bounded count rather than a percentage.

retries_to_valid is bounded below at zero. It just is not a proportion, so neither branch fit it.

The fix

Metrics now declare their statistical support explicitly, separate from how they render:

{
    "key": "retries_to_valid",
    "label": "Retries to valid",
    "format": "number",         # presentation
    "support": "non_negative",  # statistics
    "higher_is_better": False,
}

Routing reads support:

def _interval_for_support(support: str, values: Sequence[float]) -> Estimate:
    if support == "proportion":
        return wilson_interval(values)
    if support == "non_negative":
        return non_negative_mean_interval(values)
    return normal_mean_interval(values)

non_negative_mean_interval clamps the lower bound at zero. I considered a bootstrap instead, which handles skewed low-count data better, and decided against it for now. The reasoning is in the docstring:

"""Return a 95% normal interval around the mean, clamped at zero.

Clamping (rather than bootstrapping) keeps this a small, deterministic
change to an existing formula instead of introducing a new resampling
dependency for what is fundamentally the same asymptotic-normal
approximation used elsewhere in this module. It is a known-conservative
fix: for genuinely skewed low-count data the true interval is narrower
than a clamped symmetric one, but the clamped bound is never wrong in
the way an unclamped negative bound is.
"""

A clamped interval is too wide near zero. That is a real cost and I would rather state it than pretend the clamp is free.

The other half of the fix is refusing to guess. Missing metadata raises instead of defaulting:

missing = [key for key in suite.metric_keys if key not in declared]
if missing:
    raise ValueError(
        f"suite {suite.name!r} is missing declared support for metrics: "
        f"{sorted(missing)}"
    )

A silent default is how this happened. A new suite cannot now reach the leaderboard without saying what kind of quantity each of its metrics is.

The test I should have written first

Every interval test I had pinned hand-computed values for one function. All of them passed while the dashboard published a negative retry count, because not one of them asked whether the right formula reached the right column.

That test is generic, cheap, and runs across every registered suite:

@pytest.mark.parametrize(
    "suite", registry_module.list_suites(), ids=lambda suite: suite.name
)
def test_aggregate_records_intervals_stay_within_declared_support(
    suite: Suite,
) -> None:
    ...
    for row in response.rows:
        for metric_key, estimate in row.metrics.items():
            support = support_by_metric[metric_key]
            if estimate.ci_low is None:
                continue
            if support == "proportion":
                assert 0.0 <= estimate.ci_low
                assert estimate.ci_high <= 1.0
            elif support == "non_negative":
                assert estimate.ci_low >= 0.0

It does not check any particular number. It checks that no published bound falls outside the range its quantity can physically occupy. Unit tests on formulas verify the math. This verifies the wiring, and the wiring is what broke.

There is also a regression test pinned to the exact reported case, 40 records with one outlier requiring 5 retries, mean 0.125, asserting ci_low >= 0. I confirmed it failed before the fix. A regression test you never watched fail is a test you are trusting for no reason.

The backend went from 262 tests to 267. Five tests, and one of them covers a class of failure that the other 262 could not see.

Takeaway

Publish the interval and the sample size next to every mean, and test that each interval respects the support of the thing it is measuring, because a leaderboard that cannot check its own bounds will cheerfully show you a negative retry count instead.

I Ran terraform validate Inside a Lambda. Two Things Broke.

Sat, 18 Jul 2026 20:32:28 +0000

Most agent projects that generate code evaluate it by asking another model whether the code looks correct. I did that for about a week in TerraformAgent before I got tired of it.

The failure mode is boring and predictable. The judge reads a .tf file, sees resource blocks with plausible attribute names, and says yes. The file has an undeclared variable, or a reference to a resource that does not exist, or an attribute that HashiCorp removed in provider v5. The judge has no way to know. It is pattern-matching on what Terraform usually looks like, which is exactly what the generator was doing when it produced the bug.

Terraform ships a checker. It is called terraform validate. It is deterministic, it takes about a second, and it is never wrong about syntax or references. The whole reason people reach for an LLM judge here is that running the real thing inside a Lambda is annoying. It is annoying. It is also not that hard, and this post is the parts that actually bit me.

TerraformAgent is a 6-node LangGraph pipeline that turns a natural-language infra request into Terraform. Orchestrator, researcher, parallel domain subagents, aggregator, reviewer, evaluator. The evaluator is the node I care about here.

The evaluator writes to disk and shells out

The generated code lives in state as a filename-to-content dict. To run real Terraform against it, you have to put it on a filesystem. Lambda gives you /tmp, so that is where it goes:

thread_id = state.get("thread_id", "default")
temp_dir = f"/tmp/tf_eval_{thread_id}"

if os.path.exists(temp_dir):
    shutil.rmtree(temp_dir)
os.makedirs(temp_dir, exist_ok=True)

aggregated_output = state.get("aggregated_output", {})
for filename, content in aggregated_output.items():
    filepath = os.path.join(temp_dir, filename)
    with open(filepath, "w") as f:
        f.write(content)

The thread_id in the path matters. Lambda execution environments get reused across invocations, so two runs can land in the same container. Without the thread scoping, run B validates a directory containing run A's leftovers.

Then three subprocess calls, in order:

fmt_result = subprocess.run(
    ["terraform", "fmt", "-check", "-recursive"],
    cwd=temp_dir, capture_output=True, text=True, timeout=30,
)
eval_results["terraform_fmt_pass"] = fmt_result.returncode == 0

init next, then validate only if init succeeded. Skipping that guard gives you a validate failure that is really an init failure, and the retry logic then blames the wrong thing.

if init_success:
    val_result = subprocess.run(
        ["terraform", "validate", "-no-color"],
        cwd=temp_dir, capture_output=True, text=True, timeout=60,
    )
    eval_results["terraform_validate_pass"] = val_result.returncode == 0
else:
    eval_results["terraform_validate_pass"] = None
    eval_results["terraform_validate_output"] = "skipped (init failed)"

Note the three-state result. True, False, and None. None means the check did not run. If you collapse that to False you get a retry loop chasing a validation error that was never evaluated. There is a fourth case in the fmt block for the same reason: FileNotFoundError sets the result to None with the message terraform binary not found (running locally without Docker?), because the binary only exists inside the container image and I run these tests on my laptop.

terraform init wants the network. Lambda should not.

terraform init downloads the AWS provider plugin. It is about 60 MB. Doing that on every invocation is slow, costs egress, and breaks the moment HashiCorp's release endpoint has a bad minute.

So bake it into the image at build time:

RUN mkdir -p /tmp/tf-provider-cache /opt/tf-plugins && \
    cd /tmp/tf-provider-cache && \
    printf 'terraform {\n  required_providers {\n    aws = {\n      source  = "hashicorp/aws"\n      version = "~> 5.0"\n    }\n  }\n}\n' > main.tf && \
    TF_PLUGIN_CACHE_DIR=/opt/tf-plugins terraform init -backend=false && \
    rm -rf /tmp/tf-provider-cache

ENV TF_PLUGIN_CACHE_DIR=/opt/tf-plugins

A throwaway main.tf that declares nothing but the provider requirement, one init, and the plugin lands in the cache directory.

The cache goes in /opt, not /tmp. This one cost me real time. Lambda mounts a fresh empty /tmp on every execution environment at runtime. Anything you write to /tmp during the Docker build is gone. Not stale, not partially there. Gone. /opt is part of the read-only image layers and survives to runtime.

The read-only cache breaks on version mismatch

Baking into /opt fixes the download. It introduces a second problem, and this is the part I did not see coming.

/opt is read-only at runtime. Terraform treats TF_PLUGIN_CACHE_DIR as a directory it can write to. As long as the generated code asks for a provider version matching what got baked in, Terraform reads from the cache and everyone is happy. The moment the generated code pins something else, Terraform tries to download that version into the cache dir, hits a read-only path, and fails outright. Not a slow path. A hard failure.

The fix is a copy, once per execution environment:

baked_cache = os.environ.get("TF_PLUGIN_CACHE_DIR", "/opt/tf-plugins")
writable_cache = "/tmp/tf-plugins"
if not os.path.isdir(writable_cache):
    if os.path.isdir(baked_cache):
        shutil.copytree(baked_cache, writable_cache)
    else:
        os.makedirs(writable_cache, exist_ok=True)

env = os.environ.copy()
env["TF_PLUGIN_CACHE_DIR"] = writable_cache

init_result = subprocess.run(
    ["terraform", "init", "-backend=false", "-input=false", "-no-color"],
    cwd=temp_dir, capture_output=True, text=True, timeout=120, env=env,
)

The if not os.path.isdir(writable_cache) guard is what makes this cheap. Warm containers skip the copy entirely. Cold ones pay a local filesystem copy, which is nothing next to a 60 MB download. Matching versions stay fully offline. Non-matching versions still work, they just fall back to a normal network install into a directory Terraform is allowed to write to.

That is the tradeoff I would call out in a review: I could have refused non-matching versions and kept it strictly offline, which would be faster and more predictable. I chose to degrade to a network fetch instead, because an agent that generates version = "~> 4.0" for a good reason should not hard-fail on infrastructure trivia.

The LLM judge is still there. It just answers a different question.

I want to correct something, because I have seen this framed as replacing the LLM judge, and that is not what happened.

The evaluator runs both. Mechanical checks answer "is this valid Terraform." An LLM judge answers "does this do what the user asked":

judge_prompt = f"""Task: {state.get('task', '')}

Generated Terraform code:
{json.dumps(aggregated_output)}

Does the generated code satisfy the original task? Evaluate:
1. Are all requested resources present?
2. Are the configurations correct for the stated requirements?
3. Are there any obvious omissions?

Answer strictly with JSON: {{"task_success": true/false, "reason": "...", "missing_resources": [...]}}"""

Nothing deterministic can check whether "give me a VPC with a private subnet for the database" produced the right resources. That is a semantic question and it needs a model. The point is not that LLM judges are useless. The point is that they should not be answering questions a compiler already answers.

Routing the retry

The evaluator's results feed a function that decides which domains re-run. Only the flagged ones:

is_blocking = "BLOCKING_ERROR" in review_feedback
fmt_failed = eval_results.get("terraform_fmt_pass") is False
validate_failed = eval_results.get("terraform_validate_pass") is False

if not (is_blocking or fmt_failed or validate_failed):
    return []

Two things here. is False and not falsy, so the None case does not trigger a retry. And when something failed but no specific file was named in the output, every active domain retries rather than silently returning an empty list. A retry that does nothing is worse than an expensive one.

The graph wires it as a conditional edge back to the subagents, capped at 3:

workflow.add_conditional_edges(
    "evaluator",
    should_retry_or_finish,
    {"domain_subagents": "domain_subagents", "end": END},
)

What I would do differently

The fmt output is a filename list, which makes attributing a failure to a domain easy. validate output is prose from Terraform, and I am string-matching known filenames against it. That works until a filename appears in an error message for a reason unrelated to that file being broken. Parsing terraform validate -json would give me structured diagnostics with real file and line attribution. I have not done it yet.

The retry cap of 3 is a guess. I have no data on how often a third attempt fixes something a second one did not. That is a metric I should be emitting and am not.

And the judge sees json.dumps(aggregated_output) with the entire generated codebase inline. On a large request that is a lot of tokens for one boolean.

94 tests pass on this. None of them mock terraform, because the whole point was to stop trusting a stand-in for the real thing.

Takeaway

If a real tool can answer your evaluation question, run the real tool, and save the model for the questions that have no compiler.

Repo: github.com/akashpersetti/terraform-agent

Your WordPress says the email sent. It didn't.

Sat, 18 Jul 2026 16:45:34 +0000

A user tells you they never got the password-reset email. You check: WordPress says it sent. wp_mail() returned true. There is nothing in the logs, because there is nothing to log. The mail left the building and vanished.

This is the worst kind of bug. It is silent, it is invisible, and it happens in the one flow where the user is already locked out and already frustrated. And on a default WordPress install, it is not a bug at all. It is the design.

Why WordPress email is broken by default

Out of the box, wp_mail() hands your message to PHPMailer, which hands it to PHP's mail(), which hands it to sendmail on the web server. That path has no sending reputation, no SPF alignment, and no DKIM signature. To a modern mail provider, a message from a random web server, unsigned and unaligned, is indistinguishable from spam, so Gmail and Outlook quietly bin it. Often without even a bounce.

And wp_mail() reports none of this. Its entire failure vocabulary is a single false, and on the default path it usually does not even return that: sendmail accepts the message, so wp_mail() returns true, and the mail dies somewhere downstream where WordPress will never hear about it.

So the real problem is two problems. The mail is unauthenticated, and the failure is silent. Any fix has to solve both, or it has not fixed anything.

WordPress builds the same message either way. On the default path it goes to PHP's mail() and out unsigned, where wp_mail() still returns true. The plugin swaps only the transport: a PHPMailer subclass hands the finished message to the SES API, signed, and it arrives aligned.

Why the usual fix is a half-fix

Search for this problem and every answer is the same: install an SMTP plugin, point it at a real mail service, done. WP Mail SMTP, Post SMTP, and a dozen others all do it, and they do fix the authentication half. Your mail now goes through a reputable server that signs it.

But look at where the credentials live. An SMTP plugin hooks phpmailer_init, sets $phpmailer->Host, $phpmailer->Username, and $phpmailer->Password, and it has to get those values from somewhere. That somewhere is the database, in wp_options.

Sit with what that means. Your SMTP username and password, the ones that let anything send mail as your domain, are now sitting in a table that ends up in every place a database ends up. The nightly backup. The S3 bucket that backup lands in. The staging clone a contractor pulled last month. The dump you emailed yourself to debug something. Anyone who touches any copy of that database can now send mail as you, DKIM-signed by your own domain, and there is no fingerprint that says it was not you.

For a brochure site, that is a small risk. For a newsroom whose domain reputation is an asset, it is the sort of thing that should keep you up at night. The half-fix traded a deliverability problem for a credential-management one, and then hid the trade in a settings screen.

The architecture: three moves

The fix is three decisions, and each one removes an entire class of the problem rather than mitigating it.

One: send through the SES API, not SMTP. SMTP means a username and a password, which means a secret at rest. The Amazon SES API is a signed HTTPS request. There is no long-lived password in the flow at all.

Two: authenticate with an IAM role, not a stored key. This is the move that makes the difference. If your WordPress runs on AWS compute, EC2 or ECS, the instance or task carries an IAM role, and AWS hands short-lived credentials to the machine automatically, rotated for you, never written down. The plugin reads them from the instance metadata endpoint at send time and uses them for that one request. Nothing is stored. Not in the database, not in wp-config.php, not on disk. There is no secret to leak because there is no secret at rest.

Three: replace only the transport, and reimplement nothing. This is the move that keeps the plugin honest, and it has a story behind it.

A seam, not a fork

WordPress 5.7 added a filter called pre_wp_mail. Return a non-null value from it and wp_mail() short-circuits, handing you the whole job. It is tempting to take it: hook the filter, build the SES call, return true.

It is also a trap, and I know the shape of this trap well, because I walked into a version of it once already. The last must-use plugin I wrote forked a core function, a copy of sanitize_title_with_dashes(), because core gave me no seam. That fork then drifted from core for three releases without anyone noticing, dropping an entire block of character handling, until it was mangling exactly the Arabic text it was meant to protect. The postscript on that article is a confession: a hand-copy of a core contract rots silently.

Short-circuiting pre_wp_mail would be the same mistake in a new costume. Take over wp_mail() and you now own its entire contract: header parsing, Cc and Bcc, Reply-To, content type, charset, attachments, and every wp_mail_* filter a hundred other plugins hook into. Reimplement all of that and your copy will drift from core the moment core changes, silently, exactly like last time.

So this plugin reimplements none of it. Core builds its own PHPMailer object, but only when the global is not already one:

global $phpmailer;
if ( ! ( $phpmailer instanceof PHPMailer\PHPMailer\PHPMailer ) ) {
    // ...core builds the mailer...
}

That if is the seam. Hand core a subclass of PHPMailer before it looks, and it uses yours. Core still assembles the entire message, still honours every filter, still does everything it always did. The subclass overrides one method, postSend(), the point where PHPMailer would normally talk to the transport, and ships the finished message to SES instead:

class WPSES_Mailer extends \PHPMailer\PHPMailer\PHPMailer {
    public function postSend() {
        $raw  = $this->getSentMIMEMessage();     // core built this, not us
        $sent = wpses_send_raw( $raw, /* recipients */ );

        if ( is_wp_error( $sent ) ) {
            // Throwing lands in core's own wp_mail_failed path.
            throw new \PHPMailer\PHPMailer\Exception( $sent->get_error_message() );
        }
        return true;
    }
}

We take the fully assembled MIME message that core produced and put it on the wire. We replace the transport and nothing else. There is no contract to reimplement, so there is nothing to drift. The lesson from the slug plugin, learned the expensive way, is baked into the architecture of this one: when core gives you a real seam, take the seam, never the fork.

Fail loudly, because the alternative is what we are here to fix

Go back to the password reset that vanished. The reason it vanished is that the failure was silent. So the cardinal rule of this plugin is that it is never silent.

When SES refuses a message, it tells you exactly why: an unverified sender, a sandbox restriction, a throttle. The plugin does not swallow that into a bare false. It logs the real SES error code, fires an action other code can hook, and, by throwing, lets the failure land in core's own wp_mail_failed so anything already listening for mail failures keeps working. A false is not a diagnosis. The SES error is.

And there is one thing the plugin pointedly refuses to do: fall back to PHP mail(). A fallback feels like the safe, considerate choice, and it is the exact opposite. Falling back means the mail appears to send, wp_mail() returns true, and the message lands in spam, which is the silent failure we started with, dressed up as resilience. A loud failure gets noticed and fixed. A silent success that lands in spam is never even seen.

The same principle governs what happens when the plugin is misconfigured. It does not shrug and let WordPress use sendmail. It refuses to send, and says why, on every admin screen and in the log. A must-use plugin that quietly stands aside is worse than no plugin at all, because now you believe your mail is going through SES when it is going straight to the spam folder.

Same rejected send, two outcomes. The plugin throws, so the failure lands in core's own wp_mail_failed with the real SES reason and wp_mail() returns false. The path it replaces swallows the error and returns true, and the message is gone with nothing logged.

The part every tutorial skips: bounces and complaints

Send real mail at any volume and some of it will bounce: addresses go dead, mailboxes fill up, people hit the spam button. Amazon SES watches those two numbers, your bounce rate and your complaint rate, and it is not being advisory. Cross the thresholds and it throttles your sending; stay across them and it suspends the account outright. One careless WordPress install, cheerfully emailing a list of addresses that stopped existing years ago, is enough to put a whole domain's ability to send at risk.

WordPress, left alone, will do exactly that. It has no memory of a bounce. Ask it to email a deleted address a thousand times and it will try a thousand times, because nothing in wp_mail() ever hears that the last nine hundred and ninety-nine failed.

So the plugin keeps the memory WordPress lacks. SES publishes every bounce and complaint to an SNS topic; the plugin exposes one endpoint for that topic, writes each dead or complaining address to a suppression table, and checks that table before every send. An address that hard-bounced or marked you as spam is simply never sent to again. That one feedback loop is most of the distance between a plugin that works on your blog and one you would run under a masthead. And if the plugin cannot record a bounce for a moment, a database blip mid-notification, it does not pretend it succeeded: it tells SNS so, and SNS holds the message and delivers it again, rather than letting a bounce quietly evaporate.

There is a sharp edge here that is easy to miss, and getting it wrong is worse than not having the feature at all. That SNS endpoint is a public URL. Anything on the internet can POST to it. An endpoint that believes whatever it receives is an open invitation: forge a notification and you could add any address you liked to the suppression list, silently blocking a site's own users from ever receiving its mail. So the endpoint trusts nothing by default. It pins the single SNS topic it is allowed to act on, and it verifies Amazon's cryptographic signature on every message before one row is written. A notification that is not from your topic, or whose signature does not check out, is refused. The last plugin taught the same lesson from the other side: the input you do not validate is the one that hurts you.

The traps worth naming

None of these are subtle once you know them, and every one has cost someone an afternoon. Named here so they cost you none:

Sandbox mode. A fresh SES account is sandboxed: it can only send to addresses you have verified, and the quota is tiny. Everything looks configured, and mail to real recipients is refused. The one-time fix is to request production access; the plugin's verify command tells you which mode you are in.
Region mismatch. SES identities are per-region. Verify your domain in eu-west-1, point the plugin at us-east-1, and every send fails for a reason that has nothing to do with your code. Send from the region you verified in.
An unverified From. SES will only send as an identity you own and have verified. The From domain has to be a verified identity, or the send is rejected outright, loudly, which is the whole point.
Alignment is DNS, not code. DKIM and SPF passing is what carries you to the inbox, and both are records you add, not switches the plugin can flip. Turn on Easy DKIM (the three CNAMEs SES hands you), set a custom MAIL FROM so SPF aligns, and DMARC then passes. The plugin signs the request; your DNS is what makes the result trusted.
The quotas are real. SES caps both a per-second rate and a rolling twenty-four-hour volume. A blast that ignores them does not queue politely, it fails. Know your limits before you lean on them.

Verifying it, and why the checker refuses to lie

A must-use plugin that sends mail silently is the problem. A plugin that tells you it is configured when it is not is the same problem wearing a badge. So the plugin ships a command whose entire job is to refuse to lie about its own health:

wp ses verify

It signs two real requests to SES with the exact code the send path uses, so a green result is proof the signing works, not a mock of it. It reports the things that actually break delivery: which region it is talking to, where the credentials came from (a role, ideally, not a stray key shadowing it), whether the From identity is verified, whether DKIM is on, whether the account is still sandboxed, the current quota, and whether the bounce topic is wired up. And it exits non-zero when any of that is wrong, so it composes:

wp ses verify || send-me-an-alert

Put that line in cron and a drifted config pages you before your users find it. This is the same discipline as the verify command in the slug plugin: a checker earns the right to say PASS by exercising the real thing, and it reports a failure it has not disproven rather than a reassuring green it has not earned. A check that cannot fail is not a check.

The plugin

It is one file. Drop it into wp-content/mu-plugins/, add a region and a From address to wp-config.php, give the machine an IAM role that can call SES, and WordPress mail goes through SES from the next send. No AWS SDK, no settings screen, no credentials in the database.

One honest word on where it is in its life: it is new. It is feature-complete and has been hardened through repeated adversarial review, with the SES signing checked against AWS's own test vectors, but I am shipping it as 0.9.0 while I run it end-to-end against a live SES account. So it is out in the open under the GPL, on GitHub and on Packagist, being finished in daylight rather than behind a curtain:

composer require mantekio/wp-ses-mail

The repository is github.com/mantekio/wp-ses-mail. It is the same loop as the last one: the article explains the thinking, the repository is the working plugin, the package is the one-line install, and each points at the other two.

Known limits

Being honest about what a tool does not do is part of the tool. This one has edges worth stating plainly:

It is a transport, not a mailing-list engine. It sends the mail WordPress already wants to send, one message at a time, well. A newsletter blast to tens of thousands of subscribers wants a queue and probably a dedicated bulk sender; leaning on this for that just meets SES throttling as a failed send.
No retry or backoff yet. A send SES refuses fails loudly and stops. It does not queue and try again later. For transactional mail that is usually the right behaviour (you want to know now); if you need durable retries, that is a queue in front of the plugin, not the plugin.
There is a message-size ceiling. SES caps the size of a single message. A mail with very large inline attachments can exceed it, and that send will fail rather than be silently truncated.
Alignment is yours to configure. The plugin signs every message, but DKIM, SPF and DMARC alignment are DNS decisions it cannot make for you. It can tell you they are missing; it cannot add the records.
It needs WordPress 5.7 or newer. The whole clean design rests on pre_wp_mail, which arrived in 5.7. On anything older there is no seam to take: mail falls through to sendmail, and the plugin can warn you but cannot stop it. If you are on a WordPress that old, the mail path is not the first thing to fix.

Sending an email from WordPress is easy. Running mail you can trust for a newsroom, where a lost password reset is a locked-out journalist and a suspended domain is every address on it going quiet, is a different job, and the difference lives entirely in the boring parts. No secret at rest, because there is no secret. A failure you can see, because silence is the bug. A memory of every bounce, because SES keeps score whether or not you do. None of it is clever. All of it is the distance between mail that works on a blog and mail that holds up under a masthead.

That is the whole of how we treat WordPress on AWS: take the seam core gives you, store nothing you would mind leaking, and fail where someone can see it. If your WordPress mail is a mystery you would rather it were not, let's talk.

The cost of a serverless agent isn't the Lambda. It's the loop.

Fri, 17 Jul 2026 05:46:41 +0000

When I moved Wingman onto Lambda, I assumed the thing I'd have to watch was compute duration. That's the serverless mental model everyone starts with: you pay per millisecond, so keep the function fast and cheap.

That model was wrong for an agent. The Lambda bill turned out to be the boring, predictable part. The two things that actually move cost and latency are the cold start and the retry loop, and neither of them shows up where you'd look first.

Here's what the code taught me.

The setup: what Wingman actually is

Wingman is a LangGraph worker-evaluator agent. A worker node (running gpt-4o-mini) does the task. An evaluator node (also gpt-4o-mini) grades the answer against a success criteria. If the criteria isn't met, control routes back to the worker for another attempt, up to five times.

It runs as a container image on Lambda, fronted by API Gateway and CloudFront, with session history in DynamoDB. The compute is stateless: every request rebuilds the conversation from DynamoDB, so there's no persistent checkpointer to worry about across cold and warm starts. I wrote about that state design in an earlier post. This one is about what it costs to run.

Cold start #1: the image

A Lambda container cold start begins by unpacking the image. Wingman's image carries langchain, langgraph, langchain-community, langchain-experimental, boto3, plus reportlab and sendgrid for the agent's tools. That's not a small pile of Python.

The one decision I'm glad I made is in the Dockerfile. The project has a local dependency group with playwright, gradio, and uvicorn, and a dev group on top of that. None of those belong in Lambda. So the build exports only the core deps and drops the rest:

RUN uv export \
      --no-dev \
      --no-group dev \
      --no-group local \
      --format requirements-txt \
      -o /tmp/requirements.txt \
    && pip install -r /tmp/requirements.txt --no-cache-dir \
      --target "${LAMBDA_TASK_ROOT}"

Playwright alone would have pulled a browser runtime into the image. Keeping it in the local group means it exists for running the agent on my laptop and never ships to Lambda. Smaller image, faster unpack, faster cold start. This is the kind of thing you only notice you did right when you compare it to the version where you didn't.

Cold start #2: the init you pay for on import

The bigger cold-start cost is in the handler, and it's easy to miss because it happens at import time:

# backend/main.py — runs once per new execution environment
wingman = Wingman()
wingman.setup()

_dynamodb = boto3.resource("dynamodb", region_name=os.getenv("AWS_REGION", "us-east-1"))
_table = _dynamodb.Table(_dynamodb_table_name)

setup() isn't cheap. It builds the tool list, constructs two ChatOpenAI clients, binds tools to the worker, wraps the evaluator in structured output, and compiles the LangGraph:

def setup(self):
    self.tools = get_tools()
    worker_llm = ChatOpenAI(model="gpt-4o-mini")
    self.worker_llm_with_tools = worker_llm.bind_tools(self.tools)
    evaluator_llm = ChatOpenAI(model="gpt-4o-mini")
    self.evaluator_llm_with_output = evaluator_llm.with_structured_output(EvaluatorOutput)
    self._build_graph()

Putting this at module level is the right call. It runs once when a new execution environment spins up, then every warm invocation reuses the same compiled graph and the same clients. If I'd built the graph inside the request handler instead, I'd be paying graph compilation on every single /api/chat call, warm or not. Module-level init trades a heavier cold start for near-zero per-request setup, which is the trade you want when warm invocations vastly outnumber cold ones.

The lever for cold-start speed is memory:

resource "aws_lambda_function" "backend" {
  timeout     = 300   # 5 minutes
  memory_size = 1024
}

On Lambda, CPU scales with memory. At 1024MB you get roughly a full vCPU, so bumping memory doesn't just buy RAM headroom, it makes that import-time init finish faster. Memory is a latency knob here, not only a capacity one.

The real cost: the retry loop, not the duration

Now the part that surprised me.

Look at the timeout: 300 seconds. Lambda's max is 900. That's a generous ceiling for a function whose real job takes a few seconds. So why so high? Because the retry loop can run the worker up to five times, and each turn is a full round of model calls plus whatever tools the worker decides to invoke.

Here's the accounting that matters. turn_count only increments in the evaluator:

def evaluator(self, state: State) -> Dict[str, Any]:
    ...
    return {
        ...
        "turn_count": state.get("turn_count", 0) + 1,
    }

def route_based_on_evaluation(self, state: State) -> str:
    if state["success_criteria_met"] or state["user_input_needed"]:
        return "END"
    if state.get("turn_count", 0) >= self.MAX_TURNS:
        return "END"
    return "worker"

So a single /api/chat request is not one model call. In the worst case it's five worker calls, five evaluator calls, and every tool call the worker makes in between. The Lambda duration for that is trivial to reason about and cheap. The token cost is the thing that swings, and it swings by a factor of five depending on how many times the evaluator sends the work back.

That reframed the whole cost model for me. Moving to stateless compute on Lambda made the infrastructure cost flat and predictable. What it did was push all the cost variance out of the infrastructure and into the model loop. The bill I have to watch isn't Lambda-seconds. It's tokens times retries.

The 29-second cliff

One more thing the config quietly documents. API Gateway has a hard 29-second integration timeout. A five-turn loop with tool calls can blow past that even though the Lambda itself is happily configured for 300 seconds. So the request dies at the gateway while the function is still working.

The fix is in the Terraform: a Lambda Function URL that bypasses API Gateway entirely for long tasks.

resource "aws_lambda_function_url" "backend" {
  function_name      = aws_lambda_function.backend.function_name
  authorization_type = "NONE"
}

Short chats go through CloudFront and API Gateway. Long agent runs go straight to the Function URL and get the full timeout budget. It works, but authorization_type = "NONE" and allow_origins = ["*"] are both flagged in the code as things to tighten after the first deploy, and they're still open. An unauthenticated public URL that invokes a model loop is a bill waiting to happen if someone finds it. That one's on my list.

What I'd do differently

The retry loop has no token budget. It caps at five turns, but a "turn" can be an expensive worker call with several tool invocations, and nothing stops a run from burning far more tokens than the answer is worth. A turn cap is a crude proxy for a cost cap. If I rebuilt this, the evaluator would carry a token budget alongside turn_count, and the loop would stop on whichever it hit first.

Takeaway

For a serverless agent, Lambda duration is the cheap, predictable line on the bill. The cost that actually moves lives in the cold-start init you pay on import and the retry loop you pay per request, so budget those, not the milliseconds.

I grade my AI twin on real traffic, not just an offline test set

Fri, 17 Jul 2026 05:44:28 +0000

Most eval setups have a quiet lie in them. You run a fixed question set through your agent, a judge model scores the answers, you get a number, and you feel good. But that number describes a sandbox. It says nothing about the question a stranger typed into your live app at 2am, or whether your anti-hallucination guardrail held for that specific input.

Twin is my personal AI digital twin. It answers questions about me over a RAG index of my profile, running on Bedrock with Claude Sonnet 4.5, FastAPI backend, Next.js frontend. I already had an offline eval harness. What I added this week is a second judge that grades production traffic as it happens, and building it forced me to fix an integrity bug I'd been ignoring in the offline one.

The problem with the offline harness

My offline harness in evals/ does the honest-looking thing: load queries.json, run retrieval, run inference, hand the result to a judge. The judge prompt is strict about faithfulness:

JUDGE_SYSTEM_PROMPT = """You are grading whether an AI-generated answer is
faithful to the provided source material. Be strict: any claim in the answer
not directly supported by the source material counts as a hallucination, even
if it sounds plausible.
...
- "correctly_refused" is true if the source material contains nothing relevant
  to the question and the answer clearly states the information isn't available;
  false if the source material was empty/irrelevant but the answer invented
  something anyway...
"""

The judge is good. The problem was structural: the offline harness ran its own retrieval pass, separate from the answer's retrieval pass. So the judge was grading the answer against a source set that wasn't necessarily the one the model actually saw. If retrieval is non-deterministic or the index shifts, the judge is checking the wrong evidence. You can pass an eval where the model hallucinated and fail one where it didn't. The number is measuring the wrong thing.

That gap is invisible on a static question set because retrieval usually returns the same chunks twice. It stops being invisible the moment you point it at live traffic, where you have no fixed inputs at all.

The approach: capture the exact source the model saw

The fix is boring and correct. Whatever chunks the answer was generated from, those exact chunks get judged. No second retrieval.

In the chat path, right after inference, I capture the request with the same retrieved_chunks object that produced the answer:

retrieved_chunks = retrieval.retrieve(request.message, k=5)
# ... inference runs against retrieved_chunks ...
capture_live_eval(request.message, retrieved_chunks, assistant_response)

capture_live_eval is fire-and-forget. It never raises, because an eval capture failing should never break a user's chat:

def capture_live_eval(query: str, retrieved_chunks: List, answer: str) -> None:
    """Fire-and-forget capture of a real chat exchange for async
    faithfulness judging. Never raises."""
    if not EVALS_BUCKET:
        return
    try:
        retrieved_text = "\n\n".join(
            f"## {chunk.section_title}\n{chunk.text}"
            for chunk, score in retrieved_chunks
        )
        key = f"live/raw/{datetime.now().isoformat()}-{uuid.uuid4()}.json"
        body = {
            "timestamp": datetime.now().isoformat(),
            "query": query,
            "retrieved_chunk_ids": [c.chunk_id for c, _ in retrieved_chunks],
            "retrieved_text": retrieved_text,
            "answer": answer,
        }
        evals_s3_client.put_object(
            Bucket=EVALS_BUCKET, Key=key,
            Body=json.dumps(body), ContentType="application/json",
        )
    except Exception as e:
        print(f"Live eval capture failed (non-fatal): {e}")

The important line is that retrieved_text is serialized from the same chunks the answer was built on. The judge later reads exactly this. There is no second retrieval anywhere in the live path.

The judging happens out of band

I did not want to block a chat response on a judge call. So the write to S3 under live/raw/ triggers a separate Lambda via an S3 event. That Lambda reads the raw record, judges it, and writes the result to live/judged/:

def process_record(bucket: str, key: str, s3_client) -> None:
    raw = json.loads(
        s3_client.get_object(Bucket=bucket, Key=key)["Body"].read().decode("utf-8")
    )
    judged_key = key.replace("live/raw/", "live/judged/", 1)

    output = dict(raw)
    try:
        judgment = judge.judge_answer(
            raw["query"], raw["retrieved_text"], raw["answer"]
        )
        output["judgment"] = judgment
    except Exception as e:
        output["judgment_error"] = str(e)

    s3_client.put_object(
        Bucket=bucket, Key=judged_key,
        Body=json.dumps(output), ContentType="application/json",
    )

The chat request writes one small JSON object and moves on. The user waits on nothing. The judge runs whenever the event fires. The same judge_answer function backs both the offline harness and the live judge, so I am grading production and my test set with identical criteria. That reuse is deliberate. It's the only reason the two numbers are comparable.

One detail I like: the judge runs on Bedrock with temperature: 0.0 and parses a strict JSON object with a brace-depth scan rather than a regex, because judge models love to wrap JSON in prose no matter how many times you tell them not to.

What broke, and what I'd do differently

The capture path writes retrieved_chunk_ids alongside the text. I don't use those IDs in the judge yet, but they exist for a reason: my next problem is retrieval drift. If I rebuild the profile index, the chunk IDs behind a past judgment change, and a stored judgment silently starts referring to source text that no longer exists. Right now the judged record freezes retrieved_text at capture time, which protects the judgment itself, but I have no way to diff "what the judge saw" against "what the index says today." Storing the IDs is the first half of that; I haven't built the second half.

The bigger honest gap: I only judge faithfulness. The judge can tell me the model stayed inside its source material. It cannot tell me the answer was useful, or that retrieval pulled the right chunks in the first place. A perfectly faithful answer to badly retrieved context still passes. Faithfulness is the cheap half of quality, and it's the half I automated because it's the half a single model call can check.

I'd also reconsider the fire-and-forget silence. capture_live_eval swallowing every exception means a broken eval bucket produces one print and nothing else. On Lambda that print lands in CloudWatch and I will not notice for a week. If I were doing this again I'd emit a metric on capture failure, because a silent eval pipeline is the same as no eval pipeline.

Takeaway: an eval number is only trustworthy if the judge grades the exact evidence the model actually used, and the fastest way to find out whether yours does is to point it at real traffic instead of a frozen question set.

The day YouTube deleted our channel

Fri, 10 Jul 2026 22:53:46 +0000

It took a single notice. The YouTube channel that held years of a newsroom's video was gone in one action, and every video embed across the site was suddenly pointing at something that no longer existed. On a publication that runs video on most of its articles, that is the kind of morning that empties an archive.

It was a non-event. We changed one setting, the whole site began playing the same videos from our own storage, and readers never saw a gap. This is the story of why a deleted channel was a configuration change rather than a catastrophe, and the one decision, made years earlier, that made the difference.

Platforms can delete you, and you do not get a vote

Channels and accounts get removed for all sorts of reasons, and most of them are not a fair fight: an automated copyright match against music in the background, a strike from a report, a policy you did not know had changed, a regional block, an API quietly deprecated, or an entire product shut down (the internet is littered with services that were permanent right up until they were not).

The common thread is that the decision is theirs, the timing is theirs, and the content you put on their platform was never fully in your hands. For a newsroom, video on a third-party platform is best treated as reach, not as custody.

Publish to platforms, never store on them

The decision that turned the incident into a shrug was made on the very first upload: every video goes to our own S3 bucket first. The original lives with us. Pushing a copy out to YouTube, Dailymotion or Vimeo is a distribution step, for reach and free bandwidth, not the place the file actually lives.

It is the same discipline behind the rest of our media handling: the original is offloaded to storage we control on upload, and everything downstream is a copy or a rendition of it. So when a platform pulls a video, or an entire channel, nothing is actually lost. The master is still sitting in the bucket exactly where it was put.

One video, four sources

On top of that, each video is a single logical asset with up to four play sources linked behind it: YouTube, Dailymotion, Vimeo, and Amazon (the file served from our own S3 through CloudFront).

The important word is source. Where a video plays is a stored setting, not an embed baked into the post. That one distinction is the whole game. If the player reads a source field, you can change where every video comes from without editing a single article. If instead every post contains a literal YouTube iframe, changing the source later is a database migration under pressure, which is exactly when you do not want one.

The original lives in S3 and is published out to YouTube, Dailymotion and Vimeo. Where a video actually plays is a stored setting, a site-wide default with a per-video override, so any source can be the active one.

Two dials: site-wide and per video

The source has two levels of control.

Site-wide, you set the default that every video plays from. We ran on YouTube, because that is where the audience and the bandwidth were.

Per video, you can override that default, and in a newsroom that is not a rare event. Long videos that carry clips or background music from other sources get flagged by YouTube's automated Content ID constantly, often even when you hold the rights, and clearing a strike means filing an appeal and waiting days. So those videos are set, right there on the upload screen, to play from Dailymotion, Vimeo or S3 instead, while the rest of the library stays on YouTube. The content most likely to be struck simply sidesteps the platform that polices it hardest, with no downtime and no appeal queue.

The site-wide default sets the source for every video at once. A per-video override pins a single clip to another source: the long documentary full of licensed music is set to Vimeo on the upload screen, so it sidesteps the platform most likely to strike it, while everything else stays on the default.

The day the whole channel went

Then the version of the problem you cannot patch one video at a time: the entire channel was deleted at once. Every video that defaulted to YouTube would fail in the player, all at the same moment, across the whole archive.

Because the play source was a single site-wide setting, and because every original was already sitting in S3, the fix was to change the default from YouTube to Amazon and save. Every embed on the site re-pointed to the S3 copies, served through CloudFront, in one action. The outage was as long as it takes to change a dropdown.

Compare that to the alternative if YouTube had been the storage rather than a distribution target: the archive would simply be gone, and the "recovery" would be re-sourcing and re-uploading thousands of videos that no longer existed.

When the channel was deleted, every video that defaulted to YouTube failed at once. Because the play source is a single site-wide setting and the originals were already in S3, changing the default to Amazon restored the entire archive in one click.

The fallback is not a downgrade

It is worth being clear that playing from your own infrastructure is not a degraded emergency mode. The S3 source plays through CloudFront like any other CDN-backed asset. When you want adaptive streaming at scale, you put it in front: an upload event triggers AWS Elemental MediaConvert to produce HLS renditions, CloudFront caches the segments, and a standard player streams quality that adapts to each viewer's connection.

In other words, falling back to your own storage is not making do. For many sites it is the best source they have. The platforms were the convenience, not the foundation.

What it is really about: custody

Strip away the specifics and the principle is simple. Own the original, and treat every external platform as interchangeable reach you opt into, never as the home of your media. In practice that is three habits:

Keep the master somewhere you control (for us, S3).
Make playback a setting, not a hard-coded embed, so the source is a dial.
Publish to platforms for reach, but never let one become your only copy.

Do that, and platform risk (a strike, a deleted channel, a shut-down service, a pricing change you cannot stomach) stops being an existential event and becomes a switch you flip. The same thinking applies anywhere you syndicate: images to a CDN, articles to third-party readers, audio to podcast directories. Keep the master, keep the destination swappable.

A two-minute gut check for your own setup

Three questions worth answering honestly:

If your video platform closed your account tonight, do the originals exist anywhere else? If the only copy is on the platform, that is the first thing to fix, before anything clever.
Is playback a per-post embed, or a source you can change centrally? If it is baked into content, switching later is a migration, not a setting.
Do you publish to platforms, or store on them? Publishing is healthy. Storing is the trap.

If the answers make you uncomfortable, you already know where to start.

The bottom line

We keep newsrooms fast and unbreakable on WordPress and AWS, and this is the resilience side of the same coin as keeping the media pipeline lean: the originals never leave your control, so neither a platform nor a traffic spike can take the archive down.

If your video lives on someone else's platform and losing it would hurt, let's talk.

How a Single NAT Gateway Can Silently Kill Your AWS High Availability

Sat, 04 Jul 2026 08:44:42 +0000

A real-world lesson from a production-like AWS lab challenge

The Scenario That Should Scare You

Imagine this: your AWS environment has two Availability Zones, public and private subnets, an Application Load Balancer, Auto Scaling. Your architecture diagram looks solid. Then one Availability Zone goes down, your ALB fails over instantly, your EC2 instances in AZ-B are running fine. But your application is still broken.

Because every private subnet instance, including those in AZ-B, is routing outbound traffic through one NAT Gateway sitting in AZ-A. Which is now unreachable.

You didn't have a highly available architecture. You had the illusion of one.

Understanding the Problem: NAT Gateways Are Zonal

A NAT Gateway is not a regional resource. It lives in a specific Availability Zone.

When you create a NAT Gateway, you place it in a specific subnet, which belongs to a specific AZ. If that AZ goes down, your NAT Gateway goes down with it.

Many teams create a single NAT Gateway to save costs, then route all private subnet traffic across all AZs through that one gateway:

Private Subnet AZ-A → 0.0.0.0/0 → nat-09xxxxx (AZ-A) ✅
Private Subnet AZ-B → 0.0.0.0/0 → nat-09xxxxx (AZ-A) ❌

The private subnet in AZ-B is routing through a NAT Gateway in AZ-A. This is a cross-AZ dependency, and a silent Single Point of Failure.

What I Found in the Lab

The lab presented a VPC with this structure:

Resource	CIDR / Details
VPC	10.0.0.0/16
Public Subnet AZ-A	10.0.128.0/20
Public Subnet AZ-B	10.0.144.0/20
Private Subnet 1A (AZ-A)	10.0.0.0/19
Private Subnet 1B (AZ-A)	10.0.192.0/21
Private Subnet 2A (AZ-B)	10.0.32.0/19
Private Subnet 2B (AZ-B)	10.0.200.0/21

Two NAT Gateways existed: one in AZ-A, one in AZ-B. At first glance, this looked correct.

But when I inspected the Route Tables, the problem was immediately visible. All four private subnet Route Tables had the same entry:

Destination: 0.0.0.0/0 → Target: nat-09xxxxxxxx (AZ-A)

The NAT Gateway in AZ-B existed, but nobody was using it. It was provisioned but completely disconnected from the routing logic. The two private subnets in AZ-B were silently depending on the NAT Gateway in AZ-A for all outbound internet traffic.

Why This Happens

There are two common causes:

1. Cost-cutting gone wrong
Teams create one NAT Gateway to reduce costs, then forget that high availability requires one per AZ. A NAT Gateway costs approximately $0.045/hour plus data transfer charges. Running two instead of one adds roughly $32/month, a small price compared to the cost of an outage.

2. Infrastructure drift
The architecture was correct at some point, then someone modified the Route Tables manually, or via a flawed IaC change, and the second NAT Gateway became orphaned without anyone noticing. No alerts, no errors, no warnings. Everything looks fine until AZ-A goes down.

This is what makes this particular SPOF so dangerous: it is completely invisible during normal operations.

The Fix: One NAT Gateway Per AZ, One Route Table Per Private Subnet

The solution is straightforward: each private subnet must route its outbound internet traffic through the NAT Gateway in its own Availability Zone.

Correct routing after the fix:

Private Subnet 1A (AZ-A) → 0.0.0.0/0 → nat-AZ-A ✅
Private Subnet 1B (AZ-A) → 0.0.0.0/0 → nat-AZ-A ✅
Private Subnet 2A (AZ-B) → 0.0.0.0/0 → nat-AZ-B ✅
Private Subnet 2B (AZ-B) → 0.0.0.0/0 → nat-AZ-B ✅

Step 1 — Identify which NAT Gateway belongs to which AZ

Go to VPC → NAT Gateways, click each NAT Gateway and check the Subnet field, this tells you which AZ it belongs to.

Step 2 — Fix the Route Tables for AZ-B private subnets

Go to VPC → Route Tables
Find the Route Table associated with Private Subnet 2A (AZ-B)
Click Edit routes
Change 0.0.0.0/0 from nat-AZ-A → nat-AZ-B
Save changes
Repeat for Private Subnet 2B (AZ-B)

Step 3 — Verify

All four private subnet Route Tables should now point exclusively to the NAT Gateway in their own AZ. If AZ-A goes down, AZ-B is completely self-sufficient.

Getting It Right From the Start: Terraform

If you're provisioning your VPC with Infrastructure as Code, which you should be, here's how to enforce this pattern correctly with Terraform from day one.

ˋhcl

NAT Gateway in AZ-A

resource "aws_eip" "nat_a" {
domain = "vpc"
}

resource "aws_nat_gateway" "nat_a" {
allocation_id = aws_eip.nat_a.id
subnet_id = aws_subnet.public_a.id

tags = {

Name = "nat-gateway-az-a"

}
}

NAT Gateway in AZ-B

resource "aws_eip" "nat_b" {
domain = "vpc"
}

resource "aws_nat_gateway" "nat_b" {
allocation_id = aws_eip.nat_b.id
subnet_id = aws_subnet.public_b.id

tags = {

Name = "nat-gateway-az-b"

}
}

Route Table — AZ-A private subnets

resource "aws_route_table" "private_a" {
vpc_id = aws_vpc.main.id

route {

cidr_block     = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.nat_a.id

}

tags = { Name = "private-rt-az-a" }
}

Route Table — AZ-B private subnets

resource "aws_route_table" "private_b" {
vpc_id = aws_vpc.main.id

route {

cidr_block     = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.nat_b.id

}

tags = { Name = "private-rt-az-b" }
}

Associations — AZ-A

resource "aws_route_table_association" "private_1a" {
subnet_id = aws_subnet.private_1a.id
route_table_id = aws_route_table.private_a.id
}

resource "aws_route_table_association" "private_1b" {
subnet_id = aws_subnet.private_1b.id
route_table_id = aws_route_table.private_a.id
}

Associations — AZ-B

resource "aws_route_table_association" "private_2a" {
subnet_id = aws_subnet.private_2a.id
route_table_id = aws_route_table.private_b.id
}

resource "aws_route_table_association" "private_2b" {
subnet_id = aws_subnet.private_2b.id
route_table_id = aws_route_table.private_b.id
}
ˋ

The beauty of this approach: the correct pattern is enforced by design. Each AZ has its own NAT Gateway, its own Route Table, and explicit associations. Infrastructure drift becomes impossible, any change goes through code review.

The Broader Lesson: Designing for Failure

AWS high availability is built on one fundamental principle:

Assume everything will fail. Design so that the failure of any single component does not bring down the entire system.

A NAT Gateway is a component. An Availability Zone is a failure domain. When you route cross-AZ traffic through a single NAT Gateway, you create an invisible dependency that violates this principle, and the worst part is that everything looks fine until the moment it isn't.

The AWS Well-Architected Framework's Reliability Pillar specifically calls for eliminating Single Points of Failure. A shared NAT Gateway is a textbook SPOF, easy to miss precisely because the architecture looks correct at first glance.

Key Takeaways

A NAT Gateway is zonal, it belongs to one specific Availability Zone
Routing all private subnet traffic through a single NAT Gateway creates a hidden Single Point of Failure
The fix: one NAT Gateway per AZ, one Route Table per AZ
Use Terraform to enforce this pattern by design and prevent infrastructure drift
The cost of two NAT Gateways (~$32/month extra) is nothing compared to the cost of an outage

This article is part of my AWS Solutions Architect Associate (SAA-C03) preparation series. I document real hands-on lab experiences, architecture challenges, and lessons learned along the way.

Follow along for more practical AWS architecture and Infrastructure as Code content.

Stop Putting Everything in One Terraform State: Use Terragrunt Dependency Blocks

Sat, 04 Jul 2026 08:15:24 +0000

Prerequisites

Before getting started, make sure you have the following:

Basic knowledge of Terraform (HCL syntax, resources, variables, remote state)
Terraform >= 1.11 installed - Download
Terragrunt installed - Installation guide
An AWS CLI configured with sufficient permissions to create S3 buckets and EC2 instances
Visual Studio Code with the HashiCorp Terraform extension for syntax hightlighting and autocompletion
Read Part 1 of this series: [Stop Copy-Pasting Terraform State Configs: Use Terragrunt Instead](Stop Copy-Pasting Terraform State Configs: Use Terragrunt Instead)

Introduction

In Part 1 of this series, we saw how Terragrunt eliminates the repetition of remote state backend configurations across environments. if you haven't read it yet, I recommend starting there - [Stop Copy-Pasting Terraform State Configs: Use Terragrunt Instead](Stop Copy-Pasting Terraform State Configs: Use Terragrunt Instead).
Today, we go one step further.
Most of Terraform projects start the same way: everything in one state file. Your VPC, your security groups, your EC2 instances, your RDS database, all managed together. It feels simple and convenient at first. But as your infrastructure grows, this approach becomes a hidden risk.
Imagine this: a developer runs terraform apply to redeploy an EC2 instance that is rebuilt multiple times a day. Because everything is in the same state file, that single command now has access to your VPC configuration, your production database, and your security groups, resources that should never be touched during a routine EC2 redeployment.
One wrong move, one bad variable, one interrupted apply, and you could accidentally destroy or corrupt critical infrastructure that takes hours to rebuild.
In this article, we'll explore how Terragrunt's dependency blocks allow you to split your Terraform state between infrastructure components, so that frequently changed resources never put your critical infrastructure at risk.

The Problem: One State File to Rule Them All

When everything lives in a single Terraform state file, your infrastructure looks like this:

single state file
├── VPC                ← modified once a month
├── Subnets            ← modified once a month
├── Security Groups    ← modified occasionally
├── RDS Database       ← critical, rarely modified
└── EC2 Instances      ← modified 10x per day

Every terraform apply, no matter how small, touches this single state file. This creates three serious problems:

Problem 1: Blast radius: If something goes wrong during a routine EC2 redeployment, the entire state file is at risk. A corrupted state means Terraform loses track of all your resources, VPC, database, everything.
Problem 2: No separation of concerns: A junior developer redeploying an EC2 instance has the same Terraform access as a senior engineer modifying the VPC. There is no natural boundary between critical and non-critical infrastructure.
Problem 3: Slow operations As your infrastructure grows: Terraform has to refresh the state of every single resource on every terraform plan or terraform apply, even if you're only changing one EC2 instance. This makes operations increasingly slow.
The solution is to split your state between infrastructure components, and Terragrunt dependency blocks make this both simple and elegant.

The Solution: Separate States with Dependency Blocks

Instead of one monolithic state file, Terragrunt allows you to give each infrastructure component its own isolated state:

vpc/                    ← state 1 — modified rarely
security-groups/        ← state 2 — modified occasionally
rds/                    ← state 3 — critical, rarely modified
ec2/                    ← state 4 — modified daily

Each component lives in its own folder with its own terragrunt.hcl file and its own state file in S3:

s3://my-terraform-state/
├── dev/vpc/terraform.tfstate
├── dev/security-groups/terraform.tfstate
├── dev/rds/terraform.tfstate
└── dev/ec2/terraform.tfstate

Now when a developer runs terraform apply on the EC2 component, only the EC2 state is touched. The VPC, the database, and the security groups are completely isolated and protected.
But here's the challenge: if components are separated, how does the EC2 module know the subnet ID from the VPC module? How does the security group know the VPC ID?
This is where Terragrunt's dependency block comes in.
The dependency block allows a component to read the outputs of another component without sharing the same state file:

# ec2/terragrunt.hcl

include "root" {
  path = find_in_parent_folders()
}

# Declare dependency on VPC component
dependency "vpc" {
  config_path = "../vpc"

  mock_outputs = {
    subnet_id = "subnet-00000000"
  }
}

# Declare dependency on security groups component
dependency "security_groups" {
  config_path = "../security-groups"

  mock_outputs = {
    sg_id = "sg-00000000"
  }
}

# Use outputs from dependencies as inputs
inputs = {
  subnet_id      = dependency.vpc.outputs.subnet_id
  security_group = dependency.security_groups.outputs.sg_id
  instance_type  = "t2.micro"
  environment    = "dev"
}

Two things to notice here:
First, config_path points to the folder of the dependency, not a specific file. Terragrunt knows where to find the outputs.
Second, mock_outpouts provides fake values for when you run terragrunt plan without the dependencies being deployed yet. this allows you to validate your configuration before deploying anything.

The Complete Project Structure

Here is the complete project structure for a dev environment with separated state files:

project/
├── terragrunt.hcl                    # Root — remote state defined once
└── dev/
    ├── vpc/
    │   ├── terragrunt.hcl
    │   └── main.tf                   # VPC + Subnets
    ├── security-groups/
    │   ├── terragrunt.hcl
    │   └── main.tf                   # Security Groups
    ├── rds/
    │   ├── terragrunt.hcl
    │   └── main.tf                   # RDS Database
    └── ec2/
        ├── terragrunt.hcl
        └── main.tf                   # EC2 Instances

Each component exposes its key values through Terraform outputs, which are then consumed by dependent components via the dependency block.
Here is how the dependency chain flows:

vpc/
  └── outputs: vpc_id, subnet_id
        │
        ├─────────────────────────┐
        ▼                         ▼
security-groups/                 ec2/
  inputs: vpc_id            inputs: subnet_id
  outputs: sg_id                  │
        │                         │
        └─────────────────────────┘
                    ▼
                  ec2/
             inputs: sg_id

Terragrunt reads this dependency graph automatically and deploys components in the correct order: VPC first, then security groups, then EC2. You never have to think about the deployment order manually.
The VPC component is the simplest, it has no dependencies:

# dev/vpc/terragrunt.hcl

include "root" {
  path = find_in_parent_folders()
}

inputs = {
  environment = "dev"
  vpc_cidr    = "10.0.0.0/16"
}

And its main.tf exposes the values that other components need:

# dev/vpc/main.tf

output "vpc_id" {
  value = aws_vpc.main.id
}

output "subnet_id" {
  value = aws_subnet.public.id
}

The outputs are what the dependency block reads when EC2 asks for dependency.vpc.outputs.subnet_id.

The full code for this project structure is available in the GitHub repository

Deploying with Dependency Blocks

Once your structure is in place, deploying is as simple as one command from the dev/ folder:

terragrunt run-all apply

Terragrunt automatically:

Reads the dependency graph across all components
Deploys in the correct order — VPC → Security Groups → EC2
Passes outputs from one component as inputs to the next
Creates a separate state file in S3 for each component

You can also target individual components:

# Redeploy only EC2 — VPC and Security Groups are untouched
cd dev/ec2/
terragrunt apply

# Check outputs of a specific component
cd dev/vpc/
terragrunt output

# Destroy in reverse order automatically
cd dev/
terragrunt run-all destroy

Notice the power here: when you run terragrunt apply in dev/ec2/ only, Terraform touches only the EC2 state file. Your VPC and database state files are completely safe, even if something goes wrong.

The mock_outputs - Why They Matter

When you run terragrunt plan on EC2 component before the VPC is deployed, Terragrunt needs values for subnet.id and sg.id to validate the configuration. Since the real values don't exist yet, mock_outputs provides temporary placeholders:

dependency "vpc" {
  config_path = "../vpc"

  mock_outputs = {
    subnet_id = "subnet-00000000"
  }

  mock_outputs_allowed_terraform_commands = ["validate", "plan"]
}

The mock_outputs_allowed_terraform_commands parameter ensures that mock values are only used during validate and plan, never during apply. This prevents accidental deployments with fake values.

A Note on Security
Before wrapping up, a quick but important note on security, raised by Paul Marcelin in the comments of Part 1.
When your state files are separated by component, you have a natural opportunity to apply different IAM permissions per component. For example:

Developers can have read/write access to the EC2 state file
Only senior engineers or CI/CD pipelines can access the VPC and RDS state files
Production state files can be encrypted with dedicated KMS keys per environment.

This is a significant security improvement over a single state file where everyone has access to everything. Separating state files is the first step, securing them with IAM policies and KMS encryption is the natural next step.
For a deep dive on Terraform state file security, I recommend this LinkedIn post by Yaroslav Naumenko: "Your Terraform state file is a secret. Most teams don't treat it that way."

Conclusion

Let's recap what we covered in this article:

The problem: a monolithic state file creates a dangerous blast radius where routine operations can accidentally affect critical infrastructure
The solution: Terragrunt dependency blocks allow each component to have its own isolated state file
The dependency block reads outputs from other components without sharing their state
mock_outputs allow you to validate configurations before dependencies are deployed
terragrunt run-all apply automatically respects the dependency order
Separating state files is also the foundation for better security, different IAM permissions per component.

Together, Part 1 and Part 2 give you a complete Terragrunt workflow:

Part 1 → One root terragrunt.hcl    = No repeated backend configs
Part 2 → Dependency blocks          = No more monolithic state files

If you found this helpful, share it and follow me for the next article in the series.
The code for this article is available on my GitHub.

The Reality of Roadmaps 🗺️

Tue, 30 Jun 2026 09:00:00 +0000

Do roadmaps actually help people learn?

I am not sure.

Because the first time I saw one, it did the exact opposite.

I remember opening a DevOps roadmap at roadmap.sh and seeing Linux, Networking, AWS, Docker, Terraform, Kubernetes, CI/CD, Security...

The list just kept going.

Instead of feeling motivated, I felt overwhelmed.

I closed all the tabs, shut down my laptop, and went to sleep.

I am serious.

It looked less like a learning path and more like a university degree.

What made it worse was that I went back the next day and tried again.

Same feeling.

Surely there had to be a simpler version somewhere.

So I did what most people do.

I went looking for a better roadmap.

Instead, I found more roadmaps with different layouts, orders, and opinions.

One said to learn Docker before Kubernetes.

Another said the opposite.

One started with Python.

Another skipped it entirely.

I kept searching for the version that would finally make sense.

What I found instead was more confusion.

At some point, I stopped and asked myself an honest question.

Was I actually learning anything?

The answer was no.

I was researching how to learn without doing any learning.

Collecting roadmaps as if they were going to do the learning for me.

That realisation was uncomfortable.

Because it meant the problem was not the roadmap.

It was what I expected the roadmap to do.

There are AWS concepts that took me weeks to understand.

There are things I thought I understood until I tried using them.

There are topics I ignored and had to come back to later.

None of that ever shows up on a roadmap.

The roadmap only shows the clean version of the story.

The finished version.

The version that makes sense after everything has already happened.

Roadmaps show what a field looks like when you zoom out.

They do not show what learning feels like when you are inside it.

I still visit roadmap.sh.

I still think roadmaps are useful.

But I no longer expect them to remove the messy part of learning.

This roadmap feels real to me.

If you enjoy reflections like this, you can subscribe to the newsletter.

You can follow my work and projects on Twitter, LinkedIn, GitHub, and YouTube, where I share what I am building and learning along the way.

Thank you for reading.

Reqrea’s 1 Million Passport Exposure Is a Reminder Why Cloud Security Audits Matter

Mon, 29 Jun 2026 19:49:17 +0000

Incident

On May 15, 2026, Reqrea, a Japan-based KYC company, reported that more than 1 million passports had been exposed to the public internet via a misconfigured S3 bucket.

This breach impacted over 1 million travelers from around the world. Exposed passports could potentially put all impacted individuals at risk of identity theft or sophisticated social engineering attacks in the future.

This recent incident further confirms my point in the past that AWS misconfigurations remain one of the top cybersecurity vulnerabilities in 2026.

How it was identified

The vulnerability was discovered by security researcher Anurag Sen, who detected that one of the S3 buckets used by the company was accessible to the public without authentication or an authorization process.

Why it matters

KYC platforms and companies handling PII are expected to have strong change control, security governance and regular security audits to prevent such costly mistakes from happening. In this instance, since Reqrea has collected such data, they may face regulatory, contractual, and reputational exposure.

Unfortunately, such cases could potentially harm companies' reputation among customers and partners, which could hurt future sales.

>[!NOTE]
>The average data breach costs U.S. organizations about USD 10.22 million, according to IBM’s 2025 Cost of a Data Breach Report.

Lesson Learned

Companies handling PII need to adopt "Secure by Design" policy and have strong cloud governance if they want to use cloud solutions.

A good preventive strategy would entail companies bringing in third parties (e.g. consultants) before going live or before getting involved in any activity involving protected data collection or transmission of such data.

Another lesson learned is that delegating KYC to third parties may actually reduce security if the KYC platform itself is insecure.

I can easily point out the fix needed for this vulnerability. However, we have to look at this incident or vulnerability rather as a question of governance and policies first and technical as second.

IRSA(IAM Roles for Service Account) Explained

Fri, 26 Jun 2026 20:43:15 +0000

One of the most common mistakes I see in Kubernetes environments is this:

A pod needs access to AWS services, so teams simply attach permissions to the EC2 worker node and call it a day.

It works.

Until one compromised pod suddenly has access to everything the node can access.

That’s when I started appreciating the beauty of IRSA (IAM Roles for Service Accounts) in Amazon EKS.

What is IRSA?

IRSA enables Kubernetes pods running in Amazon EKS to securely access AWS services using dedicated IAM roles.

Instead of storing AWS Access Keys inside Kubernetes Secrets or relying on broad node-level permissions , each workload can assume its own IAM role using a Kubernetes Service Account.

How does it work?

Create an IAM Role with required permissions
Configure the EKS OIDC Provider
Associate the IAM Role with a Kubernetes Service Account
Deploy pods using that Service Account
AWS STS issues temporary credentials to the pod
Pod accesses AWS services securely

No static credentials.
No credential rotation headaches.
No shared node permissions.

Why IRSA Matters

Eliminates AWS Access Keys from Kubernetes
Enforces Least-Privilege Access
Temporary Credentials issued by AWS STS
Better Auditability with CloudTrail
Reduced Blast Radius if a Pod is Compromised
Each Workload Gets Its Own Identity

IRSA vs Node IAM Roles

Node IAM Roles

• All pods inherit node permissions
• Difficult to isolate workloads
• Larger attack surface
• Harder to determine which workload accessed AWS resources
IRSA

• Per-workload IAM permissions
• Better security boundaries
• Improved compliance and auditing
• Fine-grained access control

Real-World Example

Imagine an EKS cluster running:

Payment Service → Access to DynamoDB
Reporting Service → Access to S3
Notification Service → Access to SQS

With Node IAM Roles, all three services could potentially access all resources.
With IRSA, each service receives only the permissions it actually needs.

That’s security done right.!!

"The Bigger Lesson"

Cloud security is moving away from: "Long-lived credentials" To "Identity-based access"

Whether it’s:

• IRSA in EKS
• OIDC in GitHub Actions
• Workload Identity in GCP
• Managed Identities in Azure

The future is the same:

"Stop distributing secrets. Start distributing identities."

How are you managing AWS access for Kubernetes workloads today?

Coder Legion - Recent posts tagged aws

From Writing the Spec to Approving It: What AI-DLC Looks Like on a Real Build

🤖 From Writing the Spec to Approving It: What AI-DLC Looks Like on a Real Build

The app

The flip, in one table

Where everything was written

Stage by stage: what it did and why it helps

What the human actually did

Why this holds up better than a prompt

What this run does not show

Takeaway

I automated the hardest parts of Amazon SES, so you can send from your own AWS account in two minute

What it does

How it is built

Pricing

Organization-Wide EC2 DNS Automation with EventBridge & AWS Config Aggregator: One Rule Per Account

🌐 Organization-Wide EC2 DNS Automation with EventBridge and AWS Config Aggregator: One Rule Per Account

The problem (why the single-account version does not scale)

The approaches I threw out

How it fits together

Filling in the blanks

The code

The spoke's one and only rule (Terraform)

The hub's custom bus, with an org-scoped resource policy

Rules, queue, and Lambda wiring (hub)

The enrichment Lambda (Python)

Why there is an SQS queue in the middle

What actually came out of it

The part worth stealing

Wrapping up

Kubernetes Security Is Three Problems, Not One: Admission, Identity, Runtime

🛡️ Kubernetes Security Is Three Problems, Not One: Admission, Identity, Runtime

Layer 1: admission, decide what gets in

Layer 2: identity, kill the static credential

Layer 3: runtime, watch what actually happens

The layer most teams skip: supply chain

Putting it together

Further reading

How AWS Is Designed: Regions, Availability Zones, and Where a Workload Runs

WHAT

WHY

WHEN

HOW

Where the Services Fit

Takeaway

AWS PyTorch: Build, Train, and Deploy Deep Learning Models in the Cloud

tags: aws, pytorch, machinelearning, python

1. What Is PyTorch on AWS?

2. Understand the Architecture

3. Hands-On Project: Build a Digit Classifier

Prerequisites

Step 1: Create a project environment

Step 2: Create the training script

Step 3: Train the model

Step 4: Make a prediction

4. Move the Project to AWS

Step 1: Create an S3 bucket

Step 2: Run training with SageMaker AI

Step 3: Store model artifacts in S3

5. Deploy the Model for Inference

6. Production Best Practices

Cost optimization

Security

Reproducibility

Monitoring

Model evaluation

7. Where Can You Use PyTorch on AWS?

Final Thoughts

Official Resources

AWS #PyTorch #MachineLearning #DeepLearning #AmazonSageMaker

A Field Guide to Beating Amazon Bedrock Throttling

🛡️ A Field Guide to Beating Amazon Bedrock Throttling

Step one: know the quota you are hitting

Soft limits vs hard limits

Layer one: retry with backoff and jitter

The same retry in a Step Functions workflow

Layer two: cap concurrency

Layer three: a circuit breaker for throttle storms

Layer four: handle timeouts, they are not throttling

Model timeout: shrink the work

Custom Domain, Subdomain Setup (`dev.ideategudy.tech`) & SSL Configuration

Why Use a Subdomain (`dev.ideategudy.tech`)?

Connecting Namecheap Subdomain (`dev.ideategudy.tech`) to AWS:

⚡ Automated CI/CD Pipeline (`deploy.yml`)