Coder Legion - Recent posts tagged devops

From Writing the Spec to Approving It: What AI-DLC Looks Like on a Real Build

Fri, 30 Oct 2026 10:36:49 +0000

🤖 From Writing the Spec to Approving It: What AI-DLC Looks Like on a Real Build

TL;DR

AI-DLC is public and open source: awslabs/aidlc-workflows. The method comes from this AWS DevOps blog post.

My only written input was one sentence. Everything else I did was pick options and approve gates.

The run produced requirements, four architecture decisions, a unit plan, a traceability file, and 12 tests that passed when I ran them, in about 44 minutes from start to the code approval.

The human role moves from author to decision-maker. The decisions are still yours, and the record shows where they changed the outcome.

Most AI-assisted development still follows the old split. A human writes the spec, a human writes the code, and the AI helps with autocomplete or generates tests at the end. AI-DLC (the AI-Driven Development Life Cycle) reverses that. The AI drafts the requirements, the design, and the code. The human answers short questions and approves each stage before the next one starts.

To see what that looks like in practice, I built a small command-line tool with it and then read the record the workflow left behind. This post walks through that record, folder by folder.

The app

The tool is a Spooky Date Fact Generator. It picks one of 10 spooky dates (or takes --date MM-DD) and prints a spooky event for that day from Wikipedia's "On this day" feed. It is small on purpose.

The whole request was this sentence:

/aidlc build random date generator with fact around that number

The flip, in one table

	Traditional with AI help	AI-DLC
Requirements	Human writes them	AI drafts them from multiple-choice answers
Design	Human sketches it	AI proposes, human picks between options
Code	Human writes, AI assists	AI writes against an approved plan
Tests	Human or AI at the end	AI writes them to the stated requirements
Human job	Author	Decide, validate, approve
Evidence	Pull request diff	Audit log, decisions, traceability file

Where everything was written

Two locations matter. The generated app sits in its own folder, and the workflow record sits in an aidlc/ folder.

app/                                  <- the generated tool
  src/spooky-facts.ts      core: dates, fetch, keyword filter, nearest-date search
  src/cli.ts               command line: options, output, exit codes
  test/                    12 tests, offline, with a fake fetch
  package.json, biome.json, tsconfig.json, README.md

aidlc/
  codekb/                             <- 9 docs about the existing repo
  intents/<date>-date-fact-generator/ <- the record for this build
    aidlc-state.md                    stage checklist and settings
    audit/                            every event, decision, and approval
    inception/
      reverse-engineering/            scan of the existing repo
      requirements-analysis/          questions + requirements.md
      domain-design/                  questions + components.md + decisions.md
      units-generation/               unit-of-work files
    construction/u1-spooky-date-cli/code-generation/
      code-generation-plan.md, code-summary.md,
      unit-test-instructions.md, traceability.json

Each stage writes a -questions.md file, and your answers sit in the same file as the AI's questions. That makes the decision trail readable without any tooling.

Stage by stage: what it did and why it helps

Initialization. The workflow scaffolded the record folder, scanned the workspace, and classified it as brownfield TypeScript on Bun. This is automatic. It helps because every later stage starts from facts about the repo instead of guesses.

Reverse Engineering. A developer agent scanned the existing repo and wrote nine knowledge-base documents (architecture, technology stack, code structure and others). This matters most in a brownfield repo. The later code followed the repo's Bun, TypeScript, and Biome conventions because the AI had read them first.

Requirements Analysis. The AI asked six questions, such as where facts come from, what to do when a date has no spooky event, and whether this should be a web page or a command line. Each had lettered options. I answered, and one answer included a free-text note: generate dates that have spooky facts, from 10 such scenarios. The result is requirements.md with 15 functional and 5 non-functional requirements, each with acceptance criteria and a tag pointing back to the answer that produced it, such as [Q5]. This stage turns one vague sentence into testable statements before any code exists.

Domain Design. Four more questions produced components.md and four architecture decision records. The AI recommended splitting the tool into four parts. I chose two. The decision record states that the four-part split was rejected as more structure than needed. This is the clearest example of the validator role, because the human overrode the AI's recommendation and the reason is recorded.

Units Generation. One question: build as one unit or two. I chose one, approved the plan, and got unit-of-work.md with boundaries, responsibilities, and constraints. It fixes the blast radius. The unit is limited to apps/date-fact-generator/, and nothing in the framework folders may change.

Code Generation. The AI wrote a plan, then the code, then the tests. It asked one thing here: Bun was not installed, so should it install Bun or switch to Node. I chose Bun. The code-summary.md lists files created, decisions made, and deviations from the plan. One deviation worth noting: the AI added word forms such as "dies" and "sank" to the keyword list, because whole-word matching would otherwise miss "Houdini dies". The summary also states that some planned checks were grouped into fewer tests to match the 12-test count.

Build and Test. In the record I read, this stage was still marked in progress, so its summary was not written yet. I ran the checks myself from the app folder: 12 tests passed with 93 assertions, and lint and typecheck were clean.

What the human actually did

Reading the audit log, the human work was this:

Typed one sentence to start.
Approved a recomposed plan. The workflow's planner found the default express plan was missing inputs for later stages, and proposed adding Domain Design and Units Generation while dropping deployment and monitoring.
Answered 11 multiple-choice questions across three stages, plus two more decisions: build inside the existing repo, and install Bun.
Approved 5 stage gates: reverse engineering, requirements, domain design, units, and code generation.

That is the flip. Nothing in that list is writing a spec or writing code. All of it is deciding and checking.

Why this holds up better than a prompt

Two details make the validation real instead of a rubber stamp.

First, traceability. traceability.json maps every requirement ID (FR1.1 through NFR5, 20 in total) to the file that satisfies it. If a requirement has no target, it shows up.

Second, the record is complete. The audit log holds the timestamp of each question, answer, and approval. A reviewer six months later can see why the design has two components and not four.

What this run does not show

This was an express-scope build of a 658-line app, with plan approval, sensors, and learnings switched off by the scope. It shows the shape of the method, not how it behaves on a large system with many units and teams. The Build and Test stage had not completed in the record, and the tests are offline, so only one manual live smoke run, recorded in the code summary, exercised the real Wikipedia feed.

The questions are only as good as the options offered. A weak option list can steer a hurried approver, so reading the alternatives still matters.

Takeaway

The useful change is not that the AI writes more code. It is that the human's time moves to the points where judgment counts: what the tool should do, which design to accept, and whether the result meets the requirements. If you want to try it, start with a small brownfield repo and read the -questions.md files first. They show the method faster than any description.

Flyway Migration: From Legacy Database to CI/CD

Thu, 29 Oct 2026 12:05:30 +0000

Most teams think of Flyway only when they need to manage ongoing database schema changes. But what if your project already has a complete, stable schema and you’re only now introducing version control? Or what if you want to incorporate your schema into a CI/CD pipeline for reliable, reproducible deployments?

In this guide, I’ll walk you through a real-world scenario where we used Flyway to baseline an existing MySQL database and then extended it to support version-controlled schema migrations in CI/CD, without disrupting production.

What is Flyway?

Flyway is a simple but powerful tool for managing database migrations. It applies versioned SQL scripts to your database and tracks them in a special table called flyway_schema_history.

Typical script names:

V1__initial_schema.sql
V2__add_orders_table.sql
V3__rename_column.sql

Flyway ensures these scripts run only once and in the correct order. It also integrates well with CI/CD tools, making it ideal for automated environments.

Use Case: Existing Schema with No Migrations (Yet)

In our project, we had:

A production MySQL database with dozens of tables
No migration history
No current need for structural changes
A desire to version control the database and enable future migrations via CI/CD

Step 1: Baselining an Existing Schema

Dump the schema

mysqldump -u root -p --no-data your_database > V1__baseline_schema.sql

Clean the output: remove auto-increments, user references, and environment-specific SQL.

Create directory structure

mkdir -p db/sql
mv V1__baseline_schema.sql db/sql/

Configure flyway.conf

flyway.url=jdbc:mysql://your-db-host:3306/your_database
flyway.user=db_user
flyway.password=secret_password
flyway.locations=filesystem:db/sql
flyway.baselineOnMigrate=true

Run baseline

flyway baseline

This creates the flyway_schema_history table and marks version 1 as the current state without running any script.

Step 2: Add a Real Migration

After baselining, we added a real change in a new script

-- db/sql/V2__add_status_to_orders.sql
ALTER TABLE orders ADD COLUMN status VARCHAR(20) DEFAULT 'pending';

Then we ran:

flyway migrate

Result: Flyway applied version 2, recorded it in history, and our database was now safely versioned.

⚙️ Step 3: Integrate Flyway into CI/CD

You can integrate Flyway with GitHub Actions, GitLab CI, Bitbucket Pipelines, or Jenkins.

✅ Example GitHub Actions Workflow

name: Flyway Migration
on:
  push:
    branches: [ main ]

jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Set up Java (Flyway needs it)
        uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '17'
      - name: Install Flyway
        run: |
          curl -L https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/10.8.1/flyway-commandline-10.8.1-linux-x64.tar.gz | tar xvz
          sudo ln -s $PWD/flyway-10.8.1/flyway /usr/local/bin/flyway
      - name: Run Flyway Migrations
        env:
          FLYWAY_URL: jdbc:mysql://your-host:3306/your_database
          FLYWAY_USER: ${{ secrets.DB_USER }}
          FLYWAY_PASSWORD: ${{ secrets.DB_PASSWORD }}
        run: |
          flyway -url=$FLYWAY_URL \
                 -user=$FLYWAY_USER \
                 -password=$FLYWAY_PASSWORD \
                 -locations=filesystem:./db/sql \
                 migrate

Benefits of This Approach

✅ You gain auditability for all database changes

✅ Developers can build local environments using the same scripts

✅ CI/CD ensures consistency across all environments

✅ Flyway’s validate command catches missing or changed scripts

✅ Schema management becomes part of your software delivery lifecycle

Potential Challenges

Problem → Fix

Flyway re-applies the baseline script
→ Use baselineOnMigrate=true and never modify a baseline
Passwords in config files
→ Use secrets and environment variables in CI/CD
Manual schema drift
→ Disable direct DB access in prod or run flyway validate periodically
Need rollback support
→ Write manual undo scripts or use point-in-time backups

Alternatives (and Why Flyway Wins)

Liquibase
✅ Pros: Advanced features, rollback support
❌ Cons: More complex, steeper learning curve

Django ORM / Alembic
✅ Pros: Built-in for app devs
❌ Cons: Not cross-language or DB-agnostic

Manual SQL or DB dumps
✅ Pros: Simple
❌ Cons: No versioning, no tracking, no automation

Flyway is a great middle-ground: powerful enough for teams, simple enough for solo devs, and perfect for CI/CD.

Final Thoughts

Flyway isn’t just for teams writing migrations every day. It’s also ideal for bringing an existing database under version control, giving your project a foundation for safe schema evolution.

We started with a baseline, added our first real migration, and then built it into our CI/CD pipeline — all in a day’s work.

If you’re managing an existing DB without schema tracking, now’s the time to fix that. Start with a baseline. Your future self will thank you.

Let’s Connect!

Have you tried Flyway for baseline-only use cases or integrated it into your CI/CD pipeline? I’d love to hear your experience in the comments.

If you found this helpful, follow me here on DEV for more DevOps and database engineering content.

The log was deleted. Its 48M was still allocated.

Wed, 28 Oct 2026 13:00:00 +0000

A 64M scratch tmpfs, a worker process holding its log open on file descriptor 3, and the usual reflex when space runs low: rm the log. du on the directory dropped from 48M to 0. df stayed at 48M used. The next 32M write failed with No space left on device, and df read 100% while du found only the 16M that had been written.

rm removes a name. The kernel frees the blocks only when no names are left and no process has the file open, and the worker still had it open. lsof +L1 lists exactly those files:

$ lsof -w +L1 /srv
COMMAND    PID USER FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
sleep   608657 root 3w   REG   0,78 50331648     0    3 /srv/app/worker.log (deleted)
exit=0

NLINK 0, (deleted), 48M held for writing by PID 608657. Truncating through the process's own descriptor, : > /proc/608657/fd/3, took df back to 25% while the process kept running.

Next time, empty a live log with : > file instead of deleting it. With an append-mode writer (>>) that is clean. A writer that opened with > keeps its old offset: in the same test, its log came back as a 17M sparse file with 1.0M on disk.

Finding which process holds a port or a busy mount, plus a script that groups deleted-but-open files and totals them, is on the lsof command examples page.

A Stopwatch Needs an Isolated Lane

Wed, 28 Oct 2026 10:49:36 +0000

Timing tests look wonderfully objective. Start a stopwatch, call the component, stop the watch, and compare the result with a budget.

The number is precise. The experiment may not be.

That distinction matters in continuous integration, where many tests share CPU time, thread pools, network stacks, containers, and database capacity. A wall-clock assertion can fail because the product became slower. It can also fail because the test runner became busier. If we cannot distinguish those two explanations, the test is producing urgency without producing useful evidence.

The contract was reasonable

Consider a service with separate liveness and readiness checks. Liveness says the process is still running. Readiness says the instance can safely accept work.

One integration test deliberately points the service at an unreachable dependency. The expected behavior is not success. The service should return an unavailable response, and it should do so within a fixed upper bound. That is a meaningful reliability contract: an orchestrator needs a quick, honest answer rather than a request that hangs behind retry policies intended for normal user operations.

The test boots the real host, makes the request, verifies the unavailable result, and checks elapsed time.

So far, so good.

The failure blamed the wrong interval

The timing assertion began failing repeatedly in CI. Raising the limit would have made the build green. Adding an automatic retry might also have hidden the symptom.

Neither action would have explained the result.

The endpoint implementation had not changed during the relevant window. More importantly, the surrounding timing evidence showed that host startup in the same test took roughly as long as the delay attributed to the readiness request. The test also ran beside many parallel integration classes doing heavier work.

The stopwatch was attached to one request, but that request still depended on threads scheduled in a shared process. Under contention, continuation work can wait. Startup can wait. Connection failure can be observed later than expected. The test reports wall-clock time, not pure execution time.

This does not prove the service is fast. It proves the original experiment was unable to tell product latency from runner contention.

Keep the budget before changing it

When a timing test turns red, I prefer to keep the bound intact while investigating. The bound represents a real operational expectation. Relaxing it before understanding the measurement destroys evidence that may still be valuable.

A useful first pass is to separate three intervals:

Setup — host boot, fixture creation, migrations, container startup, and test data.
Contention — time spent waiting for CPU, threads, sockets, database slots, or another shared resource.
Behavior — the operation whose latency the assertion is meant to protect.

Instrumenting those boundaries does not require a production profiler. Even a few carefully placed timestamps can reveal that the supposedly slow operation accounts for only a small part of the elapsed test.

Isolate the claim, not the whole suite

The narrow repair was to move only the timing proof into a non-parallel test collection.

The response status did not change. The time budget did not change. The endpoint code did not change. Functional readiness tests stayed in the normal parallel suite. Only the test making a wall-clock claim received an isolated measurement lane.

That scope matters. Serializing an entire integration project can turn a useful diagnosis into an expensive habit. Parallel tests are valuable because they shorten feedback cycles and expose accidental shared state. The aim is not to remove parallelism. It is to stop using an uncontrolled parallel environment as a precision instrument.

I like to keep separate tests for separate questions:

Does the check return the correct status when the dependency is unavailable?
Does the response avoid leaking infrastructure details?
Does the host remain alive while readiness is false?
Under controlled contention, does the readiness answer stay within its operational budget?

Only the last question necessarily needs timing isolation.

The trade-off is real

An isolated collection adds a serial tail to the build. It also creates another test boundary that future maintainers must understand. If many tests demand exclusive execution, the suite can become slow and the isolation can conceal real capacity problems.

That is why this technique should be applied to the measurement contract, not used as a blanket cure for flakiness.

In return, failures become more actionable. A red timing check is less likely to mean “the runner happened to be crowded” and more likely to mean “the protected latency contract changed.” The team spends less time rerunning the same experiment and less time normalizing noise by widening budgets.

A practical review checklist

Before changing a failed CI time limit, ask:

Did the measured implementation change?
Is setup included in the timed region?
Which resources are shared with parallel tests?
Does the test need fixtures it is currently competing for?
Can functional correctness remain parallel while the timing proof runs alone?
Would the same isolation exist in the production condition we care about?

The last question prevents overfitting. An isolated test should remove irrelevant test-runner noise, not erase meaningful load. If the production contract is specifically about behavior under concurrency, create a controlled load test with explicit pressure instead of relying on incidental CI contention.

Precision is a property of the experiment

A stopwatch does not make a test scientific by itself. The environment, boundaries, and competing work determine what the number means.

Keep meaningful budgets. Investigate before retrying. Isolate the smallest timing claim that needs a controlled lane, and leave the rest of the suite parallel.

The result is not merely a greener pipeline. It is a test whose failure tells you something you can act on.

Organization-Wide EC2 DNS Automation with EventBridge & AWS Config Aggregator: One Rule Per Account

Tue, 27 Oct 2026 19:19:28 +0000

🌐 Organization-Wide EC2 DNS Automation with EventBridge and AWS Config Aggregator: One Rule Per Account

TL;DR

Detecting EC2 launches and terminations across a whole AWS Organization for DNS is not the five-minute single-account job it looks like.

The EC2 state-change event carries only an instance ID and a state, no IP, so it is useless for DNS on its own.

A hub-and-spoke design keeps the per-account footprint to a single EventBridge rule; all logic lives in one hub account.

The hub enriches each event by querying the AWS Config Aggregator you already run, which holds every instance's IP, tags, and state org-wide. No new data pipeline.

An aws:PrincipalOrgID resource policy on the hub's custom bus gives cross-account fan-in without per-account config.

SQS between the rule and the Lambda absorbs bulk launches and gives you retries plus a dead-letter queue.

How to detect every EC2 launch and termination across an entire AWS Organization, and turn it into automated DNS registration, without deploying repeated infrastructure in every account.

I spent longer than I would like to admit staring at this problem before the answer got simple. So if you are wiring up DNS across an AWS Organization right now, maybe this saves you a weekend.

Here is the short version of what I landed on. I needed near real-time DNS registration for every EC2 instance across a multi-account, multi-region org, feeding an on-prem enterprise DDI system (think Infoblox). The two obvious ways to do it both annoyed me: one puts infrastructure in every account, the other leans on a fragile cross-account API-scraping job.

What I used instead is a hub-and-spoke event setup:

Each spoke account gets exactly one EventBridge rule. That is the whole footprint.
One hub account does everything else: a custom event bus, an SQS queue, one Lambda.
The Lambda fills in the missing details by querying the AWS Config Aggregator, which, and this is the part I almost missed, already has every instance's IP, tags, and state for the entire org. No new data pipeline required.

Onboarding a new account came down to a one-line deploy. Below is the whole thing: the architecture, why I made the calls I made, and the code.

The problem (why the single-account version does not scale)

In one account this is a five-minute job. EC2 fires a state-change event, you point an EventBridge rule at a Lambda, done. I have written that Lambda a dozen times.

Across an organization it falls apart for a few reasons:

Instances come up in dozens of accounts and several regions.
The people who run DNS are not the people who own the workload accounts, so "just add it to the account" is not really on the table.
The EC2 state-change event does not include the IP address. You get an instance ID and a state. That is it. Useless for DNS on its own.
Nobody wants to deploy, version, and babysit a Lambda plus queue plus IAM role in every account just to catch these events.

The usual reason you end up here is a move to an enterprise DDI platform. Every instance that boots needs a DNS record, every instance that dies needs its record cleaned up, org-wide, in near real-time, with as little junk in each account as you can manage.

So two things really drove the design:

Adding an account should be almost free.
The IP is not in the event, so I have to look it up, and that lookup cannot be flaky.

The approaches I threw out

Option A: put the whole stack in every account. Rule, queue, Lambda, IAM role, shipped everywhere with StackSets. It works. But now you own N copies of the same code, and every bug fix or IAM tweak is an N-account rollout. I have maintained fleets like that. No thanks.

Option B: a central poller that assumes a role into the source account and calls DescribeInstances for the IP. Logic is centralized, which is nice, but you need an assume-role trust in every account and you are paying a cross-account API hop on every single event. More IAM surface, more things to break.

Option C: hub-and-spoke, enriched from the Config Aggregator. This is the one. All the logic sits in the hub. Spokes just forward. And instead of building anything to fetch the IP, the hub asks the Config Aggregator, which is already sitting there aggregating EC2 config (IP, tags, state) from every account and region.

Option C won for a boring but decisive reason: the hard part, the enrichment data, already existed. I did not have to build it. I just had to notice it was there.

How it fits together

Spokes forward EC2 state changes to the hub's custom bus; the hub enriches via the org Config Aggregator, then calls the DDI system. Spokes are dumb, the hub is smart, and the data already exists. That is the whole idea in one line.

That second hub rule is easy to forget, by the way. If you only listen on the custom bus, the hub account's own instances never get DNS records, because they never leave home to reach the custom bus. Ask me how I know.

Filling in the blanks

Most single-account tutorials skip this bit, which is exactly the bit that matters at scale. Here is what the event actually gives you:

{
  "source": "aws.ec2",
  "detail-type": "EC2 Instance State-change Notification",
  "detail": {
    "instance-id": "i-1234567890",
    "state": "running"
  }
}

An ID and a state. No IP, no tags, no hostname. So the Lambda goes and gets the rest, from the aggregator, not from the source account.

The nice thing: the aggregator is org-level, so one query resolves an instance no matter which account or region it lives in. No assume-role, no per-account permissions to manage.

The code

Everything here is vendor-neutral and anonymized. Swap the Infoblox call for whatever you actually talk to: Route 53, a CMDB, an IPAM API.

The spoke's one and only rule (Terraform)

This is the entire per-account footprint. Ship it with StackSets or a Terraform module.

# --- Deployed in EVERY spoke account/region ---

resource "aws_cloudwatch_event_rule" "forward_ec2_state" {
  name           = "forward-ec2-state-to-hub"
  description    = "Forward EC2 running/terminated events to the hub account"
  event_bus_name = "default"

  event_pattern = jsonencode({
    source        = ["aws.ec2"]
    "detail-type" = ["EC2 Instance State-change Notification"]
    detail = {
      state = ["running", "terminated"]
    }
  })
}

resource "aws_cloudwatch_event_target" "to_hub_bus" {
  rule           = aws_cloudwatch_event_rule.forward_ec2_state.name
  event_bus_name = "default"
  arn            = "arn:aws:events:${var.region}:${var.hub_account_id}:event-bus/ec2-dns-hub-bus"
  role_arn       = aws_iam_role.eventbridge_cross_account.arn
}

# Minimal role that lets EventBridge PutEvents to the hub bus
resource "aws_iam_role" "eventbridge_cross_account" {
  name = "eventbridge-forward-to-hub"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "events.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "put_events" {
  role = aws_iam_role.eventbridge_cross_account.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = "events:PutEvents"
      Resource = "arn:aws:events:${var.region}:${var.hub_account_id}:event-bus/ec2-dns-hub-bus"
    }]
  })
}

The hub's custom bus, with an org-scoped resource policy

This policy is the thing that makes "one rule per spoke" actually work: any account in the org can publish, and the hub needs zero per-account config.

# --- Deployed ONCE in the hub account ---

resource "aws_cloudwatch_event_bus" "hub" {
  name = "ec2-dns-hub-bus"
}

resource "aws_cloudwatch_event_bus_policy" "allow_org" {
  event_bus_name = aws_cloudwatch_event_bus.hub.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "AllowOrgAccountsToPutEvents"
      Effect    = "Allow"
      Principal = "*"
      Action    = "events:PutEvents"
      Resource  = aws_cloudwatch_event_bus.hub.arn
      Condition = {
        StringEquals = {
          "aws:PrincipalOrgID" = var.aws_org_id
        }
      }
    }]
  })
}

Do not skip the aws:PrincipalOrgID condition. It scopes that Principal = "*" down to your organization instead of, you know, the entire internet. It is the one line standing between "elegant pattern" and "open event bus."

Rules, queue, and Lambda wiring (hub)

resource "aws_sqs_queue" "events" {
  name                       = "ec2-dns-events"
  visibility_timeout_seconds = 120
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 5
  })
}

resource "aws_sqs_queue" "dlq" {
  name = "ec2-dns-events-dlq"
}

# Rule on the custom bus (events from spokes)
resource "aws_cloudwatch_event_rule" "hub_custom" {
  name           = "ec2-state-custom-bus"
  event_bus_name = aws_cloudwatch_event_bus.hub.name
  event_pattern = jsonencode({
    source        = ["aws.ec2"]
    "detail-type" = ["EC2 Instance State-change Notification"]
    detail        = { state = ["running", "terminated"] }
  })
}

resource "aws_cloudwatch_event_target" "custom_to_sqs" {
  rule           = aws_cloudwatch_event_rule.hub_custom.name
  event_bus_name = aws_cloudwatch_event_bus.hub.name
  arn            = aws_sqs_queue.events.arn
}

# Second rule on the default bus catches the hub account's OWN instances
resource "aws_cloudwatch_event_rule" "hub_default" {
  name           = "ec2-state-default-bus"
  event_bus_name = "default"
  event_pattern = jsonencode({
    source        = ["aws.ec2"]
    "detail-type" = ["EC2 Instance State-change Notification"]
    detail        = { state = ["running", "terminated"] }
  })
}

resource "aws_cloudwatch_event_target" "default_to_sqs" {
  rule = aws_cloudwatch_event_rule.hub_default.name
  arn  = aws_sqs_queue.events.arn
}

resource "aws_lambda_event_source_mapping" "sqs_to_lambda" {
  event_source_arn = aws_sqs_queue.events.arn
  function_name    = aws_lambda_function.dns_registrar.arn
  batch_size       = 10
}

The enrichment Lambda (Python)

This is the heart of it: resolve the instance against the Config Aggregator, then act on the state.

import json
import boto3

config = boto3.client("config")

AGGREGATOR_NAME = "OrganizationConfigAggregator"

def get_instance_details(instance_id: str) -> dict | None:
    """Resolve IP, tags, account, and region from the org Config Aggregator."""
    expression = (
        "SELECT resourceId, accountId, awsRegion, "
        "configuration.privateIpAddress, "
        "configuration.publicIpAddress, tags "
        f"WHERE resourceId = '{instance_id}' "
        "AND resourceType = 'AWS::EC2::Instance'"
    )
    resp = config.select_aggregate_resource_config(
        Expression=expression,
        ConfigurationAggregatorName=AGGREGATOR_NAME,
        Limit=1,
    )
    results = resp.get("Results", [])
    return json.loads(results[0]) if results else None


def handler(event, _context):
    for record in event["Records"]:
        body = json.loads(record["body"])
        detail = body["detail"]
        instance_id = detail["instance-id"]
        state = detail["state"]

        details = get_instance_details(instance_id)
        if not details:
            # Terminated instances may already be gone from the aggregator;
            # fall back to event data for deregistration.
            if state == "terminated":
                deregister_dns(instance_id)
            continue

        ip = (
            details.get("configuration", {}).get("privateIpAddress")
            or details.get("configuration", {}).get("publicIpAddress")
        )

        if state == "running" and ip:
            register_dns(instance_id, ip, details.get("tags", []))
        elif state == "terminated":
            deregister_dns(instance_id)


def register_dns(instance_id, ip, tags):
    # Replace with your Infoblox / Route 53 / IPAM call.
    # Infoblox example: POST to the WAPI record:a endpoint.
    print(f"REGISTER {instance_id} -> {ip}")


def deregister_dns(instance_id):
    print(f"DEREGISTER {instance_id}")

One thing worth calling out: that SELECT against the aggregator resolves a resource across every account and region in a single call. That is the capability that collapses the whole cross-account enrichment headache into a one-liner. I keep coming back to it for other problems now.

And notice the fallback in the handler. A terminated instance has often already dropped out of the aggregator by the time you look, so you cannot rely on the lookup for deregistration. You fall back to the instance ID from the event. Small thing, but it will bite you in testing if you miss it.

Why there is an SQS queue in the middle

You can target the Lambda straight from the rule, and for a demo I would. In production I put SQS in between on purpose:

A bulk launch, a scale-out, a StackSet rollout, can throw hundreds of events at you at once. SQS soaks that up and lets the Lambda drain at its own pace instead of getting hammered.
When Infoblox or the aggregator has a bad minute, messages retry and eventually land in a DLQ you can actually look at. Nothing quietly disappears.
Batching. Ten events per invocation, fewer cold starts, lower cost.

Decoupling ingest from processing is most of what separates "worked in the demo" from "still working at 2am during a mass launch."

What actually came out of it

Detection and DNS updates happened within seconds, across the whole org.
Adding an account or region was a single forwarding rule. No Lambda, no queue, no Config setup per account.
Registration and cleanup were fully automated across the instance lifecycle, nobody touching DNS by hand.
Cost tracked instance activity, not the clock, because there is no idle compute.
The shape is reusable. Same hub-spoke plus aggregator-enrichment works for CMDB sync, security tagging, IPAM updates, anything org-wide and event-driven.

The part worth stealing

Take Infoblox out of the picture and you are left with a pattern I now reach for often:

Fan org-wide events into one hub, then enrich them from a data source the organization already has, instead of building a new one.

Two moves do the work:

A custom event bus with an aws:PrincipalOrgID resource policy gives you cross-account fan-in and a one-rule spoke footprint.
The Config Aggregator, used as an enrichment lookup, resolves any resource in any account or region without an assume-role fan-out.

So the next time you are about to copy the same Lambda into 40 accounts, stop and ask whether a fan-in plus an aggregator you already run could do it with one rule per account instead. Usually it can.

Wrapping up

Honestly, none of the individual pieces here are clever. Cross-account EventBridge is documented. Config aggregators are documented. The win was in composing them so the per-account cost dropped to near zero and the enrichment data was something we already had lying around. That is where most of the good architecture calls seem to live, not in the shiny new component, but in the one you talked yourself out of building.

If you end up adapting the enrichment step to Route 53, a CMDB, or your own IPAM, I am happy to talk through it in the comments.

TruffleHog Secret Scanner Automation

Tue, 27 Oct 2026 16:34:53 +0000

Context and Problem

An AWS EKS Cluster was attacked, and its secrets were compromised. The Kubernetes Secrets were not encrypted, anyone who works with Kubernetes know they are only base64-encoded by default.

Task

I had a mission to scan 115 repositories across GitHub and Azure DevOps for secrets exposed anywhere in the Git history.

The goal was simple: identify exposed secrets and provide actionable reports for the engineering team to fix them.

A colleague recommended a tool called TruffleHog, which became the foundation of the workflow.

TruffleHog

"TruffleHog is a secrets scanning tool that digs deep into your code repositories to find secrets, passwords, and sensitive keys."

— TruffleHog official website

My Workflow

Read the TruffleHog documentation;
Ran a test in a personal GitHub repository to see how the tool works;
Check the total amount of repositories to scan, 115;
Designed how I was going to complete the Task:

- Decided to automate the process using Bash and jq.
- The script would receive the repository URL as a parameter.
- Used jq to filter and mask findings.
- Used jq to generate separate JSON files for active and inactive secrets.
- Generated standardized Notion reports to support remediation workflows.
- Used the generated JSON data with Gemini to create the reports.

Report Example

Lessons Learned

- Standardizing reports made remediation much faster.
- Active and inactive secrets should be separated to improve prioritization.
- Automation becomes essential when dealing with dozens of repositories.

Why Bash and jq?

Keep the workflow simple and easy to run.

jq made it easy to:

- filter findings
- mask sensitive values
- split active and inactive secrets
- generate structured outputs

Check the code

https://github.com/ja-lourenco/trufflehog-scan

A week of green runs, and the pipeline was waiting for me

Tue, 27 Oct 2026 16:02:11 +0000

On 1 September at 10:57 UTC the monthly ingest ran. It opened issue number 3 of the magazine, picked the vertical from the source registry, wrote 19 candidate sources into Postgres with stage = 'discovery', and opened a GitHub issue telling me it was my turn.

Every morning after that, the daily job ran and succeeded. Eight consecutive green runs, from the first of the month to the eighth. The article count for September stayed at zero the whole time.

The gate was doing its job

The daily job does not decide anything by itself. It asks the database which stage to run, and the answer depends on a view:

SELECT id, issue_id, source_url, tier, independent
FROM signals
WHERE stage = 'verify' AND (tier = 1 OR tier = 2 AND independent);

Nothing satisfies that view until I sit in Supabase Studio and tag the sources by hand. Tier 1 is a primary source, a regulator or the organisation that ran the thing. Tier 2 counts only when it is independent of whoever it describes, which rules out a consultancy writing up its own project as a case study.

That judgement is mine on purpose. It is the editorial bar of the magazine, and I did not want a model deciding what counts as proof of anything.

So the decision function found a draft issue with no verified signals, returned niente, and printed its reason to stderr:

advance: signal in attesa di verifica in Studio

Then it exited zero, which is correct. There was nothing for it to do.

What my alarms actually watch

I have three, and I was fairly pleased with them.

A cron that fails opens a GitHub issue on its own. A cron that never starts is caught by a daily watchman that asks the GitHub API when each schedule last fired, because a schedule that GitHub has quietly disabled produces no failure to report. And because a watchman inside the same failure domain is not a watchman, that job also checks in with a Sentry monitor that alarms on silence from outside GitHub.

Between them they cover a job that breaks and a job that never runs. Neither of those happened. What happened was a job that ran, succeeded, and did nothing, for seven days, because it was waiting for a person who had not noticed.

The information was already there

The part I find annoying is that nothing was hidden. The reason is written every single morning, in the run log, in Italian, on stderr. The job knows exactly why it is idle and says so out loud.

What it does not know is how long it has been saying it. niente on the second of the month and niente on the eighth are the same string, and the only thing that separates them is a week of my attention going somewhere else. No alarm reads that log line, and no counter turns it into a duration.

Green means the machine is healthy. I had been reading it as "the work is moving", and those are different claims.

How it ended

On the eighth I opened Studio and tagged 11 of the 19 sources: two from the regulator as tier 1, nine analyst reports as tier 2 and independent. I left out a vendor case study, which fails the independence test by construction, and seven security bulletins whose scraped body is only a title.

After that the pipeline did the rest in a few minutes. Generate wrote the piece from eight sources, embed rebuilt the chunks, export produced the two markdown files, and the judge on the pull request failed me for weak attribution, which is a story for another post.

The September issue went out a week late. Nothing had broken, and every run had been green.

So I do not need a fourth alarm. I need the daily job to count the mornings it has spent answering niente, and to say that number where I actually look.

Node 22.22 runs TypeScript. Ubuntu's Node 22.22 doesn't.

Tue, 27 Oct 2026 14:27:58 +0000

Four test files in my site's repo import TypeScript directly. faq.test.mjs pulls in src/lib/faq.ts, and three others do the same with their own modules. There is no build step and no loader in between: the script is node --test "test/*.test.mjs", and type stripping has been on by default since Node 22.18.0, so Node strips the types and runs what's left.

In CI those tests pass. The workflow asks actions/setup-node for node-version: "22", and the latest runs on main are green.

On my Ubuntu server, with Node 22.22.1, all four fail before a single assertion runs:

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for .../src/lib/faq.ts

22.22.1 is newer than 22.18.0, so on paper the feature is there. The version number was right. The binary behind it was a different build.

Same version, different binary

Node will tell you whether it strips types. The v22 docs describe process.features.typescript as "strip" by default, "transform" with --experimental-transform-types, and false if Node is started with --no-experimental-strip-types.

I did not pass that flag:

$ /usr/bin/node --version
v22.22.1
$ /usr/bin/node -p 'process.features.typescript'
false

That node comes from Ubuntu's nodejs package, version 22.22.1+dfsg+~cs22.19.15-1ubuntu1. Asking for the feature explicitly does not bring it back:

$ /usr/bin/node --experimental-strip-types t.ts
node:internal/util:226
    throw new ERR_NO_TYPESCRIPT();

To rule out the version itself, I downloaded the official 22.22.1 build from nodejs.org, checked it against SHASUMS256.txt, and asked the same question. It printed strip and ran the same .ts file without complaint.

Where TypeScript went

The package's own changelog explains most of it. The December 2024 entry, 22.12.0+dfsg-1, says:

dfsg-exclude amaro, build without it - it requires swc. This disables the ability to execute TypeScript files using the --experimental-strip-types flag.

The November 2025 entry, 22.21.1+dfsg+~cs22.19.0-1, says: "we build without-amaro for now, disable strip-types".

The March 2026 Debian entry, merged into the version I have, says: "Drop 'no amaro' patch, solved upstream". The build still answers false, and it reports how it was compiled:

$ /usr/bin/node -p 'process.config.variables.node_use_amaro'
false

The official 22.22.1 build prints true.

The changelog doesn't say what "solved upstream" changed. My tests only see what the build does.

Why CI never saw it

The setup-node README says the action first checks the runner's tool cache, then pulls LTS versions from the actions/node-versions releases and, on a miss, falls back to downloading from nodejs.org. None of those is Ubuntu's package. The four test files pass in CI, so the Node there strips types. The one on my server does not, and both report major version 22.

That is how one repository can be green in CI and red on my server without a line of code changing.

The fix

I extracted the official Node 22.23.2 tarball into /usr/local. /usr/local/bin comes before /usr/bin in PATH, so node now resolves to the official build, with npm 10.9.8 bundled.

I left the Ubuntu package installed. On this machine 15 installed packages depend on nodejs: handlebars and 14 node-* libraries. Removing it to get the right node would have meant fighting apt over things that have nothing to do with my site.

Same four files, official binary:

# tests 42
# pass 42
# fail 0

Ask Node what it can do

If a project relies on type stripping, node --version is not the check. This is:

node -p 'process.features.typescript'

On the two builds above it prints strip when the feature works and false when it doesn't, whatever the version string says. It belongs in a setup script next to the version check, because on my server the version check was the one that passed.

Three ways a jq one-liner lies to your bash script

Tue, 27 Oct 2026 13:00:00 +0000

Most "the filter looks right but the script misbehaves" bugs come down to three silent failure modes.

First, a missing -r. jq outputs JSON by default, so a string arrives with its quotes attached — your variable holds "web-01", quotes included, and a comparison against web-01 never matches. Any value headed into a shell variable, filename, or test wants -r.

Second, missing keys don't error. jq emits null, which lands in bash as a literal four-character string — non-empty, so [ -n ] guards pass and backup-null.tar.gz gets written. The // operator supplies a real fallback.

Third, select() compares exactly. A value that is almost right matches nothing, and nothing is not an error:

jq -r '.items[] | select(.active == true) | .name'

If the API sends "active": "true" as a string, that item is skipped. If you match "Web-01" against "web-01", zero elements come back. Either way jq prints nothing and exits 0, so the silence reads as success.

All three share the same fingerprint: exit code zero, plausible output, wrong result. The fastest way I've found to catch them before they ship is building the filter against the actual response: this interactive jq filter builder with a live in-browser preview lets you click through your real JSON to build the path, handles the -r, the // fallback and select() quoting for you, and shows exactly what the filter emits before anything runs.

How I Built a 24/7 AI DevOps Coding Agent for About CAD 200

Tue, 27 Oct 2026 05:18:01 +0000

Hermes + Claude Code + Telegram on a cheap always-on thin client.

Hi, I'm Amir

I'm a DevOps engineer, and a surprising amount of my time goes into work that is important but repetitive.

A server needs patching. A CI/CD pipeline fails. A script needs a small change. A Terraform module needs an update. A tool I built needs one more feature. Tests need to be run, branches need to be checked, and somebody still has to make sure the generated code actually works.

None of those jobs is necessarily difficult on its own. The annoying part is the constant context switching.

I kept thinking:

What if I had a small AI worker that was always online, understood my repositories, could call a real coding agent, verify the result, manage Git, and report back to me on my phone?

That is what this setup became.

I use Hermes as the agentic AI harness and orchestrator. Hermes is connected to OpenRouter, where I use a low-cost DeepSeek Flash v4 model for the orchestration layer. When real coding work is needed, Hermes delegates that work to the Claude Code CLI, which is my main coding agent.

The split is intentional:

Claude helps me think through the problem. Claude Code does the main coding. Hermes runs the workflow, performs QA, manages Git, pushes the repo, and keeps me updated through Telegram.

This is not meant to replace me as an engineer. It is meant to remove the repetitive overhead around the work.

How I divide the work between Claude, Hermes and Claude Code

I deliberately do not ask one AI agent to do everything.

When I have a new idea, bug, or feature, I normally start by talking it through with Claude. I explain what I want, what is currently broken, what constraints matter, and what I do not want changed.

Claude helps me turn that into a clear, compact prompt for Hermes.

Then I hand the task over.

Claude: planning and shaping the task

Claude is where I usually work through questions like:

What exactly should change?
What should stay untouched?
What edge case am I forgetting?
What is the smallest useful test?
What should count as PASS?
What instructions does Hermes actually need?

I don't want a giant five-page prompt. By the time I send the task to Hermes, the thinking should already be distilled into something actionable.

Hermes: orchestration, QA and Git management

Hermes is my manager layer.

Its job is not to out-code Claude Code. Its job is to keep the job moving.

In my workflow Hermes is responsible for things like:

checking the current repo and branch
launching the coding agent
monitoring the task
running the relevant tests itself
verifying that the result actually matches the request
checking Git status
committing only after the task passes
pushing the branch
reporting PASS / FAIL, issues and next steps back to Telegram

That separation matters to me:

The agent that writes the code is not the same layer that decides the job is finished.

Claude Code: the main coding agent

When the work gets into the actual codebase, Claude Code CLI is my primary coding agent.

That is where I want the heavier coding intelligence spent:

reading the codebase
implementing features
fixing bugs
changing tests
debugging failures
reasoning about implementation details

Hermes stays relatively lightweight and cheap. Claude Code does the expensive coding work only when I actually need it.

A real prompt from my workflow

Here is a real example.

After discussing the issue with Claude, this was the kind of short prompt I sent to Hermes:

In text form:

On feature/patch-all: after app restart, an analysis left RUNNING is not
marked INTERRUPTED and Analyze/Patch stay disabled. Startup recovery from
faa2ae6 isn't working on a real DB (schema v7, existing runs). Fix + test
with a DB containing a RUNNING analysis before startup.
Full suite + ruff. Commit after PASS, push. Report PASS/FAIL | issues | next.

I like this kind of prompt because it is short but still gives Hermes everything it needs:

where the problem is → what the failure looks like → what must be tested → when Git operations are allowed → what I want reported back.

Hermes does not just trust the coding agent

This is the part that made the setup useful for me rather than just interesting.

After Claude Code finishes, Hermes still has work to do.

Here is a real result from my Telegram workflow:

In this example Hermes exercised the fallback path, brought up the required containers, reran the integration test against the real environment, cleaned things up, and then handled the Git commit.

That is exactly the kind of repetitive workflow I want the orchestrator doing for me.

I want Claude Code focused on implementation.

I want Hermes focused on:

implement
   ↓
verify
   ↓
test
   ↓
inspect
   ↓
commit
   ↓
push
   ↓
report

Telegram is my remote control

Telegram is what makes the whole setup feel different from simply running an AI coding tool on my laptop.

I can be somewhere else and send something like:

Check the latest branch.

Have Claude fix the regression.

Run the targeted tests.

If everything passes, commit and push.

Report PASS/FAIL, issues, and next.

Then I can put my phone away.

The actual work is happening on the thin client at home.

I don't need a remote desktop session. I don't need VS Code open. I don't need my laptop to stay awake.

Telegram is just the control surface.

The hardware cost me about CAD 200

I deliberately did not build an expensive home server for this.

The machine only needs enough resources to run Linux, Hermes, Git, development tools, containers when needed, and the local Claude Code CLI. The heavy LLM computation happens remotely.

I bought a used Dell Wyse 5070 thin client for about CAD 120.

Part	My setup	Approx. cost
Thin client	Dell Wyse 5070	CAD 120
CPU	Intel Celeron J4105, 4 cores	included
RAM	4 GB physical RAM	included
Internal storage	32 GB	included
Wireless	Linux-compatible USB Wi-Fi adapter	CAD 20
External storage	256 GB external SSD	CAD 60
Approx. total		CAD 200

The original 32 GB storage was too small once Docker, repositories, package caches and development tools entered the picture.

I mounted the external SSD at:

/data

and moved the storage-heavy workloads there:

/data
├── docker
├── containerd
├── projects
└── swapfile

My project path points to the SSD:

~/projects -> /data/projects

The Wyse has 4 GB of physical RAM, so I also added a 4 GB swap file on the external SSD.

That gives the OS 4 GB RAM + 4 GB swap to work with.

Swap is obviously not the same as real RAM, but for a cheap orchestration box it gives me useful headroom during temporary spikes.

Installing Hermes on Ubuntu 26.04

I run the box on Ubuntu 26.04.

For Hermes itself, I followed the official installation method from the Hermes Agent website.

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

After installation, the important pieces in my setup are:

Hermes Agent running on the thin client.
OpenRouter as the model provider for Hermes.
DeepSeek Flash v4 as my low-cost orchestration model.
Telegram connected so I can talk to Hermes remotely.
Claude Code CLI installed and authenticated as the main coding agent.
Git / GitHub access configured so Hermes can manage the repository workflow.

The main idea is simple:

Spend the expensive model on the coding problem. Keep orchestration lightweight.

What happens when the first coding attempt fails?

One thing I did not want was this:

Claude Code says "done"
        ↓
Hermes trusts it
        ↓
bad code gets pushed

That defeats the point of having an orchestration layer.

My preferred flow looks like this:

If QA fails, Hermes has evidence it can hand back to the coding agent:

failing tests
command output
wrong behavior
dirty Git state
integration failure
missing expected changes

The coding agent gets another chance to fix the actual failure.

Only after Hermes can reproduce a passing result does the workflow move to commit and push.

I find that much more useful than simply asking the coding model:

"Are you sure it works?"

How I keep the running cost under control

The hardware cost is only one part of the story.

What matters more over time is where I spend model tokens.

Hermes mainly needs to do things like:

inspect the current state
decide which tool or agent should run next
wait for a result
run tests
check Git status
retry when something fails
report back to me

For that layer I use a cheaper model through OpenRouter.

I save Claude Code for the part where the extra intelligence actually matters: understanding and changing the codebase.

Cheap model
    ↓
orchestration / routing / checking

Expensive coding model
    ↓
used only when real coding is required

I also don't need to pay for a cloud VM just to keep the agent online because the Wyse is doing that job at home.

My ongoing cost is therefore mostly whatever I spend on Claude/Claude Code and OpenRouter usage rather than another always-on server bill.

Guardrails I use before code gets pushed

I want this system to save time, not create a faster way to break things.

Some rules I like are:

No commit before QA passes.
No push before the working tree is in the expected state.
Test the changed behavior, not just whether the code imports.
Prefer a small targeted regression test first.
Use the existing full test suite when appropriate before finalizing.
Report failures instead of hiding or working around them.
Keep production credentials and private keys outside prompts and public repos.
Do not let the coding agent silently change unrelated parts of the project.

For my personal projects, I also try not to turn every small change into a giant enterprise QA exercise.

The goal is enough verification to catch the mistake without spending more time testing than the original task was worth.

Where this setup works well — and where it doesn't

This setup is a good fit for work that is:

repetitive
repository-based
testable
scriptable
safe to perform from a development machine
easy to describe with a clear PASS/FAIL condition

Examples include:

small internal tools
bug fixes
repetitive patching utilities
CI/CD helper scripts
Terraform changes in non-production environments
report generators
automation around Git and testing

It is not something I would blindly point at production and tell:

"Do whatever you think is best."

For high-risk production changes, security-sensitive operations, destructive database work, or anything with a large blast radius, I still want explicit human review and normal change controls.

The agent is useful because it removes repetitive work.

It does not remove engineering judgment.

A few things I learned while building it

1. The orchestration machine does not need to be powerful

At first it is easy to think "AI agent" means "expensive AI hardware."

In this design, it doesn't.

The thin client mostly needs to stay online, run tools reliably, and have enough storage.

2. Separating coding from verification is valuable

Claude Code is very good at coding.

That does not mean I should automatically accept its own definition of "finished."

Having Hermes independently run the checks gives the workflow a much cleaner boundary.

3. A short prompt can be better than a huge prompt

Because I discuss the idea first, the prompt I send to Hermes can stay focused.

That reduces noise and makes failures easier to understand.

4. Reliability matters more than raw speed

The Wyse is not fast.

But it is always there.

For this use case, always available + predictable is more valuable to me than having a much faster machine that disappears when I close my laptop.

5. The best automation is the one I actually use

The biggest win is not that the architecture looks clever.

It is that I can send a task from Telegram, walk away, and come back to a tested branch instead of spending another hour doing repetitive setup and Git work.

Final thoughts

I did not build this because I wanted an AI experiment sitting in a corner of my house.

I built it because I'm a DevOps engineer and I have repetitive work.

Sometimes the right answer to a repetitive task is a Bash script.

Sometimes it is Terraform.

Sometimes it is a small Python tool.

And now, sometimes I can simply describe the problem, let my agent workflow build or fix the tool, verify the result, and give me the branch when it is ready.

My setup has a very simple division of responsibility:

I bring the problem. Claude helps shape the solution. Claude Code writes the code. Hermes manages the job, verifies it, handles Git, and reports back to me.

The Wyse provides the always-on home.

Telegram gives me access from anywhere.

The cloud models provide the intelligence.

A cheap little thin client became my 24/7 AI DevOps coding worker.

Source / setup files: Amiri83/hermis on GitHub

Original Gist: My 24/7 AI DevOps Coding Agent

IBM Z Atlas Uses AI to Prove Mainframe Changes Are Safe Before They Hit Production

Tue, 27 Oct 2026 05:00:00 +0000

Changing a mainframe is slow for a reason. One middleware upgrade can touch dozens of applications. And the knowledge of how everything connects often lives in the heads of a few senior engineers, many of whom are close to retirement.

IBM is going after that problem. At its TechXchange conference today, the company announced IBM Z Atlas. It maps a client's entire z/OS stack, from infrastructure to applications to operations. Then it uses that map, plus AI, to help teams plan, test, and document changes before they reach production.

Atlas enters public preview today. General availability is targeted for Q1 2027.

I sent Andy Bradfield, VP of IBM Z Hybrid Cloud, questions about how Atlas works, where AI fits, and what developers need to know. I edited his answers for length and clarity.

What is IBM Z Atlas, and what problem does it solve for developers and architects working on z/OS?

Andy Bradfield: Most Z teams can see pieces of their environment today. But no end-to-end solution understands how the entire IBM Z stack fits together, so teams can't understand, change, modernize, or govern mission-critical mainframe environments.

Z Atlas is an AI-powered system intelligence and change orchestration companion for IBM Z. It maps a client's entire z/OS stack across infrastructure, applications, and operations. It then uses that intelligence to reduce manual effort for tasks like patch management, staff onboarding, and regulatory response.

The leading cause of unplanned z/OS outages is changes that aren't proven to be safe. Z Atlas addresses this by mapping how a change affects the stack, testing and validating it, and generating the evidence needed for approval. Experts stay in control, and everyone works from the same view of the system.

How does a team figure out the impact of a change today, and how is that different with Atlas?

Bradfield: It often takes weeks to prove a change manually. Understanding the full impact means tracing dependencies across multiple domains and roles in the Z stack. Teams collect information from multiple sources, identify what could be affected, build test plans, and document evidence before they can deploy a change safely.

It's an arduous process. If it isn't done properly, it slows teams down or adds risk. Some clients cut corners, knowingly or by mistake, and end up testing in production.

With Z Atlas, teams already have the visibility on screen. They can generate the plan and automatically prove the change and tests. A developer can also spin up an isolated environment to investigate further.

Where do the time savings come from?

Bradfield: Results from our early design program include:

60–80% less time spent gathering context before a change
Detailed impact analysis and test planning reduced from hours to minutes
Provisioning and test generation reduced from weeks to hours
Change evidence that cuts hours of manual tracking

What data does Atlas pull from, and how does it stay current?

Bradfield: Atlas pulls from existing Z software and ecosystem solutions across system configuration, operations, performance, modernization, and development workflows. It uses configuration-as-code and environment metadata to discover and map relationships across the environment.

It combines that discovered context with AI-driven analysis and agentic capabilities. The result is shared system intelligence for every team. It doesn't replace the specialized capabilities those other tools provide.

Where does AI do the work, and where does Atlas rely on deterministic data? How do you keep a wrong AI answer from becoming a production problem?

Bradfield: Atlas uses AI where reasoning and synthesis add value. That means interpreting what the user wants, connecting context, explaining potential impact, and generating recommended change or validation plans.

The underlying facts come from deterministic sources. That includes discovered topology, configurations, software levels, dependencies, policies, and results from systems of record.

The customer's existing deployment and change-management systems continue to govern production execution. Atlas exposes the supporting evidence and assumptions, keeps people and existing change controls in the decision path, and uses isolated testing and validation before anything reaches production.

What does a real use case look like?

Bradfield: A z/OS system programmer can use Atlas to evaluate how a middleware upgrade might affect dependent applications. They can identify which systems belong in a test plan, or coordinate validation across application, infrastructure, and operations teams.

Atlas avoids the most time-consuming parts of enterprise change: manual analysis, planning, research, provisioning, testing, and documentation.

For patch planning, Atlas can help assess PTF impact, plan and test changes, and capture readiness evidence. For onboarding, it can help new system programmers understand an environment when veteran knowledge isn't available.

The announcement talks about "proving" change. What does that proof look like, and who uses it?

Bradfield: Atlas analyzes potential impacts, identifies affected dependencies, generates the required validation and testing activities, and captures the results in a complete change record. That record can include impacted resources, recommended validation steps, test coverage, test results, approvals, and deployment readiness.

It creates end-to-end traceability from the proposed change through validation. Change managers, approvers, compliance teams, and auditors get documented proof that the change was analyzed, the right systems were evaluated, testing was done, and the results supported deployment.

Where does Atlas run, and does any data leave the customer's environment?

Bradfield: Atlas is on-prem. No data leaves the customer environment.

The Atlas server is a containerized stack. It runs on IBM Z (s390x) or x86_64, deployed on Red Hat OpenShift through the Atlas Kubernetes operator, or on any CNCF-conformant Kubernetes cluster. The AI layer runs through IBM watsonx Assistant for Z, using Gemma 4 31B and IBM Granite 4.1 models.

Discovery supports z14 and later. Provisioning is expected to support Linux on Z, Linux on x86_64, and z17 environments.

How does Atlas fit into IBM's broader plans for AI agents on Z?

Bradfield: IBM's AI operating model rests on four systems working together: agents, data, automation, and hybrid infrastructure. Today's announcements, including Z Atlas, extend our portfolio to deliver on that. It starts with the trust and control every agent needs: a verified identity, only the access it needs, and one place to discover, secure, and evaluate agents.

What are early users saying?

Bradfield: One client told us: "Within minutes, Z Atlas gave me a complete map of my z/OS environment — infrastructure to application — the kind of picture that used to live only in the heads of my most experienced engineers."

Another said: "I don't just make changes anymore; I prove them. Z Atlas generates the plan and the tests automatically, and can spin up an isolated environment when I need to go further."

When is it available, and what do teams need?

Bradfield: Public preview opens October 27. General availability is targeted for Q1 2027. Atlas requires z/OS 3.1 or later. It uses configuration discovery from z/OS and supported middleware, including Db2, CICS, IMS, MQ, and z/OS Connect.

Atlas is packaged as one integrated offering that brings together the AI, discovery, agent, skill, and provisioning pieces while keeping the deployment footprint small. We'll publish specific middleware levels, dependencies, and entitlements closer to GA.

Developers who want to try it can join the public preview waitlist.

What This Means for Developers

Atlas tackles a real problem. Mainframe change management depends on tribal knowledge, and that knowledge is walking out the door. A shared, current map of the z/OS stack is useful on its own, before any AI gets involved.

The most important design choice is the split between AI and facts. Atlas uses models to reason, explain, and draft plans. But the dependencies, configurations, and software levels come from discovered data and systems of record. And deployment still runs through your existing change controls. That's the right order for systems that move money and keep airlines running.

A few things to keep in mind. The performance numbers come from IBM's early design program, not independent testing. GA is still months away, and pricing and entitlement details aren't public yet. The z/OS 3.1 requirement will also rule out some shops for now.

If your team spends weeks proving changes, or you're about to lose the one engineer who knows how everything connects, the preview is worth a look.

Limpando ~20 GB de caches de Terraform e Node.js com um único comando Bash

Mon, 26 Oct 2026 18:03:24 +0000

Mantenho todos os meus repositórios em uma única pasta — cerca de 150, entre trabalho para clientes, templates e experimentos. Cada um deles é uma pequena armadilha: um node_modules dentro de app/ e um .terraform dentro de infra/terraform/. Nenhum dos dois é versionado, ambos são reconstruídos em segundos (bem, minutos), e juntos tinham comido mais de 20 GB do meu disco sem que eu percebesse.

O gatilho foi banal: df -h mostrando 94% de uso numa partição de 233 GB, com 10 GB livres. A correção também deveria ser banal. Em vez de rodar um find ... -exec rm -rf do qual eu me arrependeria, escrevi um comando pequeno em que eu pudesse confiar — e mantive.

Este artigo traz o script, o raciocínio por trás de cada proteção e o que aprendi ao rodá-lo.

O objetivo

Um comando, de qualquer lugar, que:

Varra ~/Documentos/github (em qualquer profundidade — incluindo monorepos e pastas agrupadas).
Encontre apenas dois tipos de diretório:
- */infra/terraform/.terraform
- */app/node_modules
Informe o tamanho de cada um e os totais.
Delete esses diretórios — mas somente quando isso for pedido explicitamente.

Decisões de projeto

1. Dry-run como padrão: apagar cache de build é reversível, mas executar a operação em 30 diretórios distribuídos por 150 repositórios exige cautela. O parâmetro --delete é obrigatório para efetivar as exclusões.

2. find -prune em vez de du recursivo: evita percorrer diretórios como .git, node_modules e .terraform, mantendo a varredura eficiente mesmo com milhares de arquivos.

3. Whitelist de caminhos antes de apagar: o rm -rf só é executado em caminhos que correspondem exatamente aos sufixos permitidos e estão dentro da raiz varrida. Qualquer caminho fora dessas regras é ignorado e reportado.

4. Medição de tamanho em paralelo: executa o du em paralelo usando xargs -P "$(nproc)", aproveitando os núcleos disponíveis e reduzindo o tempo de execução em árvores grandes.

5. Saída legível, cálculos precisos: os tamanhos são somados em bytes e formatados com numfmt, evitando divergências causadas pela soma de valores arredondados.

6. Espaço livre antes e depois: compara o espaço disponível no disco antes e depois da limpeza para verificar o ganho real obtido.

Um detalhe que vale destacar: .terraform.lock.hcl é um arquivo e não é tocado. Ele fixa as versões dos providers e deve ser versionado. Apenas o diretório .terraform/ (cache de providers e módulos) é removido.

O script

Salve como limpar-cache, dê permissão de execução e coloque em /usr/local/bin:

#!/usr/bin/env bash
#
# limpar-cache — varre um workspace e levanta/remove:
#   * <projeto>/infra/terraform/.terraform
#   * <projeto>/app/node_modules
#
# Uso:
#   limpar-cache                 # só relatório (dry-run, nada é apagado)
#   limpar-cache --delete        # apaga, pedindo confirmação
#   limpar-cache --delete --yes  # apaga sem confirmação
#   limpar-cache --only node     # apenas node_modules
#   limpar-cache --only terraform
#   limpar-cache --base ~/outro/dir
#

set -uo pipefail

BASE="${HOME}/Documentos/github"
ONLY="all"
DO_DELETE=0
ASSUME_YES=0
JOBS="$(nproc 2>/dev/null || echo 4)"

usage() {
  cat <<'EOF'
limpar-cache — relatório e limpeza de caches em ~/Documentos/github

Alvos:
  */infra/terraform/.terraform
  */app/node_modules

Opções:
  -d, --delete         remove os diretórios encontrados (com confirmação)
  -y, --yes            não pede confirmação (implica --delete)
  -n, --dry-run        apenas relatório (padrão)
      --only TIPO      all (padrão) | terraform | node
      --base DIR       raiz a varrer (padrão: ~/Documentos/github)
  -h, --help           esta ajuda

Exemplos:
  limpar-cache
  limpar-cache --only terraform
  limpar-cache -d -y
EOF
}

while (($#)); do
  case "$1" in
    -d | --delete) DO_DELETE=1 ;;
    -y | --yes) ASSUME_YES=1 ;;
    -n | --dry-run) DO_DELETE=0 ;;
    --only)
      ONLY="${2:-}"
      shift
      ;;
    --base)
      BASE="${2:-}"
      shift
      ;;
    -h | --help)
      usage
      exit 0
      ;;
    *)
      printf 'limpar-cache: opção inválida: %s\n\n' "$1" >&2
      usage >&2
      exit 2
      ;;
  esac
  shift
done

case "$ONLY" in
  all | terraform | node) ;;
  *)
    printf 'limpar-cache: --only deve ser all, terraform ou node\n' >&2
    exit 2
    ;;
esac

((ASSUME_YES)) && DO_DELETE=1

if [[ ! -d "$BASE" ]]; then
  printf 'limpar-cache: diretório não encontrado: %s\n' "$BASE" >&2
  exit 1
fi

BASE="$(realpath -m -- "$BASE")"

human() { numfmt --to=iec-i --suffix=B --format='%.1f' "$1" 2>/dev/null || printf '%sB' "$1"; }

avail_bytes() { df -B1 --output=avail -- "$BASE" 2>/dev/null | tail -1 | tr -d ' '; }

# ---------------------------------------------------------------- coleta -----
# -prune evita descer em .git, node_modules e .terraform (rápido mesmo com 150 repos)
TF_DIRS=()
NODE_DIRS=()
while IFS= read -r -d '' d; do
  case "$d" in
    */infra/terraform/.terraform) TF_DIRS+=("$d") ;;
    */app/node_modules) NODE_DIRS+=("$d") ;;
  esac
done < <(find "$BASE" -mindepth 2 \( -name .git -o -name node_modules -o -name .terraform \) -prune -print0 2>/dev/null)

[[ "$ONLY" == "node" ]] && TF_DIRS=()
[[ "$ONLY" == "terraform" ]] && NODE_DIRS=()

if ((${#TF_DIRS[@]} == 0 && ${#NODE_DIRS[@]} == 0)); then
  printf 'Nada encontrado em %s (filtro: %s).\n' "$BASE" "$ONLY"
  exit 0
fi

TMP="$(mktemp)"
trap 'rm -f "$TMP"' EXIT

# tamanho aparente em paralelo: "tipo<TAB>bytes<TAB>caminho"
sizes_of() {
  local kind="$1"
  shift
  (($#)) || return 0
  printf '%s\0' "$@" |
    xargs -0 -r -n1 -P "$JOBS" bash -c '
      p="$1"
      s="$(du -sb -- "$p" 2>/dev/null | cut -f1)"
      [[ -n "$s" ]] || s=0
      printf "%s\t%s\t%s\n" "$0" "$s" "$p"' "$kind"
}

{
  sizes_of terraform "${TF_DIRS[@]+"${TF_DIRS[@]}"}"
  sizes_of node "${NODE_DIRS[@]+"${NODE_DIRS[@]}"}"
} >"$TMP"

sort -t $'\t' -k2,2nr -o "$TMP" "$TMP"

total_tf="$(awk -F'\t' '$1=="terraform"{s+=$2} END{print s+0}' "$TMP")"
total_node="$(awk -F'\t' '$1=="node"{s+=$2} END{print s+0}' "$TMP")"
total_all="$((total_tf + total_node))"
count_tf="${#TF_DIRS[@]}"
count_node="${#NODE_DIRS[@]}"

# ------------------------------------------------------------- relatório -----
printf '\nRaiz: %s\n' "$BASE"
printf '%-9s %12s  %s\n' "TIPO" "TAMANHO" "CAMINHO"
printf '%-9s %12s  %s\n' "--------" "-----------" "----------------------------------------"
while IFS=$'\t' read -r kind bytes path; do
  printf '%-9s %12s  %s\n' "$kind" "$(human "$bytes")" "${path#"$BASE"/}"
done <"$TMP"

printf '\nResumo\n'
printf '  terraform/.terraform : %3d diretório(s)  %10s\n' "$count_tf" "$(human "$total_tf")"
printf '  app/node_modules     : %3d diretório(s)  %10s\n' "$count_node" "$(human "$total_node")"
printf '  TOTAL                : %3d diretório(s)  %10s\n' "$((count_tf + count_node))" "$(human "$total_all")"
printf '  Livre em disco agora : %10s\n' "$(human "$(avail_bytes)")"

if ((DO_DELETE == 0)); then
  printf '\nModo relatório (nada foi apagado). Para remover: %s --delete\n\n' "$(basename "$0")"
  exit 0
fi

# --------------------------------------------------------------- remoção -----
if ((ASSUME_YES == 0)); then
  printf '\nRemover %d diretório(s), liberando ~%s? [s/N] ' "$((count_tf + count_node))" "$(human "$total_all")"
  read -r reply
  case "$reply" in
    s | S | y | Y | sim | SIM) ;;
    *)
      printf 'Cancelado.\n\n'
      exit 1
      ;;
  esac
fi

is_safe() {
  local p="$1" real
  [[ -n "$p" ]] || return 1
  [[ "$p" != "/" && "$p" != "$HOME" ]] || return 1
  case "$p" in
    */infra/terraform/.terraform | */app/node_modules) ;;
    *) return 1 ;;
  esac
  real="$(realpath -m -- "$p" 2>/dev/null)" || return 1
  [[ "$real" == "$BASE"/* ]] || return 1
  [[ "$real" != "$BASE" ]] || return 1
  return 0
}

avail_before="$(avail_bytes)"
removed=0
failed=0

while IFS=$'\t' read -r kind bytes path; do
  if ! is_safe "$path"; then
    printf '  IGNORADO (fora do padrão/raiz): %s\n' "$path"
    ((failed++))
    continue
  fi
  if rm -rf -- "$path" 2>/dev/null && [[ ! -e "$path" ]]; then
    printf '  removido  %10s  %s\n' "$(human "$bytes")" "${path#"$BASE"/}"
    ((removed++))
  else
    printf '  FALHOU    %10s  %s\n' "$(human "$bytes")" "${path#"$BASE"/}" >&2
    ((failed++))
  fi
done <"$TMP"

avail_after="$(avail_bytes)"
freed=$((avail_after - avail_before))
((freed < 0)) && freed=0

printf '\nConcluído: %d removido(s), %d falha(s).\n' "$removed" "$failed"
printf 'Espaço livre: %s -> %s (~%s liberados)\n\n' \
  "$(human "$avail_before")" "$(human "$avail_after")" "$(human "$freed")"

((failed == 0)) || exit 1
exit 0

Instale e deixe disponível em todo shell:

install -m 755 limpar-cache /usr/local/bin/limpar-cache

E no ~/.bashrc:

export PATH="/usr/local/bin:$PATH"
alias lcache='limpar-cache'

A linha explícita de PATH é redundante na maioria das distribuições, mas não custa nada e deixa a configuração à prova de shells de login esquisitos. (Ela já me salvou uma vez — veja a última lição.)

Rodando

Relatório primeiro, sempre:

$ lcache

Raiz: /home/user/Documentos/github
TIPO           TAMANHO  CAMINHO
--------   -----------  ----------------------------------------
node            1,4GiB  projeto-alfa/app/node_modules
node            1,1GiB  projeto-beta/app/node_modules
terraform     870,7MiB  projeto-alfa/infra/terraform/.terraform
terraform     865,2MiB  projeto-gama/infra/terraform/.terraform
node          771,2MiB  projeto-delta/app/node_modules
...

Depois, quando a lista parecer certa:

lcache --delete          # pergunta: Remover 30 diretório(s), liberando ~20,7GiB? [s/N]
lcache -d -y --only node # apenas node_modules, sem prompt

Uma execução na minha máquina:

Resumo
  terraform/.terraform :   9 diretório(s)      7,1GiB
  app/node_modules     :  21 diretório(s)     13,6GiB
  TOTAL                :  30 diretório(s)     20,7GiB
  Livre em disco agora :      9,9GiB

30 diretórios, ~20,7 GB. O disco saiu de 94% para cerca de 86% de uso — e cada um desses repositórios é reconstruível com npm ci ou terraform init.

Lições aprendidas

1. -prune é o truque inteiro. Um find ~/Documentos/github -name node_modules ingênuo desce em todo node_modules procurando outros aninhados. Em 150 repositórios isso são minutos de I/O inútil. Podar no momento do match transforma tudo em uma única varredura rápida.

2. Tamanho aparente ≠ uso de disco. du -sb dá bytes aparentes (rápido, comparável), enquanto du -s dá blocos alocados (o que o df reporta). Para um relatório de limpeza, o tamanho aparente é o número honesto e previsível; para "quanto eu vou ganhar de verdade", use --block-size=1 sem -b. Eu mostro df --output=avail antes/depois justamente porque os dois diferem.

3. O page cache frio domina o tempo de execução. A primeira execução mediu ~2m26s; a segunda foi bem mais rápida. Nada de errado com o script — são 30 árvores de diretório sendo lidas do disco pela primeira vez. Paralelismo com xargs -P ajuda, mas não vence a física.

4. Duas cópias de um script é um bug esperando para acontecer. Eu tinha instalado em ~/.local/bin primeiro e depois "movi" para /usr/local/bin — mas ~/.local/bin vem antes no PATH, então a cópia antiga sombreava a nova silenciosamente. Correção: mover de verdade (install + rm), não copiar. E conferir com command -v, não com which (que é outro binário, com outra semântica).

5. Proteções são baratas; erros de rm -rf não. A função is_safe() tem ~12 linhas e rejeita /, $HOME, qualquer coisa fora da raiz varrida e qualquer caminho que não termine nos dois sufixos esperados. Ela roda em cada caminho, mesmo os vindos do meu próprio find. Essa redundância é o ponto: o custo é microssegundos, o modo de falha é "apaguei meu diretório home".

6. Whitelist ganha de blacklist. Eu deliberadamente não apago dist/, .next/, .venv/ ou coverage/. Esses têm custos de reconstrução muito diferentes e alguns são artefatos de deploy. Se eu precisar deles depois, cada um ganha seu sufixo explícito no case — nunca uma regra de "apague tudo que pareça cache".

Checklist de recuperação

Depois de uma limpeza dessas, cada repositório precisa de uma coisa:

# Projetos Node
cd app && npm ci          # ou pnpm install / yarn install

# Projetos Terraform
cd infra/terraform && terraform init

O terraform init baixa os providers novamente; se você usa cache de plugins (TF_PLUGIN_CACHE_DIR), será bem mais rápido da próxima vez — vale configurar se costuma limpar .terraform com frequência.

Considerações finais

O script tem cerca de 200 linhas, a maioria delas printf. O que o torna seguro não é sofisticação, e sim um punhado de escolhas entediantes: dry-run primeiro, podar cedo, whitelist de caminhos, confirmar antes de apagar e imprimir o que realmente aconteceu.

Vinte gigabytes não é muito em termos absolutos, mas era 60% do espaço livre que eu tinha. Mais importante: o comando agora existe. Na próxima vez que o disco apertar, é uma linha em vez de uma tarde de du e dúvidas.

Se você mantém muitos repositórios num só lugar, o padrão se generaliza — troque os dois sufixos pelos seus próprios artefatos de build, mantenha o dry-run como padrão e nunca deixe o script adivinhar o que deve apagar.

Drill the Safe Open: How IANA's 2020 Root Key Ceremony Ran Four Days Late

Mon, 26 Oct 2026 15:36:01 +0000

In February 2020 the safe holding the internet's root signing key would not open. IANA's published, hand-initialed ceremony scans show the custody behind the cryptography: a repair script written before anyone drilled, and a signing that took four minutes and fifteen seconds.

Page 6 of the script is a table with four columns: Step, Activity, Initials, Time. In step 5 a locksmith replaces a combination lock, and the first task on the list reads "a) Drill the lock/safe open." Beside it, in blue pen, someone has initialed the box and written a time: 21:48+1.

The safe is Equipment Safe #1 in the Key Management Facility that IANA runs in El Segundo, California. It holds the hardware security modules that keep the private half of the root zone's key-signing key, the trust anchor that validating resolvers chain every signed DNS answer back to. In February 2020 it would not open. IANA publishes the paperwork from its key ceremonies, scanned with the handwriting left in, so the story of how it got into its own safe can be read page by page, initials and all.

Most writing about the root key ceremony is about cryptography. The paperwork is mostly about custody: who held what, who watched, and what got written down when something went wrong. That February the cryptography took four minutes and fifteen seconds. Getting into the safe took most of four days.

How a ceremony runs

The root zone is signed in quarters, and the key-signing key is brought out of its safe four times a year to sign the keys for the next one. The rule in force in 2020, in IANA's DNSSEC Practice Statement, was that each quarter's ceremony is held "approximately 60 days but no later than 33 days, before the time cycle commences."

The people are set out in roles. A Ceremony Administrator runs the script and an Internal Witness keeps the record. Safe Security Controllers hold the combinations, two people for each of the two safes: the Equipment Safe with the hardware, and the Credentials Safe with the safe deposit boxes. Crypto Officers hold the keys to those boxes. The Practice Statement describes "successive barriers in three tiers, requiring at least seven trusted persons from four different roles," and activating a hardware module "requires three out of seven Crypto Officers to extract activation material from above safe deposit boxes using their physical key." The Crypto Officers are Trusted Community Representatives, picked from statements of interest that IANA solicits from the internet community, and its roster page states the rule that keeps them independent: "Persons considered directly affiliated with PTI, ICANN or VeriSign may not become a Trusted Community Representatives."

The script's preamble says how the room behaves. "CA reads each step aloud prior to its performance." "Upon completion of each step, IW announces the time of completion, records the completion time, and initials their copy of the script." And then the paragraph the rest of this story runs on:

Ceremony participants who notice a problem or an error during the ceremony should interrupt the ceremony. Ceremony participants agree on a resolution before proceeding ... Unplanned events (exceptions) during the ceremony are evaluated, documented, and acted upon. It is the CA's sole responsibility to decide on proper actions after consulting with the IW. In either case, an exception is regarded as an incident, and incident handling procedures are enacted.

"Documented" has a form. The Root DNSSEC Script Exception is a one-page sheet. Its first row asks for the Act, the Step, the Page, and the date and time, and its second says "IW describes the exception(s) and action(s) below," with a box for initials and a box for the time on each row. An exception is filed against a place in the script, the way a bug is filed against a line of code.

The day the dial was due for replacement

The trouble surfaced during maintenance, and the maintenance was itself a ceremony with its own script. Administrative ceremonies are smaller affairs, with no signing, but the same rules apply: a Ceremony Administrator, an Internal Witness, cameras, and a participants table. In that week's tables the participants included a locksmith from an outside lock-and-security company.

The work was to replace the combination dials on both safes, swapping each Kaba Mas X-09 for a new X-10. The day before, at a ceremony to program the safe deposit boxes' lock assemblies, one assembly had misbehaved. That got a sheet too:

The last KMF West Lock Assembly was malfunctioning, and we delayed the bagging of the unit to allow more time for the locksmith to troubleshoot. Once the lock assembly was functioning, we placed it in TEB BB91951419 and continued where we left off in the script.

TEB is a tamper-evident bag, and the bag's serial number goes in the record.

The script for the Equipment Safe, dated Tuesday 11 February, has an Act 2 called "Safe #1 Equipment Maintenance." Step 1, initialed at 21:34, has the administrator and the witness carry "a flashlight, phillips screwdriver, change key tool" into the safe room. Step 2 reads "SSC1_A opens Safe #1 while shielding the combination from the camera," with a note that the controller "will begin by rapidly spinning the dial counter-clockwise in order to charge it." The lock powers itself from the dial. Step 4 is the job they had come to do: "a) Remove the current Kaba Mas X-09. b) Install a new Kaba Mas X-10."

Step 2 has no initials. The exception sheet behind it is filed against Act 2, Step 2, Page 6, dated 2020/02/11 and initialed at 23:14:

SSC1_A was unable to open the safe due to malfunctioning hardware. Escorted out by IW2 to verify combination with other SSC. Locksmith contacted manufacturer and they suggested the issue was a faulty logic board and the dial will require drilling. Ended ceremony to discuss steps going forward.

So the dial that failed was the one they had come to replace, and it failed at the one step that needed it to work. A second ceremony a couple of hours later opened the Credentials Safe without trouble and swapped its X-09 for an X-10.

The next day ICANN's blog announced that the 40th ceremony, "originally scheduled for 12 February 2020 in El Segundo, California, has been postponed." "One of the security mechanisms that protects the contents of the secure safes was found to be malfunctioning." "The issue was identified yesterday, 11 February 2020, during routine maintenance of the Key Management Facility." The post said there would be "no interruption to Domain Name System Security Extensions (DNSSEC) service," that "We maintain a complete replica facility in Culpeper, Virginia," and that the repair was scheduled for Friday the 14th.

The branch was written before anyone drilled

The 11 February script had no step for a safe that would not open. Step 2 simply assumed it would. When it failed, nobody improvised the repair in the room. They filed the sheet, ended the ceremony, and wrote a new script.

The repair ceremony began on the 14th, and its Act 2 opens with a section titled "Confirmation of Equipment Safe #1 Dial Malfunction." Before anyone was allowed to drill, the controller tried the safe again, under the cameras, at 17:26. The step carries a note that "Locksmith may provide additional advice/techniques to attempt to operate the safe dial." Then the script splits:

If the safe opens: Complete the safe log by following the steps below ...

If the lock malfunctions: Continue with the steps below.

If the safe had opened on that attempt, the script would have sent everyone to the safe log and the dial swap, and there would have been no drilling. It didn't open. Step 5 begins "The locksmith performs the following tasks to replace the combination lock," and lists them: "a) Drill the lock/safe open," "b) Remove the current Kaba Mas X-09 Safe Dial," "c) Return the safe door to a secure state by performing necessary repairs," "d) Install a new Kaba Mas X-10 Safe Dial." Under the first task sits a note about breaks: "A ceremony break will be performed as required. All personnel will be escorted out of the Tier 5 (Safe Room) and Tier 4 (Ceremony Room.)" Each lettered task has its own box for initials and time.

Drilling into the safe that guards the root key was written down in advance as a numbered, witnessed, timed step, with a branch for the possibility that it wasn't needed.

What the sheets caught

The repair took two days, and the packet of exception sheets behind its script reads like a log of everything the plan didn't specify. I read four of them, all filed against Act 2, Step 5, Page 6. I didn't read every page of the packet, so four is a floor.

The first, dated the 14th at 21:00, records that an external witness "left the room on a break and stayed outside until the next break," returning at 22:19, and adds: "Due to the length of the ceremony carrying into the next day, other non essential participants came and went as well." The second is one sentence: "Ceremony was paused overnight and resumed in the morning."

The third, at 19:26 on the 15th, is about the building. A security guard used the hand scanner outside Tier 3 to escort an external witness in, and that triggered the alarm. The witness was asked to step out and wait in the conference room, "and the alarm was triggered again when he manually opened the door."

The fourth is about the order of steps:

Steps e, f, g, and h were performed immediately after opening the safe for the first time which occurred between steps a and b.

Steps e to h are the safe log: the controller shows the old log to the camera, takes the pre-printed one from the witness, writes the time and signs where it says "Open Safe," and the witness checks and initials the entry. The script had them after the new dial went on. In the room, they did them the moment the drilled door first opened, and the times in the script's own columns agree: a at 21:48, e to h at 21:54, the new dial at 23:56. Doing the log first was arguably the better order, since it put the opening on paper at once. Nobody rewrote the script to match. The deviation went on its own sheet, and the record shows the steps in the order they happened.

The times in that column all carry a "+1," which the script never defines. I read it as the day after the ceremony began, which puts the drilling at 21:48 UTC on the 15th. Three other records agree with that reading: the overnight sheet is dated the 15th at 00:38, IANA's announcement that the safe was open went out at 22:10 UTC on the 15th, and the participants signed the ceremony off at 00:44 on the 16th. The discovery sheet was initialed at 23:14 on the 11th, so the drilling step was initialed 3 days, 22 hours and 34 minutes after it.

IANA's message to the root-dnssec-announce list read: "We have successfully opened the safe with the failed mechanism, and are in the final stages of remediation, with the repair work and installation of a new lock underway. We are therefore arranging resources to start the KSK Ceremony 40 at 0000 UTC, or shortly thereafter."

Four minutes and fifteen seconds

The signing went the way the script said it would. The signer's log for Ceremony 40 opens with "Starting: ksrsigner /media/KSR/KSK40/ksr-root-2020-q2-0.xml (at Sun Feb 16 02:21:33 2020 UTC)." Twenty-two seconds later the hardware module is activated. At 02:25:48 the log reads "Generated new SKR," a signed key response whose first signature period begins on 1 April 2020, made with the key whose tag is 20326. The log prints the response's SHA-256 hash in hex, and then again as a run of English words: "unwind leprosy fracture savagery tactics speculate Neptune miracle slingshot aftermath …"

From start to signed response is four minutes and fifteen seconds. Against the Practice Statement's rule, the latest date for the second-quarter ceremony was 28 February, so the signing finished with nearly twelve days to spare.

The ceremony's closing act has the administrator read every exception out before anyone signs. Step 1 of the participants' signing reads "CA reads all exceptions that occurred during the ceremony." In step 2 each attendee comes to the table and signs the witness's list, and "All signatures declare that this script is a true and accurate record of the ceremony." The read-back comes first. You hear what went wrong before you put your name to the record.

April: nobody could come

Two months later the failure was the people. Ceremony 41 was set for 23 April 2020, in the first weeks of the pandemic. IANA's published questions-and-answers document says there were no representatives for the El Segundo facility "located close enough to allow unrestricted and safe travel to the facility," and says where they were: "Mauritius, Spain, Russia, Tanzania, Uruguay and east-coast United States."

The plan IANA used kept the custody chain and moved the people. "Four trusted community representatives (TCRs) who fulfil Cryptographic Officer roles in the ceremony have transmitted their safe deposit keys to separate ICANN and PTI staff. These staff will act as surrogates, but the TCRs will participate in the ceremony remotely and monitor their credential's use." The keys travelled sealed: "The TCRs have wrapped their deposit keys with opaque material, and then transmitted them in tamper-evident bags. This bag will not be opened until within the ceremony so that each TCR can witness their key is in the same condition as when they released it." The ceremony would sign three quarters at once instead of one, and "The additional generated signatures will be initially withheld."

The plan they didn't use is in the same document. "The first version of our contingency plan assumed the trusted community representatives would not transmit their safe deposit box keys for performance of the ceremony. In such a scenario, a trained locksmith would drill and replace the locks to three safe deposit boxes during the ceremony in order to retrieve the smart cards needed to activate the hardware security module." Two months after the safe, the first version of the fallback was to drill again.

One part of the April plan broke the rules as they stood, and the document says which: "The only element of our revised approach that was inconsistent with the DPS was signing additional quarters of KSRs during the ceremony. The Policy Management Authority, an internal committee that reviews changes to the DPS, convened on 6 April and approved revisions that allow for signing additional quarters of KSRs as a disaster recovery response." The rule was changed by its owner seventeen days before it was needed, and not by the people in the room on the day.

The Ceremony 41 script also adds a step to its closing act. After reading the exceptions, the administrator "asks any COs who are participating remotely if they have any concerns pertaining to the ceremony or exceptions which may have occurred."

By November the bend was standing policy. The Practice Statement in force in February 2020, the fourth edition, has no variance clause. The sixth edition, dated 4 November 2020, carries section 4.5.4.1, and the eighth edition of 2025 still carries it word for word:

In the event of a declaration of a disaster recovery or business continuity response, the adverse event may necessitate variances to the procedural controls and technical security controls. These variances shall be limited in scope to those necessary for the successful conduct of key signing operations, and according to protocols that best ensures a proper record of the chain-of-custody and limiting sensitive operations to trusted persons.

The clause ends by requiring that variances are "consented by the RZ KSK Operator's management, are documented and communicated." I didn't read the fifth edition, so I can't say which edition first carried it, only that it arrived between February and November 2020.

The record since

IANA's schedule lists 63 held ceremonies from June 2010 to August 2026, counting Ceremony 53's two parts separately. In 2020 and 2021 there were two each, the only full years with fewer than four. Ceremonies 41 and 42 are the only ones that signed more than one quarter, and the 294 days between them is the longest gap in the schedule.

The public archive runs to 65 directories and 940 files, among them 315 video files from the ceremony cameras. Files from administrative ceremonies first appear in the directory for Ceremony 40, the safe repair, and turn up again in 15 later directories. I didn't read what those later administrative ceremonies were for. Everything here comes from IANA's and ICANN's own record. That is the point of reading it, and also its limit, because an outside auditor's view isn't quoted here.

What carries over

None of this is exotic. It is a procedure that expects to be interrupted, and a record that treats the interruption as a first-class entry. Anyone running software that acts where actions can't be taken back, a deployment pipeline or an agent holding production credentials, can copy four habits from it.

An exception there is a typed record: act, step, page, time, initials, and a description written by the witness rather than the actor. When the script doesn't cover a failure, the response in that room was to stop and write a new script, with a branch for each outcome, before taking the irreversible step. When steps happen out of order, the record says so instead of being tidied. And when a rule has to bend, its owner bends it in advance, in writing, and later turns the bend into a clause with limits.

At the back of the Ceremony 40 script is a page for the Internal Witness alone. It is two sentences long: "I hereby attest that the Key Ceremony was conducted in accordance to this script. Any exceptions that occurred were accurately and properly documented."

Sources

IANA, Key Signing Ceremonies, the schedule of held and planned ceremonies (as fetched 7 October 2026).
IANA, ceremony archive, 65 directories of scripts, logs and video.
IANA, Ceremony 40 administrative scripts, annotated scans: AC40_1, 10 February 2020 (lock assembly exception, page 7); AC40_2, 11 February 2020 (Act 2 and the discovery exception); AC40_3, 11 February 2020 (the Credentials Safe); AC40_4, 14 to 16 February 2020 (the drilling, script page 6, and the exception sheets).
IANA, Ceremony 40 script (preamble, closing act, Appendix C) and signer log, 16 February 2020.
Kim Davies, "Root Key Signing Key Ceremony Postponed," ICANN blog, 12 February 2020.
Kim Davies, "[Root KSK Ceremony 40 scheduled for 0000 UTC](https://lists.icann.org/hyperkitty/list/*Emails are not allowed*/message/Z7KNDGDKD5PNZJ52XHP5Y4W3GIYM23H3/)," root-dnssec-announce, 15 February 2020.
IANA, Questions and Answers on COVID-19 mitigations for KSK Ceremony 41, and the Ceremony 41 script, April 2020.
DNSSEC Practice Statement for the Root Zone KSK Operator: 4th edition, 1 October 2016, as published at iana.org/dnssec/icann-dps.txt in February 2020 (Wayback Machine capture, 16 February 2020), sections 4.2.2 and 6.6; 6th edition, 4 November 2020 and 8th edition, 14 April 2025, sections 4.2.2 and 4.5.4.1.
IANA, Trusted Community Representatives (as fetched 7 October 2026).

How this was checked. Typed quotations were checked word for word against the saved text of the source named beside them. The handwritten ones were transcribed from the scanned pages of the annotated scripts and checked against the images; the scans have no text layer. Times are UTC, as the scripts record them, and the "+1" day is read as explained in the text. The intervals (3 days, 22 hours and 34 minutes; 4 minutes and 15 seconds; the margin before 28 February) and the counts from the schedule and the archive listings were computed by script from those sources. Four exception sheets is the number read in the repair ceremony's packet, not a total for the month.

Originally published at vibeagentmaking.com. If you want a tamper-evident, dated record of what an automated system did and what went off-script: pip install chain-of-consciousness or npm install chain-of-consciousness, or try Hosted Chain of Consciousness.

You patched PHP. The old image is still in your cluster’s future.

Fri, 23 Oct 2026 17:56:15 +0000

title: "You patched PHP. The old image is still in your cluster's future."
published: false
description: "PHP2Shell (CVE-2026-109618, CVE-2026-109619) as seen from a Kubernetes cluster: how fast the scanners caught up, where they stay blind, and why rebuilding the image is only half of remediation."

tags: kubernetes, security, php, devops

On 21 October PHP shipped fixes for two critical bugs in the engine itself, CVE-2026-109618 and CVE-2026-109619, both CVSS 9.2, both reachable without authentication, both already being exploited by the time most people read the advisory. The community name is PHP2Shell. Fixed versions are 8.5.12, 8.4.27, 8.3.36 and 8.2.35. CISA and the UK NCSC both published "patch now" notices the same day.

I am not going to explain the bugs; Wiz, watchTowr and Qualys have done that well. I want to write about what happened in Kubernetes teams in the forty-eight hours after, because I watched it happen and because the second half of it is the part nobody schedules.

How most teams found out

From the news. Then someone opened a spreadsheet and typed "PHP" into the first column.

This is not a criticism. It is what a platform team does when the only inventory of what runs in production is the memory of the people who deployed it. The questions that followed were always the same three:

Which clusters run PHP at all?
Which images, which versions?
Who gets to say "we are done"?

The third question is the one that comes back in March, when an auditor asks what you knew on 22 October and what you did about it.

How fast the scanners caught up

I measured this on 23 October, two days after the advisory, with Trivy 0.72.0 and whatever vulnerability database it pulled that afternoon. You can repeat every step.

The database build was stamped 13:13 UTC on 23 October. Both CVEs were in it, with the Debian and Alpine advisories attached:

$ grep -a -o CVE-2026-109618 ~/.cache/trivy/db/trivy.db | wc -l
1247
$ grep -a -o CVE-2026-109619 ~/.cache/trivy/db/trivy.db | wc -l
1253

Debian's security tracker already listed php8.2 as resolved in 8.2.35-1~deb12u1 for bookworm and php8.4 in 8.4.27-1~deb13u1 for trixie. Alpine's secdb for 3.22 carried both CVEs. A debian:bookworm-slim image with php8.2-cli installed from the archive that day came out as 8.2.35, already fixed; a alpine:3.22 image with php84 came out as 8.4.27, also fixed.

So the honest answer to "did the open-source scanners keep up" is yes, within two days, for PHP installed as a distribution package. That is good news and it is the ordinary case for a lot of internal images.

Now the uncomfortable part.

$ trivy image --scanners vuln --severity CRITICAL,HIGH php:8.3-fpm

126 CRITICAL and HIGH findings. Not one of them about PHP.

The official php:* images build PHP from source. There is no php8.3 package in the image for a package scanner to look at, so Trivy, and every scanner that works from the package database, has nothing to match the advisory against. The 126 findings are the Debian base. The thing you actually care about is invisible.

This is not a Trivy bug. It is what package-based scanning is. But it means that if your PHP workloads run on the official image, or on any image that compiles PHP, then a quiet scanner is not a clean bill. It is silence.

What you can do about it this week, in order of effort:

Ask the binary. docker run --rm <your-image> php -v tells you the truth in one line. Do it for every PHP image tag you run, not the ones you think you run.
Rebuild on a base that was pushed after 21 October, pin the digest, and write the digest down somewhere an auditor can read.
If you can, move to distribution packages for the runtime. It costs you a newer PHP, it buys you a scanner that can see it.

Patching is half the job

Here is the part that gets skipped. You rebuild the image, you roll the Deployment, the pods come up on 8.3.36, everyone exhales.

The old image did not go anywhere. It is still in the registry. It is still in helm history. It is the image someone will roll back to at 03:00 when the new build has a problem that has nothing to do with PHP. It is in the CI cache, in the staging cluster nobody looks at, in the Job that runs once a month.

Remediation has two halves:

Run the fixed image.
Make it impossible for the vulnerable image to run again, and keep a record that proves both.

Most teams do the first half and call it done. The second half is what an admission controller is for.

What this looks like with an admission controller in the path

This is where I talk about what we built, so take the next paragraphs for what they are. Attestkeep is an admission controller for Kubernetes that scans, verifies signatures, decides at admission, and keeps the record. Here is exactly what it did, and did not do, for PHP2Shell.

The database refreshes itself. The scanner pulls a fresh vulnerability database every six hours. Trivy publishes one every six hours too, so the lag between an advisory landing upstream and the cluster knowing about it is measured in hours, not in someone's calendar.

Every admitted image is rescanned. Once a day by default (scan.rescanInterval in the chart, if you want it shorter), every image the cluster has ever admitted is scanned again against the current database. An image that was clean on Monday and carries a CVE on Thursday shows up as not compliant on Thursday, with the CVE named. You do not open a spreadsheet; the inventory is the admission ledger.

It tells you. A scan that finds critical vulnerabilities raises a critical_finding event, which goes to whichever channel you configured: email, Slack, Teams, a webhook, Jira, GitLab. When a running image flips from "would be admitted" to "would be denied", that is a second, separate message: "Running image now fails policy". That is the message an operator actually needs.

New pods with the image are refused. If your policy blocks critical findings, the vulnerable image cannot be admitted again. The rollback at 03:00 fails with a reason, the Job that runs monthly fails with a reason, the staging cluster with the same policy refuses it too. The old image is still in the registry. It is no longer in your cluster's future.

Running pods are not killed. This is deliberate. Attestkeep does not evict a workload because a database update changed its verdict; that is an operator's decision, with the blast radius in front of them. What it does is make sure the next decision, any next decision, goes the right way, and that the running population is reconciled against the ledger on a schedule so drift is visible.

If the exploit lands first, there is a second line. PHP2Shell ends in a shell. If Falco is connected, a shell spawned inside a container is a runtime event, and a runtime policy can turn that event into a notification, an issue, or stopping the pod. That is not prevention, it is containment, and it is worth having when the window between advisory and rebuild is days.

Every decision is signed. The admission ledger records the image digest, the policy revision that ran, the verdict and the reason. When the auditor asks in March what you knew on 22 October, the answer is a record with a signature, not a recollection.

And the blind spot above applies to us exactly as it applies to Trivy, because Trivy is the scanner inside. If you run the official php:* image, Attestkeep's rescan will not name CVE-2026-109618 either. It will show you the digest you admitted, when, and under which policy. The php -v step is still yours. I would rather tell you that here than have you find it out from a pentest report.

The checklist, tool-agnostic

For this week, whatever you run:

List every image whose name or Dockerfile mentions PHP. Not from memory; from the cluster.
For each, run php -v inside the image. Trust that over the scanner.
Rebuild on a post-21-October base, pin digests.
Add a rule that refuses the old digests at admission, so a rollback cannot resurrect them.
Write down what you found, what you changed, and when, in a form that survives until the audit.

If step 4 is a policy in an admission controller and step 5 is produced automatically, this is a Tuesday. If step 4 is a Slack message and step 5 is a Confluence page, it is a project, and it will be a project again at the next CVE.

Closing

The lesson of PHP2Shell for platform teams is not "patch faster". Everyone patched. The lesson is that the news is a bad way to learn what is in your cluster, and that a patched Deployment with the old image still admissible is not a remediated cluster.

Let the cluster do the checking. Keep the record. Make the old image a thing that cannot come back.

Sources: CISA alert, 21 October 2026 · NCSC advisory · watchTowr FAQ · Qualys · Google Cloud response. Scanner measurements are mine, 23 October 2026, Trivy 0.72.0; the commands are in the text.

Attestkeep runs inside the cluster it protects; every security feature is in the free edition. attestkeep.com · docs.attestkeep.com

SOC 2, CRA, NIS2: they all ask your cluster the same five questions

Fri, 23 Oct 2026 17:49:26 +0000

title: "SOC 2, CRA, NIS2: they all ask your cluster the same five questions"
published: true
description: "A working engineer's read of SOC 2 and the EU instruments that landed alongside it, and why the hard part is not the controls but the evidence that they ran."

tags: security, kubernetes, devops, compliance

In eleven days, on 11 September 2026, the reporting obligations of the EU Cyber Resilience Act start applying to anyone who puts a product with digital elements on the European market. Not the full regulation. Just the part where, if you find out an actively exploited vulnerability is in your product, you have 24 hours to tell ENISA about it.

I have watched a lot of engineering teams meet this class of deadline for the first time. It usually goes the same way. Somebody in sales gets a security questionnaire. Somebody in engineering gets forwarded the questionnaire. Three weeks later there is a shared folder called evidence-final-v3 with 200 screenshots in it, and nobody can tell you which screenshot answers which question.

I have spent the last several months building a tool whose entire job is that folder, so I read the instruments properly. This is what I found out. It is written for engineers, not for a compliance team, and I try to be specific about what the text says rather than what a vendor blog says it says.

Where SOC 2 came from, and why that still shapes it

SOC 2 exists because of a misuse.

In 1992 the AICPA published SAS 70, an auditing standard for service organisations. Its purpose was narrow: if you outsourced your payroll, your auditor needed some assurance that your payroll provider's internal controls did not corrupt your financial statements. It was an accounting instrument, for accountants, about financial reporting.

Then the industry outsourced everything else. By the mid-2000s companies were sending their customer data to service providers, and they wanted assurance about that, not about financial reporting. There was nothing designed for it, so they asked for the thing that existed. Vendors started waving SAS 70 reports around as proof they were secure. They were not proof of that. SAS 70 had no defined control set at all: the service organisation wrote its own control objectives, and the auditor tested against whatever had been written. Two SAS 70 reports could have nothing in common.

In 2010 the AICPA split the thing in half. Financial reporting controls stayed, renamed SOC 1 under SSAE 16 (SSAE 18 since 2017). The security question got its own report: SOC 2, with an actual defined set of criteria behind it.

That history is worth carrying, because SOC 2 still has SAS 70's shape in one important respect. It defines criteria, not controls. It tells you what has to be true. It does not tell you how. More on that in a moment, because it is the single most common misunderstanding.

Type I and Type II

Two report types, and the difference is most of the value.

Type I is a point in time. An auditor looks at your controls on one date and gives an opinion on whether they are suitably designed. It answers: do you have the right things in place?

Type II covers a period, typically 3 to 12 months, with 6 being the usual choice for a first report. The auditor gives an opinion on whether the controls were suitably designed and operated effectively throughout that period.

A Type I says you built the fence. A Type II says the fence was standing every day for six months.

Almost every buyer who asks for SOC 2 means Type II, or will mean it at renewal. And it matters enormously to how you should think about evidence, because "operated effectively throughout the period" is not a thing you can produce at the end. The auditor samples across the whole window. If your quarterly access review was supposed to run four times and it ran once, in the week before fieldwork, that is an exception in your report and everybody reading it can see it.

That property, that the evidence must have existed continuously rather than being assembled afterwards, is the one I would burn into a wall. It is what makes screenshot folders a bad answer.

The five criteria, and the four you probably do not need

SOC 2 is built on the Trust Services Criteria (AICPA TSP section 100, 2017, revised 2022). Five categories:

Security — the Common Criteria, CC1 to CC9. Mandatory. Every SOC 2 includes it.
Availability — is the system up when you said it would be
Confidentiality — is information designated confidential protected
Processing Integrity — is processing complete, valid, accurate, timely, authorised
Privacy — personal information handled per your notice

Only Security is required. You choose the rest, and you should choose as few as you can defend, because each one is more scope, more testing, more cost. Most infrastructure and SaaS companies do Security, or Security plus Availability. Add Confidentiality if your contracts talk about it. Processing Integrity is genuinely rare and mostly belongs to payments and payroll.

Inside the Common Criteria, the ones an engineer actually touches:

Criterion	Roughly
CC1–CC3	Governance, communication, risk assessment. Mostly organisational.
CC4	Monitoring: do you check your own controls are present and functioning?
CC5	Control activities.
CC6	Logical and physical access. Who can reach what, and how you keep the outside out.
CC7	System operations. Detecting vulnerabilities and configuration drift, spotting anomalies, handling incidents.
CC8	Change management. Nothing untested or unauthorised reaches production.
CC9	Risk mitigation, including vendor and third-party risk.

If you run containers on Kubernetes, most of your technical evidence lands in CC6, CC7 and CC8. CC6.8 is the one people miss: "the entity implements controls to prevent or detect and act upon the introduction of unauthorized or malicious software." That is your image supply chain, stated in 2017 language.

SOC 2 does not tell you what to do

This is the part that catches teams out.

There is no SOC 2 control list. There is no "you must scan images" clause. The criteria are outcome statements, and you define the controls that meet them, and then the auditor tests the controls you defined. You write a control saying "all container images are scanned before deployment and images with critical vulnerabilities are blocked", and now you are on the hook to prove that happened every day of the period.

This cuts both ways. The freedom is real: you are not forced into somebody else's architecture. The trap is also real, and it is that a weak control, written vaguely, passes the audit while protecting nothing. "We use industry standard scanning tools" is a sentence that can be true while your cluster runs whatever anybody pushes.

The auditor is not there to tell you the control is weak. They are there to test whether it operated as described.

The other instruments, and their clocks

SOC 2 is voluntary and market-driven; you do it because a customer asked. Most of what arrived after it is not voluntary. Here is the landscape as it stands at the end of August 2026.

EU Cyber Resilience Act — Regulation (EU) 2024/2847. Applies to products with digital elements sold in the EU, which for most of us means the software itself. Three dates: conformity assessment bodies from 11 June 2026 (passed), vulnerability and incident reporting from 11 September 2026 (eleven days away), and the substantive obligations, including technical documentation and the SBOM requirement, from 11 December 2027. Annex I Part II point 1 is the SBOM clause: components documented in a commonly used machine-readable format, covering at least the top-level dependencies. You do not have to publish it. You do have to have it and hand it over when a market surveillance authority asks.

NIS2 — Directive (EU) 2022/2555. Applies to entities in essential and important sectors, and reaches software vendors indirectly, forcefully, through Article 21(2), which names supply chain security and requires covered entities to consider the security practices of their direct suppliers. That is why NIS2-derived questionnaires arrive at companies that are not themselves in scope. Being a supplier to somebody in scope is enough.

DORA — Regulation (EU) 2022/2554, in force since 17 January 2025 for EU financial entities and the ICT providers serving them. Chapter V Article 28 is the third-party risk chapter, and it has teeth in a way NIS2 does not, including audit rights written into contracts.

ISO/IEC 27001:2022 — certification rather than a report. Buyers outside the US often prefer it. Annex A control 8.8 (technical vulnerabilities), 8.25 through 8.31 (secure development, separation of environments) are the ones your pipeline touches.

NIST — SP 800-190 for container security specifically, SP 800-218 (SSDF) for secure development, SP 800-53 Rev. 5 for the full federal control catalogue. Not law unless you sell to the US government, but the language is precise and much of the rest of the world quotes it.

GDPR Article 32 — security of processing. Short, general, and still the clause that turns a container escape into a regulatory event if personal data was on the other side of it.

They are all asking the same five questions

Here is the thing I did not expect when I started reading these side by side. The instruments are written by different bodies, in different decades, for different legal systems, and underneath they converge hard.

CRA Annex I Part I (3)(g), on availability and resilience of essential functions, and SOC 2 A1.2 both turn on outages and what the system does when a component fails. CRA (3)(c) on access control and CC6.1 both turn on hardening and federated access. CRA (3)(k) on recording and monitoring activity and CC7.2 both turn on the audit trail. DORA Article 9's protection and prevention requirements land in the same place as CC6.8.

For a workload running in a cluster, strip away the citations and every one of them is asking:

What is running? An inventory. Not "what did we deploy", what is actually admitted right now.
Where did it come from? Provenance. A signature, an identifiable registry, a build you can point at.
What is wrong with it? Known vulnerabilities, against a database that was current at the time of the check.
Who allowed the exceptions? Because there are always exceptions. An unrecorded exception is indistinguishable from a control failure.
Can you prove all of the above for a period you did not choose in advance?

Question 5 is the one tools get wrong.

What good evidence looks like

Most compliance tooling is built to produce a dashboard. A dashboard tells you the state now. An auditor is not asking about now, they are asking about February.

Working backwards from what a Type II sampling exercise actually needs, evidence has to carry five properties:

A period, stated on the document. Not "generated today", but "covers 1 February to 31 July, and here is what happened in that window."

Provenance of the evidence itself. Who generated it, from which system, at what version. A compliance mapping is an interpretation, and interpretations change as you read a framework more carefully. Evidence that does not record which reading it used cannot be re-examined later.

Integrity. A PDF is a suggestion. Anybody can edit a PDF. If the document that goes to the auditor is not cryptographically bound to the bytes that were generated, then the strongest statement you can make about it is "someone in our company typed this."

Citation, not paraphrase. Every claim should name the clause it answers, quoted close to the instrument's own words. Auditors read the criterion, not your marketing vocabulary. If your report says "we are compliant with CC7.1" without restating what CC7.1 asks, you have made the reader do a lookup, and made yourself impossible to check.

Honest gaps. This is the one I care most about. A container security tool can evidence maybe a third of NIS2 Article 21. The rest is staff training, disclosure policies, reporting to a national CSIRT, business continuity. A report that quietly maps a feature list onto the other two thirds is worse than no report at all, because it tells an organisation a duty is discharged when nobody has done it. Anything that comes back all green is lying to you, and the lie is expensive precisely when it matters.

Why the admission controller is the right place to stand

For the Kubernetes-specific part of this: scanning in CI is necessary and insufficient.

CI scanning proves you scanned an image. It does not prove that only scanned images ran. Those are different claims and the second is the one CC6.8 asks for. Between the pipeline and the cluster there is a gap wide enough to drive a kubectl apply through: a manually applied manifest, a Helm chart from a vendor, a sidecar injected by a mesh, a latest tag that resolved to something different this morning than it did last week.

The admission webhook is the last point where the answer to "may this run" is still no. It is also the only point where a record of the decision is a record of what actually entered the cluster, rather than a record of intent. Every admission review is a dated, attributable event: this digest, this policy, admitted or refused, at this moment. That is the raw material a Type II sample is made of, and it accumulates whether or not anybody remembers to collect it.

That property, evidence that exists because the control ran and not because somebody prepared for an audit, is the whole point.

How we built this into Attestkeep

Attestkeep is a Kubernetes admission controller that scans images, verifies signatures and attestations, and enforces policy at admission. That part is not novel and I would not write a post about it. The part I want to describe is what happens after six months of it running.

You pick a period and a framework. It generates a report against eight of them: SOC 2, EU CRA + NIS2, DORA, NIST SP 800-190, SSDF, SP 800-53 Rev. 5, ISO 27001, and GDPR Article 32. Every requirement in the report carries its citation, the obligation restated close to the instrument's own words, a status, and a note written in the period's own numbers.

There are four statuses, and the fourth is the reason I am writing this section:

met — the control operates and the records show it operating
partial — it operates but something narrows what it proves: a permissive failure policy, audit-only mode, an exemption
gap — squarely within what this thing can enforce, and the records say it is not being enforced
outside_scope — a duty of the organisation, not a property of a cluster. Named rather than omitted, so the reader can see where the document's coverage ends.

Here is an unedited fragment from a real 30-day report generated against a demo cluster:

{
  "id": "CC4.1",
  "citation": "CC4.1",
  "title": "Evaluations to ascertain that controls are present and functioning",
  "asks": "The entity selects, develops, and performs ongoing and/or separate evaluations to ascertain whether the components of internal control are present and functioning.",
  "status": "met",
  "note": "the control's operation is measured continuously rather than sampled: 293 reviews, 31 admitted, 262 refused, and 3 windows in which it was not operating",
  "evidence": ["admission", "availability"]
}

and, from the same report, one that did not pass:

{
  "id": "CC6.6",
  "citation": "CC6.6",
  "status": "gap",
  "note": "no registry allow-list is in force, so an image may be pulled from any registry the node can reach",
  "evidence": ["control_statement.allowed_registries"]
}

The summary for that period was 3 met, 6 partial, 2 gap, 2 outside scope. That is a demo cluster with a loose policy, and the report says so. I would rather ship a tool that tells you that in August than one that tells you everything is fine and lets an auditor find it in February.

The evidence field is a pointer into the sections of the same document that carry the underlying numbers, so a reader can check the claim rather than accept the note.

Integrity. The report is signed with Ed25519 into a DSSE envelope carrying an in-toto Statement v1. The signature covers report.json and nothing else. The subject is the report's SHA-256, so the signature binds to the exact bytes, and any copy of those bytes can be checked against it.

What the auditor receives is a zip:

README.txt          what is signed, and three ways to check it
report.json         the evidence. these exact bytes are what is signed
attestation.json    DSSE envelope, in-toto Statement v1
certificate.jwt     the public half of the signing key, signed by us
<id>.pdf            a readable rendering. convenience only, carries no signature
<id>.attestkeep.json  all of the above in one file, for single-file verifiers

The README is always in English, whatever language the console is set to, because an auditor's file gets read in places you cannot predict. It states plainly that if the PDF and report.json ever disagree, report.json is the record and the PDF is wrong. Verification works three ways: a browser page that runs entirely client-side and uploads nothing, a Python script with no dependencies, or cosign if you already have it. The archive is byte-for-byte reproducible: the zip entry timestamps come from the report's own generation time rather than the clock, so two people downloading the same package get the same file.

And the scope statement, which is printed at the top of every SOC 2 report we produce:

It is evidence for an audit, not an audit: only a service auditor issues an opinion, and only the entity defines its own system boundary.

I put that there because I have seen what the alternative does to people. A tool cannot make you compliant. It can make the day the auditor asks a boring one.

What this does not do

Being straight about the limits, since I just spent 400 words demanding honest gaps from everybody else.

The compliance mappings have not been through legal or audit review. They are my careful reading of published texts, with citations included specifically so a professional can check my work rather than trust it. The NIST SP 800-190 citations were corrected once already, when I went back and compared them against the published guidance instead of my notes.

Nothing here binds a report to a trusted clock. generated_at is an assertion by the signer, not proof. If you need trusted timestamping, you need it from somewhere else, and the predicate says so in a comment rather than hiding it.

And the honest scope line applies to me too: one cluster's admission control answers part of SOC 2 and a smaller part of NIS2. It does not answer your access reviews, your onboarding, your incident response drills, or your vendor register.

If you take one thing

Whatever you use, and it does not have to be mine, make the evidence a byproduct of the control operating rather than a project you run before an audit. The frameworks converge on this even where their words differ. SOC 2 Type II wants the whole period. CRA wants technical documentation that was maintained, not assembled. DORA wants an audit right that is meaningful when exercised without notice.

Update (September 2026). Vinh Nguyen pointed out in the comments that questions 1 and 5 come apart exactly where this article admits they might: the admission ledger records honestly when the control was not operating, but nothing in it says what crossed during those windows — or what predates the install, or what walked in under failurePolicy: Ignore. Whatever did is still running long after the window closed, and a ledger of boundary crossings has no row for it.

He was right, so the control grew a second half. A periodic sweep now reads every running container's observed imageID — not the spec's image reference, which under a mutable tag still matches its original admission event while the bytes underneath have changed — and reconciles each digest against the admission ledger. A running digest with no allowed admission event behind it lands in the evidence package as its own population section, right next to the availability section it completes: the document that admitted the two questions come apart now stitches them back together. Policy decides what a finding does — recorded, announced, or the pod is stopped so its controller recreates it through the webhook, onto the ledger this time. The details, including the honest edges (out-of-scope namespaces, side-loaded images with no registry digest, truncated lists that say so), are in the comment thread below.

Evidence you can only produce by preparing for the question is not evidence that the control was working. It is evidence that you can prepare.

I build Attestkeep, a Kubernetes admission controller with compliance evidence built in. It is early. If you have been through a Type II with containers in scope and something above is wrong, I would genuinely like to hear it.

Pub/Sub Isn't Event-Driven Architecture: Here's What Most Teams Miss

Fri, 23 Oct 2026 14:38:21 +0000

Pub/Sub is a messaging pattern. Event-Driven Architecture is an architectural discipline. Every EDA implementation may use Pub/Sub, but not every Pub/Sub implementation is truly Event-Driven Architecture.

For many years, I would have agreed with the statement that Pub/Sub equals Event-Driven Architecture (EDA).

Why? Because early discussions around EDA focused primarily on three characteristics:

Real-time communication
Publish/Subscribe messaging
Asynchronous processing

At that time, implementing a Pub/Sub broker was often considered synonymous with implementing EDA.

Today, however, the answer is no.

As organizations have adopted EDA at scale, the discipline has evolved from a messaging concept into a comprehensive architectural practice. Modern EDA encompasses far more than simply publishing and subscribing to messages.

📡EDA Is More Than Messaging

A mature Event-Driven Architecture consists of multiple foundational elements:
📝Event Definition

Events must be clearly defined and categorized.

Examples include:
Business Events: CustomerRegistered, OrderPlaced, PaymentCompleted
System Events: FileUploaded, ServiceStarted, DatabaseUpdated, EOD

Without a consistent event model, organizations quickly end up with thousands of loosely defined messages that are difficult to understand and maintain.

👤Event Ownership

Just like data, events are business assets.

Every event should have a clearly identified owner responsible for:
Event definition
Schema management
Versioning
Lifecycle governance

When nobody owns an event, everyone assumes somebody else does.
📏Event and Queue Standards

Topic (event) and queue naming conventions form the foundation of a scalable EDA platform.
A well-defined naming structure enables:
Event discoverability
Consistent routing
Easier troubleshooting
Better governance
Without standards, the event landscape quickly becomes chaotic.

🔀Event Flow Design

EDA is not merely about sending messages.

It is about how events collaborate to achieve business outcomes.

Modern architectures require a clear understanding of:
Event relationships
Event choreography
Event dependencies
Business process visibility

While services remain loosely coupled, the overall event flow must still be intentionally designed.

🛡️Event Governance

Governance is the difference between a sustainable platform and an uncontrolled messaging ecosystem.

Effective governance covers:
Schema management
Data classification
Security controls
Access management
Event lifecycle management
Compliance and auditing

Governance should be embedded throughout the entire lifecycle, from design through implementation and operations.

📬Pub/Sub Is Only One Pattern Within EDA

Pub/Sub is an important communication pattern, but it is only one building block of a broader architecture.
A Pub/Sub platform without governance, ownership, standards, and event design is simply a collection of disconnected messages.

In such environments, organizations commonly encounter problems such as:
⚠️Consumer Failures

Consumers may crash when:
Unexpected event types are received
Event schemas change without coordination
Required fields are removed or modified
Data Exposure Risks

Sensitive information, including Personally Identifiable Information (PII), may be unintentionally delivered to systems or teams that should not have access to it.
Message Loss or Misrouting

Poorly governed subscriptions can result in:
Events being consumed by unintended systems
Important messages being ignored
Business processes becoming unreliable
Rising Infrastructure Costs

Unnecessary event distribution increases:
Network traffic
Broker workload
Infrastructure consumption
Operational costs
What begins as architectural flexibility can quickly become architectural sprawl.

💡 Final Thoughts

Many organizations begin their event-driven journey by deploying a message broker and enabling Pub/Sub communication. While this is an important first step, it is only the foundation.
The real challenge is not moving events between systems. The challenge is managing events as long-lived business assets.

Without clear ownership, governance, standards, and lifecycle management, organizations often end up with an event ecosystem that is difficult to understand, secure, and evolve. Teams gain asynchronous communication, but lose visibility, consistency, and control.

Successful Event-Driven Architecture is not measured by how many events are produced each day. It is measured by how effectively those events enable business capabilities while remaining discoverable, governed, and trusted across the enterprise.

In other words:
A message broker can make your systems event-enabled.
Governance, ownership, and standards make your organization truly event-driven.

❓ What do you think? Have you seen organizations mistake Pub/Sub implementation for Event-Driven Architecture? What governance challenges emerged as event adoption grew?

This typically drives significantly more comments and discussion from architects, developers, and integration professionals.

Don’t Let the LLM Do Everything: Designing a Hybrid AI SRE With Deterministic Controls

Fri, 23 Oct 2026 09:16:16 +0000

An alert fires because a checkout service has started returning HTTP 500 errors.

An AI SRE begins investigating. It needs to determine which service is failing, which version is deployed, whether anything changed recently, what the logs and traces show, what probably caused the failure, and whether to restart the service or roll it back.

It is tempting to give all of that work to an LLM.

That is also where the architecture starts going wrong.

Some parts of incident investigation genuinely require reasoning. An LLM can help correlate a deployment with a latency spike, interpret an unfamiliar exception, or compare two competing root-cause hypotheses.

The Problem With an LLM-First AI SRE

The simplest AI SRE architecture looks appealing:

Alert
  |
  v
 LLM --------> Metrics
  | ----------> Logs
  | ----------> Traces
  | ----------> Kubernetes
  | ----------> Git
  | ----------> Incident history
  v
Diagnosis
  |
  v
Action

The model receives an alert, decides which tools to call, interprets the results, produces a diagnosis, and possibly chooses a remediation.

This can work impressively in a controlled demo.

Production incidents are less forgiving.

Consider a model attempting to query a metric called:

checkout_connection_pool_active

The real metric is:

checkout_db_pool_active_connections

The query returns nothing.

The LLM now has at least two possible interpretations. The metric does not exist, or the query was wrong.

If there is no deterministic validation between the model and the observability system, the agent may confidently choose the first explanation and continue its investigation using a false assumption.

This is not theoretical as an architectural failure mode. Recent analysis of multi-agent systems for infrastructure observability describes how an agent can issue an incorrect metric query, receive an empty response, and interpret that response as evidence that the metric itself does not exist.

The larger problem is that an LLM can blur three very different things:

A fact retrieved from a system, an inference made from evidence, and an assumption created because evidence is missing.

For incident response, those distinctions matter.

An AI SRE should not infer a deployment SHA when the deployment platform already knows it. It should not decide whether a production restart is authorized when an access-control policy can answer the question exactly. It should not assume a query is valid when its schema can be checked before execution.

Better prompting can reduce some mistakes. Architecture can eliminate entire classes of them.

Divide Incident Response Into Deterministic and Probabilistic Work

A useful design exercise is to take every task performed during an incident and ask one question:

Could this operation be reliably expressed as a lookup, schema, policy, threshold, state machine, or executable test?

If the answer is yes, the LLM probably should not be the primary authority for that operation.

Consider a typical production investigation:

Incident task	Best owner	Reason
Identify service owner	Deterministic	Service catalog already contains the answer
Retrieve deployed commit	Deterministic	Deployment system contains the exact state
Check SLO threshold	Deterministic	Mathematical evaluation
Validate log or metric query	Deterministic	Schema can be checked
Retrieve logs and traces	Tooling	Execution does not require reasoning
Correlate unusual signals	LLM	Requires contextual interpretation
Interpret an unfamiliar exception	LLM	Semantic reasoning is useful
Generate competing root-cause hypotheses	LLM	Requires inference from incomplete evidence
Verify infrastructure permissions	Deterministic	Policy decision
Check whether evidence contradicts a hypothesis	Hybrid	Reasoning plus explicit verification
Execute production remediation	Deterministic or human-gated	High-consequence operation

This division does not make the AI SRE less capable. It gives the model a smaller and better-defined job.

The model should reason about what operational facts mean. It should not be responsible for inventing those facts.

A Hybrid Architecture for AI SRE

A safer architecture puts deterministic controls both before and after LLM reasoning.

                 +-------------------+
                 |       Alert       |
                 +---------+---------+
                           |
                           v
            +-----------------------------+
            | Deterministic Enrichment   |
            | Service / Owner            |
            | Environment                |
            | Deployment SHA             |
            | Recent Changes             |
            | SLO State                  |
            +-------------+---------------+
                          |
                          v
            +-----------------------------+
            | Evidence Retrieval          |
            | Logs, Metrics, Traces       |
            | Events, Runtime State       |
            +-------------+---------------+
                          |
                          v
                 +-------------------+
                 |   LLM Reasoning   |
                 | Correlate         |
                 | Hypothesize       |
                 | Explain           |
                 +---------+---------+
                           |
                           v
            +-----------------------------+
            | Deterministic Verification  |
            | Query validation            |
            | Evidence checks             |
            | Policy evaluation           |
            | Permission boundaries       |
            +-------------+---------------+
                          |
                          v
                 Human / Action Gate

The first deterministic layer enriches the incident with facts that should not require interpretation.

Instead of prompting an LLM to work out which deployment caused the problem, the system can provide:

{
  "service": "checkout-api",
  "environment": "production",
  "deployment_sha": "a83f9c2",
  "deployment_time": "14:03:17Z",
  "incident_start": "14:07:42Z",
  "owner": "checkout-platform"
}

The evidence layer then retrieves the logs, metrics, and traces relevant to that incident.

Only then does the LLM perform the work it is actually good at: correlating signals, interpreting unstructured information, finding relationships, and generating hypotheses.

The final layer is equally important.

The LLM produces a hypothesis. It should not be allowed to turn its own hypothesis into a verified fact.

That requires a verification step outside the reasoning loop.

A Worked Example: The Database Is Slow, but Is It the Root Cause?

Consider an API that begins returning intermittent 500 errors four minutes after a deployment.

The initial telemetry looks suspicious:

14:03 Deployment a83f9c2 completed
14:07 HTTP 500 rate begins increasing
14:08 Database latency increases
14:09 Checkout error rate reaches 7.8%
14:10 CPU remains normal

A model looking at this evidence might generate a reasonable hypothesis:

Increased database latency indicates connection pool exhaustion, causing checkout requests to fail.

The explanation sounds plausible. That does not make it true.

Start With Facts

The deterministic enrichment layer establishes the production context:

Service: checkout-api
Environment: production
Version: a83f9c2
Deployment time: 14:03 UTC
Incident start: 14:07 UTC
Owner: Checkout Platform

None of these values needs to pass through probabilistic reasoning.

Retrieve the Available Evidence

The investigation layer retrieves failed request traces, matching application errors, database pool metrics, the deployment diff, and recent infrastructure events.

The LLM now sees three important facts.

The failures started shortly after the application deployment. Almost every failed request executes the same validation path. Database latency increased at roughly the same time.

That produces at least two viable hypotheses:

H1: Database connection pool exhaustion is causing requests to fail.
H2: The new application code is allowing unexpected request state to reach the database layer.

The important step is what happens next.

The system should not ask the model to choose whichever explanation sounds more likely. It should determine what evidence would falsify or support each hypothesis.

For H1, that might mean checking active and maximum connection counts.

For H2, the missing piece might be the value of a request field at a specific line in the validation path.

When the Evidence Does Not Exist Yet

This is where incident investigation sometimes hits the limit of conventional observability.

Logs only contain values the application was instructed to record. Traces record instrumented operations and context. Metrics aggregate measurements chosen in advance.

If the variable needed to confirm a hypothesis was never captured, the LLM cannot recover it by reasoning harder.

The correct response to missing evidence is to obtain better evidence.

One approach is bounded runtime instrumentation. For example, a production debugging system such as HyperProbe can place a temporary read-only probe at a specific line and capture runtime state when live traffic reaches it. The important architectural property here is not the product itself. It is that the investigation gains new evidence without allowing the reasoning model to modify application state.

Suppose the capture returns:

customer.region = null
connectionPool.active = 34
connectionPool.max = 100

The investigation has changed materially.

The database pool is nowhere near exhaustion. The first hypothesis can be rejected.

The unexpected null value, combined with the deployment diff and failing code path, now provides evidence for the second hypothesis.

This is a better incident-response pattern than asking an LLM to repeatedly reinterpret the same incomplete telemetry.

It also highlights an important distinction between AI reasoning and observability: an agent can only reason from evidence that exists.

Put Deterministic Controls Around Tool Calls

The same principle should apply whenever an AI SRE interacts with production systems.

An LLM can decide that querying a particular metric would help the investigation. It does not follow that the raw model output should be executed directly.

A validation layer can first check whether the requested tool exists, required parameters are present, the metric or field is valid, the requested time range is permitted, and the caller has access to the target environment.

This becomes more important as agents gain access to more systems. DZone's coverage of tool-call observability makes the related point that teams need visibility into which tools an agent invoked, what parameters were provided, whether those calls succeeded, and whether access controls were respected.

For an AI SRE, I would go one step further.

Tool calls should not only be observable after execution. Where possible, they should be validated before execution.

Suppose the model generates:

{
  "tool": "query_prometheus",
  "metric": "checkout_connection_pool_active",
  "range": "15m"
}

A deterministic metric registry can reject the nonexistent metric before the query reaches Prometheus.

The LLM receives:

{
  "status": "invalid_metric",
  "suggested_matches": [
    "checkout_db_pool_active_connections"
  ]
}

Now the model is reasoning from a known validation failure rather than an ambiguous empty result.

Separate Diagnosis From Authority

Production access is another place where prompting is a poor substitute for deterministic controls.

Telling an agent, “Never restart production unless absolutely necessary,” is not an authorization system.

A better design divides actions by consequence.

Reading telemetry may be fully automatic. Capturing bounded diagnostic evidence may be permitted under policy. Restarting a staging workload might be automatic. Restarting a production service could require approval. Database mutations might always remain behind an explicit human gate.

The actual thresholds will vary by organization. The architectural principle does not.

The model can recommend an action.

A separate system decides whether that action is authorized.

This is where human-in-the-loop approval gates become more useful than placing a human behind every single AI operation. Human review should concentrate on consequential or ambiguous actions rather than routine evidence gathering.

An investigation agent that requires approval every time it reads a log will create new toil. An agent that can restart a stateful database because the prompt described it as “safe” has the opposite problem.

The LLM Still Has an Important Job

A hybrid architecture is not an argument against using LLMs in SRE.

Quite the opposite.

Incidents routinely produce large amounts of heterogeneous and partially structured evidence. A human might need to move between logs, dashboards, traces, deployment histories, runbooks, source code, and previous incidents before forming a plausible explanation.

LLMs can be effective at synthesizing that context.

They can recognize relationships across signals, interpret unfamiliar errors, search previous incident reports, generate competing hypotheses, identify what evidence is still missing, and explain a complex investigation to the on-call engineer.

These are probabilistic tasks.

The distinction can be summarized simply:

Deterministic systems are good at answering what is true when the truth is encoded somewhere. LLMs are useful for reasoning about what those facts might mean.

An effective AI SRE needs both.

Even agentic SRE architectures that allow systems to observe, reason, and act still depend on strong guardrails and observability around that reasoning.

The goal should not be maximum autonomy. The goal should be useful autonomy inside explicit operational boundaries.

Verify the Architecture With Operational Metrics

A hybrid design should also be evaluated differently from a chatbot.

Response quality alone is not sufficient.

An SRE team should be able to measure whether an agent generated invalid tool calls, retrieved evidence relevant to the actual failure, made claims unsupported by telemetry, attempted actions beyond its permissions, or required unnecessary calls before reaching a useful hypothesis.

For example:

Metric	What it tells you
Invalid tool-call rate	Whether the agent understands its operational interfaces
Evidence precision	Whether retrieved signals actually support the investigation
Unsupported-claim rate	How often conclusions outrun available evidence
Time to confirmed hypothesis	Whether the system reduces investigation time
Tool calls per incident	How efficiently it investigates
Unsafe action attempts	Whether policy boundaries are working
Root-cause accuracy	Whether the final diagnosis matches known outcomes

This is also why end-to-end observability for agents and workflows matters. If an AI SRE reaches the wrong conclusion, engineers need to reconstruct the investigation: which context was provided, which tools were called, what each tool returned, what hypothesis was generated, and why the workflow proceeded to the next step.

Without that trace, debugging the AI SRE itself becomes another production incident.

Use AI for Uncertainty and Software for Certainty

The interesting question in AI SRE is not how many operational tasks can be handed to an LLM.

It is where probabilistic reasoning creates enough value to justify probabilistic behavior.

Service ownership, deployed versions, access policies, thresholds, schemas, and permissions already have deterministic representations. Keep them deterministic.

Use the LLM where SRE work genuinely requires interpretation: correlating signals, understanding unstructured evidence, generating hypotheses, comparing explanations, and deciding what evidence to gather next.

Then verify its conclusions outside the model.

And when the available telemetry cannot confirm a diagnosis, do not ask the LLM to become more confident. Gather the missing evidence.

That architecture may look less autonomous than an LLM connected directly to every production tool.

In practice, it is much closer to what reliability engineering has always been about: allowing flexible reasoning where it helps while making critical behavior predictable where it matters.

Can AI SRE Really Reduce MTTR? What It Can and Cannot Automate

Fri, 23 Oct 2026 07:58:29 +0000

Most SRE teams do not lose the bulk of an incident because nobody noticed something was broken. In mature environments, alerting is already fast enough to wake the right people, interrupt the right channel, and open the right incident record. The harder question starts after the page: what changed, which dependency is involved, how wide is the blast radius, which dashboard matters, what did we try last time, and what action is safe enough to take right now?

The practical case for AI SRE is narrower than “replace the on-call engineer,” and more useful: reduce the time between alert and evidence-backed hypothesis. Good incident response depends on actionable alerts, prepared on-call teams, up-to-date playbooks, clear coordination, and automation that removes repetitive work so responders can focus on judgment.

So, can AI SRE reduce MTTR? Yes, mostly by compressing investigation, triage, handoffs, runbook selection, and documentation. In practice, the largest MTTR gains usually come from shortening two intervals: alert-to-hypothesis during investigation and mitigation-to-postmortem during recovery review. AI can reduce decision-making drag. It should not be treated as a blanket license for autonomous production changes.

Where incident-response time actually goes

MTTR is often discussed as one number, but responders experience it as a sequence of smaller clocks. Time to detect, time to acknowledge, time to orient, time to investigate, time to mitigate, time to verify, time to communicate, time to resolve, and time to document. Monitoring and paging tools have already improved the first two. The middle of the incident is where time still disappears.

That middle stage is messy because it is not one task. It is a search problem under pressure. Engineers move between dashboards, logs, traces, deploy history, feature flags, ownership maps, recent pull requests, dependency graphs, previous incidents, runbooks, customer reports, and tribal memory. Some of this work is mechanical. Some of it is judgment. The expensive part is that both happen at the same time.

A useful operating metric is the time from alert to first useful hypothesis. Not the first guess. Not the loudest theory in the incident channel. A useful hypothesis is specific, falsifiable, and supported by evidence: “The checkout latency spike started three minutes after deployment x; error volume is concentrated in service y; downstream dependency z is healthy; rollback is likely safe.”

A second clock starts after mitigation. Once the service is stable, teams still have to reconstruct the incident from chat threads, dashboards, timestamps, and memory. If that work takes days, the team loses context and ships weaker corrective actions. AI can preserve the evidence while it is still fresh, making the postmortem less dependent on whoever happened to take notes under pressure.

This is where AI SRE reduces MTTR. It gathers context that humans would otherwise collect manually, ranks likely explanations, and keeps a traceable record of what it found. During the first minutes of an incident, responders usually need the same pieces of context: the relevant runbook, recent changes, logs from the failing path, ownership details, downstream impact, and whether the same pattern has appeared before. AI is useful when it surfaces that context before the incident channel fills with guesses.

What AI SREs can automate today

The near-term value of AI incident response automation is strongest when the work is read-heavy, repetitive, and evidence-based. Claims weaken when vendors imply that any incident can be diagnosed and fixed without human review.

Alert triage

AI can help with alert triage by grouping related alerts, suppressing obvious duplicates, identifying impacted services, suggesting ownership, and connecting a new incident to past incidents. It can also summarize severity signals for the incident commander: which regions are affected, which customer-facing paths are degraded, whether error budgets are burning, and whether the pattern resembles a known failure mode.

The caveat is simple: AI does not rescue bad alert hygiene. If alerts are noisy, cause-based rather than symptom-based, or detached from user impact, the AI has to reason from weak inputs. That usually produces faster confusion, not better triage.

Log and metric investigation

AI is well suited to pulling together logs, metrics, traces, deploy events, and topology data. A human can ask a focused question, and the system can retrieve the relevant slices without requiring the engineer to remember every query language or dashboard path at 3 a.m.

Summarization is not proof. Correlation is not causation. The output has to remain inspectable: links to raw logs, metric windows, traces, deploy diffs, and the reasoning path. If the AI cannot show why a signal matters, the responder still has to redo the investigation manually.

Root-cause analysis

Root-cause analysis automation is most credible when it produces ranked hypotheses, rather than a single magical answer. A good AI SRE should say what it thinks happened, what evidence supports that view, what evidence argues against it, and what validation step should come next.

This is the philosophy HyperProbe is built around. The goal is not to ask engineers to trust an opaque RCA summary. It is to capture diagnostic evidence through read-only probes, preserve the context around the failure, and help the engineer confirm the root cause faster. That distinction matters. In production incidents, “the AI says so” is not an acceptable causal chain.

RCA is also where many AI systems overstate their confidence. Distributed failures rarely announce themselves cleanly. A downstream timeout may be the visible symptom, while the real cause is a slow dependency, a bad config push, a connection-pool limit, a partial deploy, or an unexpected interaction between services. An AI SRE should make that uncertainty explicit and keep alternative hypotheses alive until the evidence eliminates them.

Incident timelines and postmortems

Incident postmortem automation is one of the least controversial uses of AI SRE. The system can capture actions, decisions, timestamps, communications, metric changes, deploy events, and remediation attempts while the incident is still active.

Afterward, it can draft a timeline, summarize customer impact, propose action items, and turn recurring patterns into playbook candidates.

This will not make postmortems automatically good. Humans still need to decide what was causal, which actions to prioritize, and how to write a blameless analysis. AI can remove a lot of reconstruction toil.

Runbook execution

AI can make runbooks more usable. It can find the right runbook, map the current incident to the relevant branch, fill in service names or regions, check preconditions, and propose an action.

In a mature setup, it can also verify postconditions after the action: alert cleared, error rate normalizing, affected pods healthy, latency returning to baseline.

The key is to separate runbook recommendation from runbook execution. Suggesting “restart this worker pool” is not the same as restarting it. Parameterizing a rollback is not the same as executing it across production. AI should earn execution authority workflow by workflow, not inherit it globally.

Where autonomous remediation becomes risky

Autonomous remediation becomes risky when an AI system moves from reading production to changing production. That boundary is operational. A read-only investigation can waste time or lead to the wrong hypothesis. A state-changing action can expand the outage.

The risk increases when an action has a broad blast radius, ambiguous rollback, hidden coupling, incomplete telemetry, customer or revenue implications, or interaction with another live change.

Database failovers, schema changes, region-level traffic shifts, global cache flushes, broad restarts, security-policy changes, and feature-flag flips tied to billing or conversion are not good first candidates for full autonomy.

The right mental model is copilot before autopilot. A safer workflow uses deterministic tools, scoped permissions, policy checks, human authorization, and a durable audit trail before mutating production.

The AI can propose the action, explain the expected effect, list the rollback path, and verify the result. That does not mean it should always be allowed to execute the action by itself.

The hard part is not generating a plausible command. The hard part is knowing whether the command is safe under the current production state, whether it conflicts with another action, whether it can be reversed, and whether the system is healthier afterward.

This is why autonomous remediation needs more than model confidence. It needs bounded actions, rollback semantics, post-action verification, and clear rules for giving control back to humans.

Human-in-the-loop vs. fully autonomous workflows

The practical question is which workflow deserves which level of autonomy.

A read-only AI investigation workflow is the easiest place to start. The AI gathers evidence, summarizes context, ranks hypotheses, and drafts updates, but it cannot mutate production.

HyperProbe fits naturally here because its read-only probe architecture, default PII redaction, and immutable audit logs support evidence capture without turning every investigation into a write-capable automation path.

The next level is human-approved execution. The AI proposes a runbook-backed plan with preconditions, expected impact, rollback steps, and verification checks. A human approves the action, and the system executes through scoped tools.

Fully autonomous remediation should be narrower. It belongs to pre-approved, low-blast-radius, reversible actions with clear success criteria: restart a known stateless worker, scale a saturated deployment within policy, roll back a recent canary, or rotate traffic away from a single unhealthy instance. Even then, autonomy needs scoped identity, RBAC, dry runs, policy gates, rollback, audit logs, and post-action verification.

Teams should not flip a global switch from manual operations to autonomous operations. They should graduate individual workflows. A stateless restart might move to approval-based execution after several successful recommendations. A canary rollback might become automatic only after the team has clear checks for customer impact, deploy state, and rollback success. A database failover may remain human-approved indefinitely. That is not a failure of AI adoption; it is good production engineering.

What teams should realistically expect from AI SRE

Teams should expect AI SRE to reduce investigation drag before it eliminates on-call work. The realistic gains are fewer blind pivots, faster context gathering, more consistent triage, cleaner handoffs, better incident summaries, faster postmortems, and more disciplined runbook use.

They should not expect perfect RCA, universal zero-touch remediation, or a system that can compensate for missing telemetry and stale documentation. AI SRE depends on operational inputs: service ownership, deploy metadata, logs, metrics, traces, runbooks, dependency maps, past incidents, and clear access boundaries. If those inputs are incomplete, the AI will either ask better questions or make weaker guesses.

The best implementations start with a measurement plan. Track time from alert to first useful hypothesis. Track time spent gathering logs and change context. Track how often AI-suggested hypotheses were confirmed, rejected, or partially useful. Track whether postmortems became faster and more complete. MTTR reduction is the outcome, but diagnostic compression is the leading indicator.

Trust matters as much as speed. A fast suggestion that engineers ignore does not reduce MTTR; a cited, reviewable investigation has a better chance of changing responder behavior.

The maturity model: AI-assisted → AI-investigated → AI-recommended → AI-remediated

A useful way to evaluate AI SRE adoption is to treat autonomy as a maturity model, not a binary switch.

AI-assisted

At this stage, AI helps responders read and write faster. It summarizes alerts, explains dashboards, searches documentation, drafts status updates, and answers questions about known systems. It does not perform independent investigation, and it does not take production action. This is low risk and useful, but it will only modestly reduce MTTR if engineers still have to gather all evidence themselves.

AI-investigated

Here, the AI actively collects incident evidence. It pulls relevant telemetry, correlates recent changes, builds an incident timeline, finds similar historical incidents, and produces ranked hypotheses. Engineers still verify the facts, but they start from a prepared diagnostic brief rather than a blank page. This is where many teams will see the first meaningful MTTR reduction.

AI-recommended

At this stage, the AI proposes a remediation plan. The recommendation includes the suspected cause, evidence, runbook reference, expected impact, rollback path, and verification criteria. Humans approve execution. This is the right default for high-value production services because it accelerates action without removing accountability.

AI-remediated

The final stage is limited autonomy. The AI executes only pre-approved, reversible actions inside strict policy boundaries. It verifies the result, rolls back when conditions fail, and records the full trail for review. This can work for narrow classes of incidents, but it should be earned through repeated evidence, not assumed because the model sounds confident.

Conclusion: reduce MTTR by reducing uncertainty

The strongest case for AI SRE is not that it can take production away from engineers. It is that it can remove the investigative drag that keeps engineers from making a confident decision sooner.

MTTR stays high when responders have to rebuild context from scratch during every incident. They need to know what changed, what is affected, which signal matters, which hypothesis is most likely, which action is safe, and how to prove the system recovered. AI can help with that. It can gather evidence, preserve timelines, connect symptoms to recent changes, surface similar incidents, draft postmortems, and recommend the next step with enough context for a human to challenge it.

That is a meaningful reduction in operational load. It gives engineers back minutes during the incident and hours after it. It also improves response quality because the team spends less time searching and more time deciding.

Autonomous remediation should come later, and only for workflows that have earned it. The safer path is to move from AI-assisted to AI-investigated to AI-recommended before allowing AI-remediated actions. Each step should be backed by evidence, scoped access, rollback, verification, and auditability.

For teams evaluating AI SRE, the question should not be “Can this system fix production by itself?” A better question is: “Can this system help our engineers reach the right conclusion faster, with evidence they trust?” That is where MTTR reduction starts.

HyperProbe is built for that evidence-first path: faster investigation, verified root cause, and human-controlled production decisions.

FAQ

Can AI SRE really reduce MTTR?

Yes, but usually by reducing investigation time rather than by fully automating production fixes. AI SRE is most useful when it helps engineers move faster from alert to evidence-backed hypothesis, then preserves the context needed for verification and postmortems.

What part of incident response can AI SRE automate today?

AI SRE can automate or assist with alert triage, log and metric investigation, change correlation, root-cause hypothesis generation, incident timelines, postmortem drafts, and runbook recommendations. These workflows are valuable because they are evidence-heavy and often repetitive.

Can AI SRE automate root-cause analysis?

AI SRE can assist root-cause analysis by producing ranked hypotheses, supporting evidence, counter-evidence, and next validation steps. It should not be treated as a final authority because production incidents often involve incomplete telemetry, hidden dependencies, and multiple plausible causes.

When does autonomous remediation become risky?

Autonomous remediation becomes risky when the action changes production state, has broad blast radius, is hard to roll back, depends on incomplete data, or affects customers, revenue, security, or compliance. These actions should usually require human approval.

What is the safest way to adopt AI SRE?

The safest path is a staged maturity model: AI-assisted, AI-investigated, AI-recommended, and only then AI-remediated for narrow, reversible workflows. Each stage should include evidence, scoped permissions, rollback, verification, and audit logs.

Tolerant Decoding, Strict Release Gates

Fri, 23 Oct 2026 06:38:59 +0000

A release verifier has two jobs that are easy to blur together. It must understand the representation produced by the build chain, and it must enforce the policy that decides whether an artifact may leave that chain.

Those jobs need different attitudes.

Decoding can be narrowly tolerant of documented, meaning-preserving variation. Release policy should remain exact and fail closed.

That distinction surfaced in a recent mobile build repair. A generated JSON settings file could begin with the optional UTF-8 byte-order marker. The JSON content was valid, but the verifier decoded it as plain UTF-8 and failed before it could evaluate the values that actually governed release safety.

The fix was not to “clean up” the input generally. It was to accept one specific encoding variation, then run the same strict semantic checks as before.

Representation is not policy

A byte-order marker sits at the representation boundary. In this case, accepting UTF-8 with or without that marker did not change the JSON values.

An endpoint or feature-handoff setting is different. It changes what the package will do after installation. That is policy-bearing data.

Treating both categories as raw bytes creates a brittle gate: a harmless encoding detail can block a valid package. Treating both categories as broadly normalisable creates a permissive gate: a meaningful difference may be trimmed, defaulted, coerced, or discarded before verification sees it.

The useful middle ground is narrow tolerance:

Decode only the representation variants the producer is documented to emit.
Parse into an explicit data shape.
Compare policy-bearing values exactly.
Publish nothing until every check has passed.

This is a small design choice, but it keeps compatibility and authority in their proper layers.

Why broad normalisation is risky

“Normalise before comparing” sounds sensible. Sometimes it is. The danger is that normalisation is often written without naming which differences are harmless.

Should surrounding whitespace be trimmed? Should an empty string become null? Should a missing value receive a default? Should a hostname lose its trailing separator? Should key casing be ignored?

Each transformation might be correct for one contract and wrong for another. Once a release gate performs all-purpose cleanup, reviewers can no longer see which deviations the gate intentionally accepts.

A better question is: who owns this variation?

The decoder owns permitted encoding markers. The JSON parser owns JSON syntax. The release policy owns the approved endpoint and capability settings. Package signing owns artifact identity. Device acceptance owns behaviour on the target platform.

Let each boundary make only the decision it has enough information to make.

Test tolerance and strictness together

A positive test for the newly accepted variant is necessary, but it is not enough. It proves compatibility; it does not prove that policy remained intact.

The stronger test set combines the representation variant with both approved and disallowed semantics:

a marker-bearing file with approved values passes;
the verifier emits a clean, parseable receipt;
a marker-bearing file with an unapproved endpoint fails;
a marker-bearing file with an unapproved handoff configuration fails;
either failure produces no distributable package and no success receipt.

Those negative cases are the important counterweight. They show that the change expanded the accepted representation, not the accepted deployment policy.

The no-output assertions also matter. A verifier that reports failure after copying the package has already lost the race. Publication must happen after verification, not alongside it. The absence of a receipt is useful too: downstream automation must not mistake a partially completed run for approval.

Make the artifact boundary transactional

File-based pipelines rarely have a database transaction around the whole release. We can still design them with transactional thinking.

Build into a private staging location. Verify the package, its embedded settings, its signature, and any required metadata there. Only after the full decision succeeds should the pipeline copy or promote the artifact and write the corresponding receipt.

On failure, the public output location should remain unchanged.

This ordering turns the output directory into a simple commit boundary. Before promotion, the candidate is merely work in progress. After promotion, the package and its evidence agree.

A practical review checklist

When one tool generates a file and another tool gates a release from it, review four things:

1. Enumerate representation variants

Record the encodings, markers, line endings, ordering guarantees, and optional fields the producer may legitimately emit. Avoid accepting variants merely because a parser can.

2. Identify policy-bearing fields

List the values that change destination, authority, enabled capabilities, signing identity, or runtime behaviour. Compare them deliberately rather than through a generic object cleanup step.

3. Add crossed tests

Test each permitted representation with approved values and with at least one disallowed value. This catches the accidental move from narrow compatibility to broad permissiveness.

4. Prove fail-closed publication

Assert not only that verification returns failure, but also that no package, receipt, or other success signal appears in the publish location.

The leadership lesson

Strictness is not the same as rejecting every variation. Good gates are precise about what they tolerate and equally precise about what they refuse.

That precision makes a pipeline easier to operate. Engineers can fix harmless toolchain differences without weakening release policy. Reviewers can see the exact compatibility allowance. Operators can trust that a failed gate did not leave an ambiguous artifact behind.

The goal is not a gate that never bends. It is a gate that bends only where meaning is unchanged—and stays immovable where deployment meaning begins.

A Clean Scan Is a Data Point, Not a Verdict: Lessons from PHP2Shell Week

Thu, 22 Oct 2026 05:13:41 +0000

PHP's two critical flaws, CVE-2026-109618 and CVE-2026-109619 (PHP2Shell), were patched in 8.5.12, 8.4.27, 8.3.36 and 8.2.35. The advice everywhere was to update. I want to write about something less covered: how much you can honestly claim afterwards.

The scan that found nothing

I ran Laravel Scalpel, an open source filesystem intrusion scanner, against three Laravel applications after updating. No backdoor files. My first instinct was relief. My second was to ask what that result actually rules out.

Why a negative result is weak evidence

A scan can only report on what it can see, and it can only recognise what it knows how to recognise. Scalpel looks at five things: PHP files in places they should not be, known obfuscation patterns, risky directives in .htaccess and .user.ini, the integrity of .env, and differences from a saved baseline. Each is a question it answers well. Anything outside them is a question it never asks. Coverage and sensitivity multiply, and if either is low, 'found nothing' tells you little.

There are at least two explanations for my result. My applications are small, which lowers the odds that anything needed to hide. Or something sits outside the questions above. A scanner inside the application also shares the web server's user, so a flaw that grants code execution in that process gives an attacker the means to tamper with the tool. I ran Scalpel from inside the application and through an external runner for that reason, which narrows the gap without closing it. My strongest evidence is not the scan but a routine that predates it: since April, a scheduled external diff against my baseline has reported to a Telegram bot. I refreshed the baseline myself after the update, so any unexpected change before that should already have alerted me, and none did. That evidence has conditions too: the runs must actually have happened in the window, and the alert level must cover what an attacker would change. Scalpel's documentation states the first limit plainly, and I would rather state all of them than have a reader discover them. From the output alone I cannot separate these cases.

A ladder of claims

It helped me to treat each check as supporting a different, specific statement:

Version verified: the PHP that serves web traffic is a fixed version, containers included. Supports: 'this flaw can no longer be used against us.'
Logs reviewed from 20 October: no repeated 500s or memory limit errors that look like probing. Supports: 'we saw no sign of attempts in these logs.'
Filesystem scan clean: no known backdoor patterns in the application files. Supports: 'we found no evidence of intrusion in these files.'
Secrets rotated: exposed credentials no longer work. Supports: 'even if memory was read, those secrets are dead.'
Rebuilt from a known good image: supports the only claim close to 'clean.'

Most of us stop at rungs one to three and announce we are fine. Only the last rung justifies that word.

Why the timing makes this sharper

The identifiers leaked about a day before the fix, and a security firm reported exploitation attempts shortly after release. For teams that patched hours or days later, the unprotected window was real, and a patch does nothing about what happened inside it.

What I would write in a post-patch note

Something like: 'Updated PHP on this date. Reviewed logs from 20 October. Scanned the application filesystem. Found no evidence of intrusion in those places. Rotated these secrets.' Each sentence is a claim you can defend, and the places you did not check are visible by omission.

I would be glad to hear what steps other teams include in their own post-patch verification.

Kubernetes Security Is Three Problems, Not One: Admission, Identity, Runtime

Thu, 22 Oct 2026 03:19:32 +0000

🛡️ Kubernetes Security Is Three Problems, Not One: Admission, Identity, Runtime

TL;DR

Most Kubernetes compromises start inside the perimeter, where the firewall verifies nothing.

Treat security as three independent problems: admission (what deploys), identity (what a pod can reach), runtime (what a pod actually does).

Admission: Pod Security Admission as a free floor, Kyverno for per-resource policy, Gatekeeper where you need Rego. Run in audit mode first, then enforce.

Identity: IRSA kills static AWS keys, ExternalSecrets kills hardcoded app secrets. Scope roles per workload.

Runtime: Falco watches syscalls via eBPF and catches what the manifest never declared. Budget time to tune per-workload rules.

Supply chain is the layer most teams skip: Cosign signing plus Kyverno verifyImages closes the mutable-tag gap, and SLSA provenance plus SBOMs answer "which running images have this CVE?"

Kubernetes security failures usually do not look like network intrusions. An attacker who gets code execution inside a container, through an application bug, a poisoned dependency, or a policy gap that let a privileged pod schedule, is already inside the perimeter the firewall was watching. From there the firewall verified nothing: which secrets the process can reach, which syscalls it can make, whether the running image is even the one you signed.

That is the useful reframe. Kubernetes security is not one problem at the boundary. It is three problems inside it, and each one needs its own control that keeps working when the others fail:

Admission decides what is allowed to deploy, and with what configuration.
Identity decides what each running pod is allowed to reach.
Runtime watches what a pod actually does, which is often not what its manifest claimed.

None of the individual tools below are new, and none of this is a new technique. What is worth sharing is how the layers fit together, so a failure in any one of them does not open an exploitable gap. The model maps cleanly onto NIST SP 800-207 (zero trust) and NIST SP 800-190 (container security), and you can hold it to the CIS Kubernetes Benchmark as a measurable floor.

Layer 1: admission, decide what gets in

Start with what Kubernetes gives you for free. Pod Security Admission (PSA) enforces security profiles per namespace with no external tooling. The Restricted profile blocks host namespace sharing, privilege escalation, root contexts, and unsafe seccomp settings. A practical split is Restricted for externally facing namespaces and Baseline for internal platform namespaces with documented exceptions.

PSA is a floor, not a scalpel. It enforces at the namespace level, so it cannot say "this specific Deployment must run as UID 1000 with this seccomp profile." That is where Kyverno comes in. Its policies are plain YAML, they live in the same GitOps repo as your manifests, and they go through the same pull request review as any other change. Four rule types cover most needs:

validate rejects non-compliant resources (for example, require every image to come from an approved ECR registry).
mutate corrects omissions automatically, which turns "you forgot a security context" from an admission failure into a silent fix.
generate creates dependent resources on creation.
verifyImages enforces signature checks at admission time (more on that below).

For constraint logic that depends on the state of other cluster resources, or cross-namespace rules, OPA Gatekeeper and Rego express things Kyverno's YAML cannot. Running both is a reasonable choice: Kyverno for the common case, Gatekeeper where you need Rego's relational power.

One rollout lesson worth stating plainly: both tools have an audit mode, and you should use it first. Measure violations with enforcement off, remediate systematically, then switch to enforce. Flipping straight to enforcement is how a security rollout turns into a production incident.

Layer 2: identity, kill the static credential

A static AWS key in a ConfigMap or an environment variable is a persistent risk with a wide blast radius. IRSA (IAM Roles for Service Accounts) removes the category entirely. The AWS SDK in a pod annotated with an IRSA service account exchanges a short-lived, audience-scoped JWT for temporary IAM credentials through STS. Nothing static is created, stored, or rotated. The IAM trust policy validates the token's subject claim against a specific service account and namespace, so a compromised pod in one namespace cannot assume another namespace's role.

The payoff is blast-radius reduction you can describe concretely. Without IRSA, a compromised pod inherits whatever the node's instance profile can reach. With IRSA, it is limited to that one pod's scoped role.

Application secrets need a parallel mechanism, and that is the ExternalSecrets Operator. It syncs values from AWS Secrets Manager or Parameter Store into Kubernetes Secrets and refreshes them on the source's rotation schedule. When Secrets Manager rotates a database password, the new value propagates without a deployment. IRSA plus ExternalSecrets together removes every static credential: cloud access through IRSA, application secrets through ESO, nothing hardcoded in a Helm values file.

Scaling per-workload IAM roles across hundreds of workloads is the part teams tend to defer. Attribute-based access control with IAM permission boundaries makes it tractable: a role that expresses "this service account, in this namespace, for this environment" can be generated from workload metadata instead of hand-authored each time.

Layer 3: runtime, watch what actually happens

Admission decides what can deploy. Identity decides what it can reach. Neither sees what a pod does after it is running. Falco does, using eBPF probes at the kernel level to observe every syscall from every container on the node. A shell spawned in a container whose image has no shell in its approved process list raises an alert, regardless of how the compromise happened or whether the attacker edited the deployment manifest. Because eBPF sits below the container runtime, it keeps working even if the runtime is tampered with, and it needs no change to the image or the application.

Two practical notes. First, uniform rules across every workload generate false positives that erode trust in the alerts. A data-processing pod that legitimately reads many files will trip rules tuned for an application server. Per-workload rule sets, stored in Git next to the manifests, let you tighten baselines for payment or secret-handling services and loosen them for CI runners. Second, route Falco alerts into a SIEM so individual detections become correlated events. A shell-spawn alert followed within seconds by an outbound connection to an unknown IP is a pattern a correlation engine can act on, including automated network-policy quarantine of the affected pod.

Plan for a calibration period. Default Falco rules produce false-positive volume that takes iterative tuning, with both security and application knowledge, before the layer runs at production-grade sensitivity.

The layer most teams skip: supply chain

An image that passes admission is trusted to be what its manifest says, but only if the reference is cryptographically verified. Tags are mutable. Someone can overwrite the content behind a trusted tag and the cluster will deploy it under the same name, which bypasses every admission policy that checks the image's properties rather than its identity.

Cosign signs images at the registry, and Kyverno verifyImages rules reject anything at admission that is not signed by an approved identity. That closes the tag-mutability gap. SLSA provenance is the next step up: signing proves an image came from your build system, provenance proves what went into that build (source commit, build platform, resolved dependencies). A Software Bill of Materials attached as an attestation then lets you answer "which running images contain this newly disclosed CVE?" by querying the SBOM store instead of inspecting containers one by one.

Putting it together

The point of three independent layers is that no single failure is fatal. A policy gap that lets a bad pod through is still caught by scoped identity and runtime monitoring. A runtime compromise still cannot reach credentials the pod's IAM role never granted. A swapped image is rejected before it schedules.

If you are starting from scratch, a reasonable order is:

Turn on Pod Security Admission at Baseline or Restricted. Free, immediate.
Add Kyverno in audit mode, learn your violations, then enforce.
Move workloads to IRSA and ExternalSecrets to delete static credentials.
Deploy Falco, then budget real time to tune per-workload rules.
Add Cosign signing and verifyImages, then SLSA provenance and SBOMs.

The honest tradeoff is maintenance. Policy-as-code needs upkeep as Kubernetes APIs and workloads change, and Falco rule tuning is ongoing work. The reason to pay that cost is coverage no single control gives you: admission, identity, and runtime failures are different problems, and defending only one of them leaves the other two open.

php2shell: how are you confirming which PHP your web traffic actually runs on?

Thu, 22 Oct 2026 01:34:44 +0000

PHP's 21 October fixes (8.5.12 / 8.4.27 / 8.3.36 / 8.2.35) cover two critical runtime flaws. Because they sit below the application, updating a CMS or framework doesn't help — the runtime has to change and workers have to restart.

The step I'd expect to trip teams up is inventory. php -v on a shell tells you about the CLI, not necessarily what PHP-FPM, mod_php or a bundled appliance is serving.

What are you using to answer "where does PHP run and is it patched?" — package queries, phpinfo() behind auth, image scanning, something else? And has anyone found a clean way to confirm old FPM workers are gone after the upgrade?

Write-up with versions, interim settings and detection signals: https://dev.to/_firelinks/php2shell-a-request-that-fails-halfway-still-leaves-state-behind-5877

Use service_healthy gating in Compose — prerequisites, decision, and a limitation

Wed, 21 Oct 2026 23:23:24 +0000

Prerequisites

You control the service image (or can add a lightweight wrapper) and can add a healthcheck that exercises the real readiness condition.
You use Docker Compose v2-style depends_on conditions (support varies by Compose implementation/version).

Implementation decision

Put the readiness check on the dependency, not on the dependent. Define a healthcheck that verifies the real work boundary (for example, a small query or a completed bootstrap signal). Then set the dependent’s depends_on to use condition: service_healthy. This makes Compose wait for the backend’s own claim of readiness instead of relying on timed waits.

Expected result

Compose starts services in right order and then blocks the dependent until the backend reports healthy. This eliminates the startup race where the dependent is running and fails because its backend is still booting. If the backend lacks a healthcheck, Compose can’t honor service_healthy — which surfaces the missing readiness contract rather than silently permitting a race.

One limitation

A healthcheck must be honest and relevant. If it only checks that a TCP port is open, it may still return healthy before the backend can serve the app’s real needs. When the healthcheck is superficial, the gating gives you a false sense of correctness. Keep the check aligned to the actual dependency boundary.

Working example

export function evaluateStartability(services, targetServiceName) {
  if (!services || typeof services !== 'object' || Array.isArray(services)) {
    throw new TypeError('services must be an object')
  }

  const target = services[targetServiceName]
  if (!target) throw new Error(`service not found: ${targetServiceName}`)

  const deps = target.depends_on ?? {}
  const normalizedDeps = {}

  if (Array.isArray(deps)) {
    for (const name of deps) normalizedDeps[name] = { condition: 'service_started' }
  } else {
    for (const [name, val] of Object.entries(deps)) {
      if (val == null || val === 'short') {
        normalizedDeps[name] = { condition: 'service_started' }
      } else if (typeof val === 'string') {
        normalizedDeps[name] = { condition: val }
      } else {
        normalizedDeps[name] = { condition: val.condition ?? 'service_started' }
      }
    }
  }

  const reasons = []
  for (const [depName, opts] of Object.entries(normalizedDeps)) {
    const dep = services[depName]
    if (!dep) {
      reasons.push({ dep: depName, ok: false, why: 'missing' })
      continue
    }

    if (dep.state !== 'running') {
      reasons.push({ dep: depName, ok: false, why: `not running (${dep.state})` })
      continue
    }

    if (opts.condition === 'service_healthy') {
      if (!Object.prototype.hasOwnProperty.call(dep, 'health') || dep.health == null) {
        reasons.push({ dep: depName, ok: false, why: 'no healthcheck defined' })
        continue
      }
      if (dep.health !== 'healthy') {
        reasons.push({ dep: depName, ok: false, why: `health=${String(dep.health)}` })
        continue
      }
    }

    reasons.push({ dep: depName, ok: true, why: 'satisfied' })
  }

  return { ok: reasons.every((r) => r.ok), reasons }
}

export function makeService(state = 'running', health = null, depends_on = undefined) {
  return { state, health, depends_on }
}

Read the full article: https://chriseugenerodriguez.com/blog/compose-wait-for-ready-service

Mapping 32k+ MCP Servers: How we are handling tool discovery without schema drift

Wed, 21 Oct 2026 14:28:26 +0000

Hey devs! 👋

If you are currently running automated code generation or orchestrating local agent loops inside Claude Code, Cursor, or Windsurf, you've probably noticed how fast legacy REST integrations break down under heavy production prompts.

The ecosystem is rapidly moving toward the Model Context Protocol (MCP) using JSON-RPC 2.0 transport layers. To map out where this tech is heading, we've indexed and tracked the data footprints of active servers across three main directories:

• Glama Layer: 22.7k–32.7k live environments tracked via active usage signal telemetry.
• mcp.so Index: 20k+ community schema setups.
• Smithery Layer: 6k–7.3k configurations supporting single-command automated setups.

We wrote a comprehensive engineering guide breaking down how to securely connect trendy components like GitHub repository hooks, PostgreSQL data layers, and Playwright automation routines without blowing up context limits or leaking environmental tokens.

Check out our structural layout mapping and read the full technical walkthrough on our main blog:
🔗 https://www.seosiri.com/2026/10/mcp-server-registries-guide.html

Let's discuss in the comments below: What custom MCP servers are you currently spinning up in your workspace? 🛠️👇

SoftwareEngineering #AIAgents #ModelContextProtocol #WebDev #BuildInPublic

bash ran a path that no longer existed, and exited 127

Wed, 21 Oct 2026 13:00:00 +0000

A demo script called report, two directories on PATH, and one mv between them, all on a scratch tmpfs:

$ report
report 2.1: 14 jobs ok
exit=0
$ mv /opt/jobs/bin/report /opt/jobs/newbin/report
$ ls -l /opt/jobs/newbin/report
-rwxrwxr-x 1 root root 42 Oct  3 18:26 /opt/jobs/newbin/report
$ report
session.sh: line 5: /opt/jobs/bin/report: No such file or directory
exit=127
$ type -a report
report is /opt/jobs/newbin/report
$ hash -t report
/opt/jobs/bin/report
$ command -v report
/opt/jobs/bin/report

The file is there, on PATH, and bash ran the old path anyway. The first time it finds a command through a PATH search, it stores the full path in a hash table and skips the search after that. When the file moves, it executes the remembered path and reports No such file or directory with exit 127, the same code a typo gets.

The confusing part is the end of that output. type -a searches PATH fresh and finds the new location. command -v reports what bash would run, which is the stale one. hash -r clears the table, a new terminal starts empty, and shopt -s checkhash makes bash re-search when a hashed path has gone: with it set, the same sequence printed the tool's output and exit 0.

The stale hash is one of six causes behind command not found. The others, from a cron PATH to a CRLF shebang, each reproduced with a script that names the cause, are in command not found in bash, six causes.

TSB Went Live on a Letter Regulators Called "Statements of Good Intention"; £330M Aftermath

Wed, 21 Oct 2026 12:29:53 +0000

TSB's 2018 move off the Lloyds platform could not be rolled back. The FCA and PRA notices show the go decision rested on a supplier letter, a test run on one of two data centres, and a memo that left the letter's caveats behind.

On 10 April 2018, TSB's board met to decide whether to serve notice on Lloyds Banking Group. Serving it would commit the bank to moving its customers off the Lloyds IT platform it had run on since it was carved out of Lloyds in 2014, and onto Proteo4UK, a platform built for it by its new owner, Sabadell. It would also give up the fallback, a copy of the Lloyds platform that TSB had the option to carve out and keep. That same day, the last outstanding confirmation from the platform's supplier chain arrived.

The board went ahead. On the weekend of 20 to 22 April, TSB moved its customers' accounts across, predominantly in a single event. When the new platform opened, its online and mobile channels failed, and they kept failing. TSB "did not return to a BAU position until 10 December 2018," the FCA found, and between April 2018 and April 2019 it received 225,492 complaints.

There was no way back. The FCA's Final Notice puts it plainly: "after MME it was, technically speaking, essentially impossible to roll back from the changes made and revert to using the LBG IT Platform." MME is the Main Migration Event, that April weekend.

So the go decision was the whole decision, and what it rested on is the story. The record is unusually complete: two Final Notices issued on 20 December 2022, one from the Financial Conduct Authority and one from the Prudential Regulation Authority; a third from the PRA in April 2023, against the bank's chief information officer; and the bank's own 2018 accounts. Read together, they show a cutover with no way back approved on three pieces of evidence: a letter, a test and a memo. None of the three was what it looked like.

Why the bank was moving at all

Sabadell bought TSB in July 2015, and the migration was the point of the purchase. Its offer document promised "efficiency cost savings in IT amounting to approximately £160 million per annum on a pre-tax basis, in the third full year after completion of the Offer," and said those savings would "derive from a full migration" onto Sabadell's Proteo platform. The FCA records that "The strategic rationale for the takeover included the expected financial returns from a full migration," and that "The aim was to achieve migration by the end of 2017."

Sabadell knew how to do this. It "had previously migrated seven banks onto" Proteo, the PRA notes, and its "existing methodology for IT migrations was to use a 'big bang' or single operation data migration approach." This one was different in size, as the FCA notes: "The scale of the migration project was unprecedented," and the Proteo4UK platform "was not yet proven."

TSB's board saw the shape of the risk early. In December 2015 the minutes recorded that "the Board could not be put in a position where they gave up the carve-out option, migration "stalled" or was found to be unworkable leaving the bank in a position without an IT infrastructure." That is, in so many words, the risk the go decision would later take.

The schedule slipped once. In September 2017 the board was told the bank could not migrate in November as planned. It acknowledged that the November date, set two and a half years earlier, had been "deliberately very ambitious", had acted as a "forcing mechanism", and had been "based on very little information". The new "Defender Plan" was approved on 24 October 2017. Then, according to the PRA, "The Migration Programme quickly fell behind the Defender Plan schedule, with the result that critical testing plans and principles had to be deviated from to keep on track for MME."

The letter

The supplier was SABIS, Sabadell's IT subsidiary, which built and ran the platform. On 5 April 2018 it wrote to TSB expressing "confidence as to the migration readiness of the platform." The letter gave "an early report on certain test results (noting that some tests were still to be completed)" and referred to confirmations from SABIS's own suppliers, the "Critical Fourth Parties." Those confirmations "were referenced in (but not appended to) the letter."

Both regulators later read those confirmations and reached the same verdict, in the same words. They were "to some extent forward looking statements of good intention or expectation rather than statements of fact about the completeness of readiness activities undertaken." And "All but one was caveated with a number of outstanding tasks or tests which had not yet been completed."

The letter then travelled up the bank inside a summary. The memo that recommended going ahead carried the chief information officer's attestation, which "asserted that SABIS was ready for MME and, without any further explanation, that Mr Abarca was satisfied that the SABIS Confirmation could be relied upon." The memo "did not annex the letter itself. Nor was the letter included in any of the papers for the Board." The caveats were not in the papers of the board that committed the bank.

The dates make it starker. The last supplier confirmation "was received on 10 April 2018, the same day that the TSB Board was meeting to determine whether to serve the definitive notice of migration on LBG." The tests the letter called outstanding "were completed on 17 April 2018." The board's decisions came on 10, 18 and 19 April, and the migration began on the 20th.

One more change sat underneath. On 9 February 2018, confirming that non-functional testing was complete became the IT function's sole responsibility. The FCA found that this change "does not appear to have been explicitly drawn to the attention of the TSB Board" when it made those April decisions.

The test that ran on half the system

The failure itself was not in the data. The PRA is explicit that "the data migration itself was successful." What failed was the way customers reached the new platform.

Proteo4UK ran in two data centres in an "Active-Active" configuration, with a Global Load Balancer in front deciding which centre served each session. Each centre "had the capacity to run the entirety of the customer sessions on its own." And there was nowhere safe to rehearse: "TSB did not have a pre-production environment on which to conduct testing which would stress the platform without affecting the live production environment."

TSB first decided to performance-test both data centres together, "to ensure that the conditions were the same as expected at MME." After a testing forum on 28 February 2018, it accepted a "counter-proposal" from SABIS to test only one, "due to concerns that the live services would otherwise be unavailable (for example ATMs would not have been available for a period of approximately three to four hours)."

The notice records the reasoning. The configuration of the two centres was assumed to be "symmetric following assurances received," and "if it could pass with half the capacity, it would be able to easily pass using data centres." So the test, too, stood on an assurance. The NFT Final Report and the memo that went with it did not refer "to any risks associated with conducting performance testing in only one data centre."

Then the platform went live, and "following MME it became apparent that both the Global Load Balancer and the Security Session (OpenAM) component of the data centres had not in fact been correctly configured, leading to the issues experienced by customers in trying to access the digital channels."

The notice states those facts and stops. The inference is ours, and it is the operator's lesson of the whole case. A capacity test on one node proves capacity. It cannot prove that two nodes are the same, and it never exercises the component that routes between them. The fault sat in exactly the parts the one-centre test left out: the load balancer, whose job is to divide traffic between the two centres, and the session component, whose configuration was assumed to match. The test was real and it passed. It answered a different question from the one the go decision needed answered.

TSB's incident plan had made a similar assumption about failure itself. The bank "was prepared to deal with "bumps in the road"" but not "a massive failure in the configuration leading to an erratic digital service."

The bill

TSB's 2018 annual report puts the post-migration charges at £330.2 million. The two largest pieces were customer redress and associated costs, £107.3 million, and additional resource and advisory costs, £122.4 million. Against that the bank booked "the estimated recovery of £153.0 million of post migration charges" from SABIS under its two service contracts, "up to the level of liability caps in each agreement." Net of the recovery, the charges came to £177.2 million.

That recovery deserves a sentence of context. SABIS is "a subsidiary of the Banco Sabadell Group," and so is TSB. The £153.0 million moved money from one Sabadell subsidiary to another. The accounts read here do not net the group's position, so they cannot say what the group as a whole lost.

Then there were the costs of getting to the cutover. The same report shows £417.3 million of costs for preparing the migration, including "£249.0 million payable to Sabadell under the Migration Services Agreement." Preparation and aftermath together come to £747.5 million in one year's accounts. That is 4.7 times the annual IT saving the takeover promised. The comparison is a sense of scale, not a payback calculation: the promise was for the third full year after the deal, and nothing read here says whether it was ever met. (Our list of the most expensive software failures in history does not include TSB; on these figures it would not look out of place there.)

The regulators priced the failure more than four years later. The FCA fined TSB £29.75 million and the PRA £18.9 million, £48.65 million in all. That is 14.7% of the gross post-migration charges, or 27.5% of the net. Both fines ran the standard discount chain: the FCA's started at £50 million and came down by 15% for remediation and 30% for settling. The FCA found the breaches "committed negligently", and the PRA found them "neither deliberate nor reckless."

The customers' redress is in the record too. To 7 April 2019, TSB paid £32,705,762 in redress against 225,492 complaints, about £145 per complaint. The complaints equalled about 4.3% of a customer base the notices put at "approximately 5.2 million," roughly one for every twenty-three customers.

The one person in the record

One individual penalty appears in the published record. In April 2023 the PRA fined Carlos Abarca, TSB's chief information officer through the migration, £81,620. That was 15% of his relevant income of £777,356.86, cut by 30% for settling. He owned, by his senior-manager responsibilities, the risk that "migration causes operational instability or a degradation in resilience and poor customer outcomes."

The PRA's notice to him adds a detail. Before TSB, "Mr Abarca was Chief Process and Information Officer at Banco de Sabadell, S.A. ('Sabadell'), and at the same time, the Chief Executive Officer ("CEO") of Sabadell Information Systems, S.A. ('SABIS Spain')." At TSB he was responsible for the bank's "key outsourcing relationship with Sabadell Information Systems Limited ('SABIS UK')." The man whose job was to judge the supplier's word had run its Spanish counterpart.

The PRA drew the general rule from it: "The fact that a firm and its service provider are within the same group does not do away with the need for a careful assessment." Sam Woods, then the PRA's chief executive, put it in one line when the fine was announced: "the PRA has fined Mr Abarca because his management of a key outsourcing relationship fell below the standard we expect."

His fine was 0.17% of the firms' fines and 0.025% of the post-migration charges. The failure the PRA made personal was the management of an outsourcing relationship, down to an attestation that a supplier's letter could be relied upon.

What a readiness claim has to say

Strip the bank out of it and the case is about the grammar of "ready." SABIS's letter was written in the future tense: confidence, expectation, tests still to be completed. The board received it as a conclusion: the memo that carried it asserted readiness and left the letter, and its caveats, behind. Regulators later separated the two tenses for them: statements of good intention on one side, statements of fact about completed activities on the other.

The test made the same move in a different register. That the second data centre matched the first was a claim of fact. It rested on assurances, nobody had tested it, and the test plan treated it as settled.

For a team running any cutover, the lessons are plain. A readiness claim counts as evidence only when it says what was completed, not what is expected. An equivalence assumption is a test that has not been run, and it belongs on the list of open items rather than in the reasoning that closes them. And the less reversible the step, the more the go decision has to rest on evidence of the first kind. TSB's migration could not be rolled back, so its go decision was the last point at which the difference mattered.

The same grammar now turns up in software that reports on itself. An AI agent that says a task is "done" is writing SABIS's letter: a confident summary of activities, some of them still outstanding, written by the party that did the work. The fix is the one the regulators applied after the fact: ask for the statements of fact, attached, and read them before the step you cannot undo.

TSB's board met again on 18 and 19 April. The outstanding tests had finished on the 17th, and the memo in its papers still carried the attestation, not the letter. On 20 April the weekend began.

Sources

Financial Conduct Authority, Final Notice to TSB Bank plc, 20 December 2022 (115 pages), paragraphs 2.1-2.8, 2.24-2.25, 4.48-4.52, 4.179-4.205, 4.257-4.258, 4.267 and section 6.
Prudential Regulation Authority, Final Notice to TSB Bank plc, 20 December 2022 (108 pages), including paragraph 1.9 (Sabadell's offer document) and the penalty annex.
Prudential Regulation Authority, Final Notice to Carlos Abarca, 13 April 2023 (35 pages).
TSB Banking Group plc, Annual Report and Accounts 2018.
Financial Conduct Authority, press release, 20 December 2022; Bank of England, press release, 20 December 2022 and press release, 13 April 2023.

How this was computed. Every ratio above (fines against charges, the £747.5 million, the multiple of the promised saving, the redress per complaint and the CIO's fine against his income) is arithmetic on figures quoted in this piece from the notices and the 2018 accounts, so each can be recomputed from the text. Page and paragraph references are to the PDFs as published. Every quotation was checked against the saved text of its source.

Originally published at vibeagentmaking.com. If you want a tamper-evident, dated record of what a system actually completed, and when: pip install chain-of-consciousness or npm install chain-of-consciousness, or try Hosted Chain of Consciousness.

Software Maintenance Cost: What to Budget Each Year

Wed, 21 Oct 2026 00:02:58 +0000

Most people budget carefully for the build and then forget the bill that never stops: software maintenance cost. A web app, SaaS product or internal system is not a one-off purchase. It needs hosting, security patches, dependency upgrades, backups, monitoring and fixes for the problems real users find. If nobody pays for that work, the system quietly decays until it becomes the legacy application you were trying to avoid.

I look after production systems as part of my cloud, DevOps and deployment work, so this guide covers what maintenance actually includes, what drives the cost, and how to budget for it without guessing.

What software maintenance actually covers

"Maintenance" is a bucket for four different kinds of work. Knowing which is which makes the budget conversation much easier.

Corrective: fixing bugs found in production.
Preventive: security patches, framework and library upgrades, and clean-up that stops problems before they appear.
Adaptive: changing the system because the world changed, such as a new PHP version, a payment provider update or a browser change.
Perfective: small improvements and requests that make the product better but are not a new project.

New features are usually a separate budget. The trap is letting small feature requests quietly consume the maintenance hours that keep the system safe.

The recurring costs people forget

Beyond developer time, a live system has running costs:

Hosting and infrastructure: servers, databases, storage and bandwidth. My comparison of Laravel hosting options and cost shows how widely this varies.
Third-party services: email, SMS, payments, search, error tracking and AI APIs, often billed by usage.
Domain, SSL and licences for paid plugins or tools.
Backups and disaster recovery, including actually testing a restore.
Monitoring and alerting, so you hear about problems before customers do.
Security work: patching, access reviews and responding to vulnerability notices.

These costs are easy to estimate once you have a list. The surprise usually comes from usage-based services that grow with your traffic.

What drives maintenance cost up or down

I will not quote a percentage that pretends to fit every system. These factors decide where yours lands:

Age and technology. Systems on unsupported frameworks cost more to keep safe. This is the main link to rewriting or refactoring a legacy application.
Test coverage. Tested code is cheaper to upgrade because you can see what broke.
Number of integrations. Every third-party API is something that can change or fail.
Traffic and data volume. More load means more tuning and more infrastructure.
Quality of documentation. Missing knowledge makes every task slower.
Deployment process. Manual releases cost time and cause errors. A clean pipeline, as in my guide to zero-downtime deployments, lowers the cost of every future change.
Required uptime. A system that must always be available needs monitoring and on-call, which costs more.

Three ways to pay for maintenance

1. Retainer. A fixed monthly amount that covers a set number of hours or a defined scope. It suits systems with steady needs and gives you predictable budgeting and fast response.

2. Time and materials. You pay for hours as they are used. It is flexible for quiet systems, but costs are unpredictable and urgent issues compete with other clients' work.

3. Fixed-scope packages. Specific tasks, such as a framework upgrade or security audit, for a fixed price. These are good for planned one-off work on top of a baseline.

Many businesses use a small retainer for monitoring and urgent fixes, plus fixed-scope projects for bigger upgrades.

How to estimate your own maintenance budget

Use this simple worksheet:

List every running cost: hosting, services, domains, licences. Check last month's invoices for each.
Estimate routine hours: security patches, dependency updates, backup checks and monitoring review, per month.
Add a bug-fix allowance based on how often production issues have appeared so far.
Add planned upgrade work for the year, such as a major framework or PHP version.
Add a contingency for the unexpected, and treat it as real money.
Separate new features into their own budget so they do not eat maintenance time.

If the system is brand new and has no history, expect the early months to need more attention while real usage reveals problems. A short audit gives you a better starting point than a guess.

Why support dates decide your upgrade calendar

Two clocks set most of your planned maintenance: the language and the framework.

Language versions each get a limited support window, after which they stop receiving security fixes. PHP publishes these dates on its supported versions page.
Framework versions follow their own policy. Laravel documents which releases still receive bug and security fixes in its release notes and support policy.

Put both sets of dates in a calendar and plan an upgrade before a version runs out of support, not after. Doing one planned upgrade a year is far cheaper and less risky than jumping across several versions at once, which is the situation I describe in my Laravel upgrade checklist.

A good maintenance plan therefore includes a yearly "version review": check what you run, check the support dates, and decide what gets upgraded next. It takes a few hours and avoids an emergency later.

What happens when you skip maintenance

Skipping maintenance does not save money. It moves the cost to later and makes it larger:

Known security holes stay open
Upgrades pile up until they become a risky big-bang project
Small bugs turn into outages
Only one person understands the system
Hosting and dependencies freeze because nobody dares touch them

On the Henceforward WordPress project, the work combined website development with migrating infrastructure to Scaleway with Laravel Forge, strengthening security and optimising the production environment. That kind of foundation is what makes ongoing care cheaper. You can read the Henceforward case study for the details.

A simple way to report maintenance value

Maintenance feels invisible when nothing breaks, which makes it easy to cut. Ask for a short monthly note covering four things: what was patched or upgraded, which incidents occurred and how fast they were resolved, how backups and monitoring performed, and what is planned next. A one-page summary turns "nothing happened" into evidence that the system is being looked after, and it gives you the data to adjust the plan when needs change.

Questions to ask before you agree a maintenance plan

What exactly is included, and what counts as a new feature?
What is the response time for an urgent production issue?
Who monitors the system, and how will I be alerted?
How are backups tested?
What happens to the plan if the system grows?
Can I see a report of what was done each month?

A clear answer to these is a better signal than the lowest price. My thoughts on why experience beats low-cost outsourcing apply here too.

Key takeaways

Maintenance is four kinds of work: fixing, preventing, adapting and improving.
Running costs go beyond developer time: hosting, services, backups, monitoring and security.
Age, tests, integrations and deployment process drive the price more than size alone.
Choose a retainer for steady needs, fixed-scope packages for planned upgrades.
Skipping maintenance makes problems larger and more expensive later.

If you want a clear maintenance budget for a system you already run, book a call and I will review it and tell you what it really needs.

Originally published at aqibjavaid.site.

One Bash Script to Audit a Polyglot Codebase (Python, TS, Java, Go, C, HTML)

Mon, 19 Oct 2026 11:43:05 +0000

If your project mixes languages, one terminal command can run each language's own tests and security checks and fail the build when anything is red.

The script (1) counts files per language, (2) runs build and type checks, (3) runs tests and SAST, (4) scans dependencies and secrets, (5) runs a quick concurrency check, and (6) writes a checksummed report.

I tested it on a small demo project: python compile passed, missing scanners showed as SKIP, an invalid HTML file produced a real FAIL, and the run exited with code 1.

Full guide with per-language settings, a troubleshooting table and a CI example: https://www.seosiri.com/2026/10/polyglot-code-audit-bash-testing-pipeline.html#conclusion

Which scanner would you add first?

Database Migrations: Can Your Previous Release Still Run?

Mon, 19 Oct 2026 07:45:55 +0000

A deployment can roll back cleanly while the database stays incompatible with the old application. Before shipping a schema change, check whether the previous release can still read and write the data that the new release creates.

Consider a fictional account service. Version 1 reads a column called display_name. Version 2 expects public_name. Renaming the column and deploying the new code together looks tidy, until an old worker, a rolling-deployment instance, or an emergency rollback still asks for display_name.

The migration succeeded. The service can still fail.

Separate compatibility from cleanup

Danilo Sato’s explanation of parallel change describes three stages: expand, migrate, and contract. The useful idea is to introduce the new interface while the old one remains available, move consumers, then remove the old interface.

For a database rollout, turn that idea into explicit release gates:

Expand: introduce the new storage shape without removing what the currently deployed application needs.
Migrate: move reads and writes deliberately, checking compatibility during the transition.
Contract: remove the old shape only after its consumers and the intended rollback window are accounted for.

These stages are a planning model, not a guarantee of zero downtime. The exact migration still depends on the database, data volume, application behavior, and operational limits.

Work through the awkward middle

In the account-service example, adding public_name is only the beginning. Before copying values, decide which field is authoritative at each stage.

A possible transition is to deploy compatibility code that continues serving old readers while maintaining both values. If both columns live in the same row, update them in the same database transaction where practical. That still leaves an important question: can any old application instance or background job write only the original column?

Do not switch reads to the new column while those writers can silently make it stale. Either finish upgrading writers first or design a deliberate synchronization mechanism. A fallback read helps with missing values; it does not detect an outdated non-empty value.

Backfills need the same care. A naive copy can overwrite a newer value if it races with a live update. Define an update condition, concurrency strategy, and restart behavior before running the job. Record progress so an interrupted backfill can resume without blindly replaying the entire table.

Test the rollback with new data

A useful rehearsal includes more than deploying version 2 successfully:

Start with representative data, including empty and unusual values.
Apply the expansion and deploy the compatibility release.
Run the backfill alongside representative writes.
Create and edit records using the new release.
Restore the application version you actually intend to roll back to.
Verify that it can read and update those new records correctly.

Include background workers, scheduled jobs, exports, and direct database consumers in that exercise. A healthy web endpoint does not prove that every writer is compatible.

Use synthetic or appropriately protected test data. This checklist proposes evidence to collect; it does not claim that a production migration has been tested.

A compatible change can still block traffic

In PostgreSQL, ALTER TABLE generally acquires an ACCESS EXCLUSIVE lock unless the documentation specifies otherwise. Compatibility at the application level does not eliminate lock waits.

PostgreSQL also supports separating certain constraint additions from validation with NOT VALID and VALIDATE CONSTRAINT. The permitted constraint types and locking details matter. Check the documentation for your deployed version rather than applying the technique to every schema operation. See the PostgreSQL ALTER TABLE reference.

Before execution, define acceptable lock wait, query latency, replication lag, and backfill load. Decide who can stop the rollout and what the safe next step is if a threshold is crossed.

Make cleanup its own decision

Keep a short release note that answers four questions:

Which application versions can run against this schema?
Which data changes can the previous version understand?
What evidence allows the old column or interface to be removed?
After removal, does recovery mean application rollback, a forward fix, or data restoration?

Give the cleanup an owner. Leaving parallel paths indefinitely creates its own maintenance cost, but removing them too early destroys the compatibility you built.

The practical takeaway: rehearse rollback against data written by the new release before treating it as a recovery plan.

What is the most easily missed database consumer in your deployment process?

Waqas Ahmad | Software Engineer | Content Creator

Building the MyZubster Metaverse: From Public Characters to Secure World Actions

Mon, 19 Oct 2026 04:58:09 +0000

Building the MyZubster Metaverse: From Public Characters to Secure World Actions
MyZubster is building an open-source metaverse that connects public characters, verified knowledge, contributor identities, realtime interactions, and GitHub-based development.
The project is not limited to a graphical virtual world. Its technical foundation is based on authenticated actions, explicit user consent, verifiable evidence, secure automation, and independently inspectable deployments.
This is a technical update on what we have implemented, what we recently stabilized, and what remains before the platform can be considered production-ready.
The Architecture
The current architecture follows this general flow:
MyZubster account

↓

Authenticated session

↓

Time-limited session lease

↓

Zorgax or metaverse character

↓

Authorized world action

↓

Realtime event distribution

↓

Persistent and verifiable evidence
Each state-changing request should answer four questions:

Which account initiated the request?
Is its session still valid?
Which character is associated with that session?
Is the requested action permitted?
This is important because a client-provided character ID cannot be considered proof of ownership.
The Public Metaverse
The public interface is available at:
https://www.myzubster.com/metaverse
Our latest production verification confirmed that the page loads correctly and exposes:
- 25 public characters
- 7 verified explorers
- A deployed Vercel production build
- No blocking browser-console errors
  This public page represents the visible part of a larger system. Most of our recent work has focused on the security and infrastructure beneath it.
  Session-Bound World Actions
  World actions are being migrated to a session-bound authorization model.
  The server must derive the acting identity from an authenticated session instead of trusting identifiers supplied by the client.
  const session = await resolveAuthenticatedSession(request);

if (!session) {
throw new UnauthorizedError();
}

const lease = await resolveActiveLease(session.id);

if (!lease || lease.expiresAt <= new Date()) {
throw new SessionExpiredError();
}

const character = await resolveCharacterForAccount(session.accountId);

await authorizeWorldAction({
session,
lease,
character,
action: request.body.action,
});
This approach reduces the risk of:

Character impersonation
Cross-account actions
Replayed requests
Actions performed after logout
Use of expired sessions
Client-side manipulation of ownership fields
Authentication identifies the account. The session lease determines whether its authorization is still active. The policy layer determines which world actions are allowed.
Session Lease Lifecycle
An authenticated session should not provide unlimited access.
We introduced a lease lifecycle that can represent the following states:
CREATED
↓
ACTIVE
↓
RENEWED
↓
EXPIRED

ACTIVE
↓
REVOKED
A lease can be invalidated when:

The expiration time is reached
The user signs out
The account is disabled
Security credentials are rotated
An administrator revokes the session
A security policy invalidates older leases
Metaverse endpoints must verify the lease immediately before performing a protected action.
Zorgax and Contributor Consent
Zorgax connects several areas of MyZubster:
GitHub contributors
Public professional profiles
Knowledge Cards
Verified Knowledge
Metaverse characters
Development Requests
A public GitHub identity can generate an optional character proposal, but registration must require explicit consent.
GitHub contribution
```
  ↓
```
Character draft
```
  ↓
```
Contributor reviews proposal
```
  ↓
```
Explicit acceptance
```
  ↓
```
Public character registration
A character proposal does not imply:
Legal identity verification
Employment
Partnership
Ownership
Payment eligibility
Institutional endorsement
Keeping these distinctions explicit is essential for an open contributor ecosystem.
Realtime Communication
The realtime layer still needs to be completed, but its event structure must support reliable synchronization.
A typical event could look like this:
{
"eventId": "evt01...",
"worldId": "myzubster-main",
"characterId": "character...",
"sessionId": "session_...",
"sequence": 1842,
"type": "character.position.updated",
"occurredAt": "2026-10-19T04:50:00.000Z",
"payload": {
"x": 12.4,
"y": 0,
"z": -8.1
}
}
The event identifier and sequence number allow clients to detect:
Duplicate messages
Out-of-order updates
Missing events
Stale world state
After reconnecting, a client should send its last accepted sequence number and recover any missing state.
Distributed Rate Limiting
In-memory rate limiting works only for a single application instance. It is insufficient when requests are distributed across several workers or serverless deployments.
Production enforcement therefore requires shared state:
Client
↓
Application instance A ─┐
Application instance B ─┼→ Shared atomic rate limiter
Application instance C ─┘
Different policies are required for:
Login attempts
Session creation
Lease renewal
Character movement
World interactions
Realtime messages
Search endpoints
Profile updates
The system must also define what happens when the rate-limit service becomes unavailable: fail open, fail closed, or apply a limited fallback policy.
CI Runtime Modernization
We recently completed an important GitHub Actions upgrade.
GitHub warned that older JavaScript actions were using the deprecated Node.js 20 runtime. The affected references were updated from:
uses: actions/checkout@v4
uses: actions/setup-node@v4
to:
uses: actions/checkout@v5
uses: actions/setup-node@v5
The migration covered:
12 workflow files
26 action references
CI and lint
Security auditing
SBOM generation
Vercel deployment
Evidence validation
Seller policy regression testing
MYZ ledger verification
Pilot API testing
Decentraland SDK7 proof workflows
We validated all 16 workflow YAML files before publishing the change and confirmed that no affected v4 references remained.
The update was integrated through Pull Request #38.
The final commit on main is:
69ef67716df50482e8fbf9960ccab7dfc37d3000
Post-Merge Results
The merge triggered seven workflows on the production branch.
All completed successfully:
CI — Test and Lint
Security Audit
SBOM and Vulnerability Scan
Continuous Evidence Gate
MYZ Canonical Ledger Contract
MYZ-164 Seller Free Policy
Deploy to Vercel
The deployment workflow also completed successfully, confirming that the runtime migration did not break the production delivery process.
What Is Currently Working
The current baseline includes:
Public metaverse interface
Public characters
Verified explorers
GitHub-connected contributor identities
Optional Zorgax character onboarding
Session-bound authorization development
Session lease lifecycle enforcement
Knowledge and profile integration
Automated security auditing
SBOM generation
Continuous evidence validation
Node.js 24-compatible GitHub Actions
Automated Vercel deployments
What Still Needs to Be Built
Complete Authentication Enforcement
Every state-changing endpoint must use the same canonical authentication and authorization process.
Required scenarios include:
Expired sessions
Revoked leases
Logout
Credential rotation
Concurrent sessions
Disabled accounts
Replay attempts
Cross-account character access
Finish Distributed Rate Limiting
We still need to validate:
Atomic counters
Multiple application instances
Concurrent requests
Counter expiration
Boundary conditions
Backend failure behavior
Recovery after temporary outages
Complete Realtime Synchronization
The realtime layer must support:
Presence
Character movement
Interactions
Reconnection
Missed-event recovery
Duplicate suppression
Event ordering
Backpressure management
Add Privacy-Safe Observability
Operational monitoring should measure:
Active sessions
Rejected world actions
Lease-renewal failures
Realtime disconnections
Rate-limit activations
API latency
Backend error rates
The system should minimize personal data and avoid storing unnecessary raw identifiers.
Reliability and Load Testing
Before expanding public access, the platform needs tests for:
Concurrent users
Movement bursts
Realtime fan-out
Session-store latency
Infrastructure restarts
Partial failures
Deployment rollback
Rate-limit contention
End-to-End Acceptance
The complete flow must be tested as one system:
Sign in
→ select a character
→ create a session lease
→ enter the metaverse
→ perform an authorized action
→ receive a realtime event
→ disconnect
→ reconnect and recover state
→ sign out
→ verify that further actions are rejected
WebXR
WebXR remains part of the roadmap, but it should follow security, realtime communication, and reliability work.
Our planned order is:
1. Authentication and session enforcement
2. Distributed rate limiting
3. Realtime synchronization
4. Privacy-safe analytics
5. Reliability and load testing
6. End-to-end acceptance
7. WebXR experience
  Conclusion
  The MyZubster Metaverse now has a concrete technical foundation: public characters, contributor identity integration, authenticated session work, automated security checks, reproducible deployments, and a modernized CI runtime.
  It is not finished, and we do not want to describe experimental components as complete systems.
  Every future milestone should be accompanied by:
Public source code
Automated tests
Deployment evidence
Clear limitations
Reproducible verification steps
Our goal is to build the metaverse transparently, incrementally, and with evidence that contributors can inspect independently.

How We Secured the MyZubster Metaverse: Session Ownership, Verified Explorers and the Road to Realti

Sat, 17 Oct 2026 05:20:55 +0000

A technical update on the architecture, security controls, testing and remaining roadmap behind the open-source MyZubster Metaverse.
The MyZubster Metaverse is moving from an experimental interface toward a secure, testable multiplayer environment.
Our goal is not simply to display avatars in a browser. We want to create an open-source world connecting public profiles, verified contributors, Zorgax characters, community knowledge and software-development workflows.
The public Metaverse is available at:
https://www.myzubster.com/metaverse
This article describes what we have implemented, the security problem we recently solved and what remains before the platform can support realtime interaction at scale.
The current experience
A user can enter the MyZubster world as a guest without providing:

a cryptocurrency wallet;
identity documents;
a GitHub account;
an installed application.
The current interface supports guest onboarding through a public display name and also presents verified explorers associated with authorized public accounts.
This creates two identity levels:
1. Guest explorers use temporary world sessions.
2. Verified explorers have characters linked to a MyZubster account or confirmed GitHub identity.
  “Verified” does not mean KYC or legal identity verification. It only confirms control of the connected public account.
  Building the presence layer
  The backend maintains a presence record for every active participant.
  A simplified record may contain:
  {
  sessionId,
  displayName,
  characterId,
  accountId,
  githubLogin,
  position,
  status,
  lastSeenAt
  }
  This model supports the main world operations:
  Join
  Synchronize
  Move
  Chat
  Emote
  Leave
  Retrieve events
  Public players must be discoverable by other clients, so each presence has a public session identifier.
  However, a public identifier must never be treated as proof that a request belongs to that player.
  That distinction became the focus of our latest security milestone.
  The original session-ownership problem
  A multiplayer client needs to identify which session it wants to update.
  For example:
  {
  "sessionId": "public-session-id",
  "x": 120,
  "y": 80
  }
  If the backend trusts only sessionId, a malicious user may be able to submit another player’s identifier and perform actions on their behalf.
  Depending on the exposed routes, this could allow an attacker to:
move another character;
send chat messages under another identity;
publish unauthorized emotes;
retrieve session-specific events;
terminate another player’s session.
A public session ID is an identifier, not an authorization credential.
Implementing possession-bound sessions
We implemented a new security boundary in:
Pull Request #34 — Bind world actions to session credentials
The change was merged into main as commit:
148a798
Every new Metaverse session now receives two different values:
Public session ID
Private session token
The public ID can appear in world-state responses. The private token proves that the client controls the session.
Cryptographically random credentials
The server generates a token using 32 random bytes:
const sessionToken = crypto
.randomBytes(32)
.toString("base64url");
This provides 256 bits of entropy.
The raw token is returned only during session creation and is never included in public presence data.
Storing only a digest
The server does not persist the raw credential.
Instead, it calculates a SHA-256 digest:
const sessionTokenHash = crypto
.createHash("sha256")
.update(sessionToken)
.digest("hex");
Only sessionTokenHash is stored with the presence record.
This prevents a database reader from immediately obtaining active bearer credentials.
Constant-time comparison
When a protected request arrives, the server hashes the submitted credential and compares it with the stored digest.
The verification uses a constant-time comparison:
crypto.timingSafeEqual(expected, received);
A shared ownership middleware verifies that:
the session exists;
a credential was supplied;
the credential has the expected format;
its digest matches the stored value;
the caller owns the requested session.
Invalid or missing credentials return:
HTTP 401 Unauthorized
Protected world routes
Session ownership is now required for mutable and session-specific operations such as:
/sync
/move
/chat
/emote
/leave
/events
Knowing another participant’s sessionId is no longer sufficient to control their character.
The authorization rule is now:
Public session ID
```
  +
```
Private possession token
```
  =
```
Authorized world action
Keeping credentials out of public payloads
The frontend sends the private token through a dedicated HTTP header:
X-Metaverse-Session-Token: <private-token>
Example:
const createSessionHeaders = (sessionToken) => ({
"X-Metaverse-Session-Token": sessionToken,
});
The world-state body remains limited to operational data:
{
"sessionId": "public-session-id",
"x": 120,
"y": 80
}
This separation reduces the probability that credentials are accidentally copied into:
world events;
analytics;
public API responses;
application logs;
debugging output;
cached payloads.
The credential is also kept in React runtime memory instead of being written to localStorage or public profile objects.
An in-memory token is not a complete defense against browser compromise, but it reduces persistent credential exposure.
Identity boundaries
The Metaverse now distinguishes between several layers:
Account identity
```
↓
```
Public profile
```
↓
```
Character identity
```
↓
```
Temporary world session
```
↓
```
Private session credential
These layers have different meanings.
A verified GitHub account does not automatically prove legal identity, employment, payment eligibility or institutional affiliation.
Similarly, a public session ID identifies a player’s presence but does not authorize requests for that player.
Documenting these boundaries is essential for avoiding security and governance problems later.
Testing the implementation
The session-ownership change includes backend, frontend and architecture-level tests.
Backend tests
The targeted backend suites verify:
secure token generation;
SHA-256 digest storage;
valid credential acceptance;
missing credential rejection;
invalid credential rejection;
cross-session mutation prevention;
protected movement and chat;
protected emotes and leave operations;
exclusion of secrets from public responses.
Results:
3 backend suites
22 tests passed
Frontend tests
Frontend coverage verifies:
token retention in application memory;
use of the session-token header;
authenticated world operations;
removal of tokens from normal JSON bodies;
correct handling of protected session requests.
Results:
1 frontend suite
5 tests passed
Integration and deployment
The completed verification included:
Production frontend build: successful
git diff --check: clean
Marketplace regression suite: passed
Vercel preview deployments: successful
Main CI workflow: successful
Production deployment: successful
The public Metaverse route was also loaded after deployment.
Current architecture
The current system can be summarized as:
React Metaverse Client
```
  |
  | HTTPS
  | X-Metaverse-Session-Token
  v
```
Node.js / Express API
```
  |
  | Session ownership middleware
  | Hash verification
  v
```
Metaverse Presence Storage
```
  |
  v
```
Shared world synchronization
This is a practical foundation, but it is not yet the final realtime architecture.
What remains to be built
Realtime communication
The current synchronization model still depends on polling in important areas.
We need a versioned realtime protocol supporting:
authenticated WebSocket connections;
session-to-connection binding;
reconnect handling;
ordered events;
duplicate-event protection;
presence expiration;
backpressure;
slow-client management.
The protocol should remain usable by multiple clients, including future community-built and WebXR clients.
Distributed rate limiting
Per-process rate limiting is not enough when the backend runs across multiple instances.
We need a distributed limiter for:
Session creation
Authentication attempts
Movement
Chat
Emotes
Reconnects
Limits should be calculated using combinations of:
IP address;
account ID;
session ID;
route;
world ID;
time window.
Chat and authentication need stricter limits than normal movement updates.
Session expiration and revocation
The possession credential is now implemented, but lifecycle controls still need improvement:
explicit expiration;
server-side revocation;
abandoned-session cleanup;
token rotation;
reconnect-safe renewal;
active-session limits;
multi-device logout;
suspicious-session invalidation.
Canonical authentication
Verified characters require a stable account-linking process.
The remaining work includes:
canonical GitHub OAuth;
verified account-to-character relationships;
account-link substitution protection;
reauthorization;
identity-change audit trails;
account recovery procedures.
The backend must never trust a GitHub username supplied directly by a browser.
Privacy-safe analytics
We need enough telemetry to maintain the platform without introducing unnecessary tracking.
Useful operational metrics include:
concurrent sessions;
API latency;
reconnect frequency;
server errors;
world capacity;
rate-limit events;
event throughput.
Retention, aggregation and access rules must be documented before expanding analytics.
Reliability and load testing
Future tests must simulate:
simultaneous joins;
movement bursts;
reconnect storms;
malformed messages;
database latency;
backend restarts;
multiple server instances;
slow or disconnected clients.
We also need measurable targets for latency, error rate, recovery time and maximum concurrent participants.
Complete E2E acceptance testing
The complete guest flow should be automated:
Join
→ receive credential
→ become visible
→ move
→ chat
→ publish an emote
→ reject impersonation attempts
→ reconnect safely
→ leave or expire
The same acceptance flow is required for verified characters.
WebXR
WebXR remains part of the roadmap, but it should follow security and realtime stabilization.
The eventual world protocol should support:
2D web client
3D browser client
WebXR client
Mobile client
Independent community clients
All clients should use the same documented API and event protocol.
Dependency-security debt
The repository currently reports vulnerabilities during npm audit.
These findings were already present and were not introduced by the Metaverse session-ownership pull request, which did not change package manifests or lockfiles.
However, they remain technical debt.
A separate dependency-hardening milestone should include:
1. classifying production and development exposure;
2. updating direct dependencies;
3. reviewing transitive dependencies;
4. replacing abandoned packages;
5. regenerating lockfiles;
6. running complete regression tests;
7. updating SBOM and audit evidence.
  Disabling the audit to produce a green badge would hide the problem instead of solving it.
  Recommended roadmap
  The recommended implementation sequence is:
8. Session expiration and revocation
9. Distributed rate limiting
10. Canonical account authentication
11. Versioned realtime protocol
12. Authenticated WebSocket transport
13. Privacy-safe monitoring
14. Reliability and load testing
15. Complete E2E acceptance tests
16. Dependency-security remediation
17. WebXR client development
  Security, reliability and protocol stability must come before graphical complexity.
  Open-source participation
  We are looking for contributors interested in:
React;
Node.js and Express;
MongoDB;
Redis;
WebSockets;
application security;
privacy engineering;
abuse prevention;
load testing;
WebXR;
technical documentation.
Repository:
https://github.com/danieldirimini-myzubster/myzubster
Public Metaverse:
https://www.myzubster.com/metaverse
Security implementation:
https://github.com/danieldirimini-myzubster/myzubster/pull/34
Conclusion
The MyZubster Metaverse now has a stronger session-security foundation.
We moved from public identifier-based operations to possession-bound authorization using:
256-bit random credentials;
SHA-256 token digests;
constant-time verification;
dedicated authorization headers;
memory-only client credentials;
protected world routes;
backend and frontend security tests.
The next milestone is transforming this foundation into a distributed realtime system with strong authentication, rate limiting, privacy-safe observability and measurable reliability.
The Metaverse is online, but the deeper work is still ahead: building an open world protocol that other developers can inspect, test and extend.

Dynamic GitHub Actions Dropdowns — A Clever Workaround Made Reusable!

Fri, 16 Oct 2026 19:57:00 +0000

GitHub Actions doesn't natively support dynamically populated dropdown options for workflow_dispatch inputs. This can become a maintenance headache, especially when managing multiple packages in an npm monorepo.

I came across a clever workaround shared by @jeevan-vj, demonstrating how to dynamically maintain dropdown options by automatically updating GitHub Actions workflow YAML files.

Inspired by that approach, I built a reusable GitHub Actions workflow that uses a dedicated WORKFLOW_TOKEN to update workflow files and automatically discover publishable packages in an npm monorepo.

Here's what my implementation adds:

✅ Automatic package discovery — Detects publishable npm workspace packages without maintaining a hardcoded list.

✅ Multi-workflow synchronization — Keeps package dropdown options consistent across multiple GitHub Actions workflows.

✅ YAML formatting preservation — Uses Python and ruamel.yaml to update workflow configurations while preserving their structure and formatting.

✅ Automatic and manual execution — Supports synchronization on push as well as manual triggering when needed.

✅ Configurable token authentication — Accepts a workflow-token secret, allowing each consuming repository to choose its own secret name.

✅ No unnecessary commits — Updates workflow files only when package options actually change.

The end result?

My release, publish, restore, reset, and release-draft workflows now automatically stay synchronized with the publishable packages in my npm monorepo.

No more manually updating package dropdowns every time a package is added, renamed, or removed.

Even better, the solution is reusable across repositories, so I have adopted the same automation in other repos without duplicating the synchronization logic.

Original inspiration — @jeevan-vj:
https://github.com/jeevan-vj/iamjeevan/blob/master/data/blog/dynamically-populating-parameter-options-in-manual-github-action.md

My reusable workflow implementation:
https://github.com/moyarich/reusable-workflows/blob/main/.github/workflows/reusable_sync-release-package-choices.yml

Real-world implementation — Workspace Tools:
https://github.com/moyarich/workspace-tools/blob/main/.github/workflows/sync-release-package-choices.yml

Huge credit to @jeevan-vj for sharing the original workaround. It's a great example of how an open-source idea can evolve into a reusable solution that benefits other developers.

How are you managing dynamic inputs in your GitHub Actions workflows? Have you found another approach?

OAuth Is Not “Login.” I Found That Out the Hard Way in Production

Fri, 16 Oct 2026 17:09:26 +0000

Most people, including a lot of engineers, think OAuth means “the button that says Sign in with Google.” That’s not wrong exactly, but it’s like saying a car is “the thing with a steering wheel.” Technically true, misses basically everything that matters.

I want to actually walk through what OAuth is, why it exists, what it costs you, and the specific way it bit me in production, because the bug taught me more about OAuth than any tutorial did.

If I use a term that sounds like engineer-speak, I’ll flag it like this:

🚨 Jaggy jaggy alert: here’s the jargon, here’s what it actually means in plain English.

Let’s go.

What OAuth actually is
OAuth is not a login system. OAuth is a permission system. It answers one question: “can this app do this specific thing on my behalf, without me handing it my password?”

That’s it. That’s the whole idea.

Before OAuth, if you wanted some third party app to, say, post to your Twitter or read your Gmail contacts, you had exactly one option: give that app your actual Twitter or Gmail password. The app would log in as you, directly, using your real credentials. Which means that random app now has full access to your account forever, until you go change your password, and even then you’re trusting a third party company stored your password responsibly in the first place.

OAuth replaces that with a system where you never hand your password to the third party app at all. Instead, you briefly go to Google (or whoever), log in there, and Google hands the app a token, a kind of limited, revocable permission slip, instead of your actual credentials.

🚨 Jaggy jaggy alert: token. Think of it as a hotel key card instead of your house key. It opens specific doors, for a specific amount of time, and the hotel can deactivate it instantly without you having to change your actual house locks.

Could the internet work without OAuth
Technically, yes. Plenty of early 2000s apps worked exactly how I described above, you just gave them your password. It’s not that the internet collapses without OAuth.

What actually happens without it is worse in a slower, quieter way. Every app you connect to anything ends up holding a copy of your real password. A breach at any one of those apps means your actual account password leaks, not just some limited token. You can’t selectively revoke access (“let this app see my calendar but not send emails as me”) because the app either has your full password or it has nothing. And you can’t tell your bank, your email provider, or your social accounts “trust these five apps, not that one,” because from their point of view every connected app looks identical: someone logged in with your password.

So yes, the world could work without OAuth. It would just be a world where a breach anywhere is a breach everywhere, and “revoke access” isn’t really a button that exists.

A quick word on OAuth 1.0, the version before this one
Before the OAuth everyone actually uses today, there was OAuth 1.0, back around 2007. Same basic idea, token instead of password, but the mechanics were painful. Every single request had to be cryptographically signed by the app using a shared secret, a process involving HMAC-SHA1 and exact timestamps, before the server would trust it.

🚨 Jaggy jaggy alert: cryptographic. Just means “based on math that’s specifically designed to be really, really hard to fake or reverse without the right secret ingredient.” When something is described as cryptographically secure, breaking it isn’t impossible, it’s just so expensive and slow that nobody sane bothers trying.

🚨 Jaggy jaggy alert: signature (signing). Not a handwritten signature. A short code generated from the actual contents of a message plus a secret value, proving two things at once: this message wasn’t altered in transit, and it genuinely came from whoever holds that secret. Change one character of the message and the signature stops matching, which is exactly how tampering gets caught.

🚨 Jaggy jaggy alert: secret. A private value, basically a password, known only to the two parties who are supposed to trust each other. Whoever holds the secret can prove they are who they claim to be. Whoever doesn’t, can’t fake it convincingly. The whole system falls apart the moment that “secret” stops actually being secret.

🚨 Jaggy jaggy alert: HMAC-SHA1 signing. A way of mathematically stamping each request so the server can verify nothing was tampered with in transit, instead of just trusting an encrypted connection to do that job. You don’t need the math, just know it meant extra, fiddly cryptography homework on every single request.

In practice this meant developers spent real time fighting signature mismatches and clock sync issues instead of building their actual app. It worked, but it was miserable, and it still assumed every app had somewhere safe to keep a permanent secret. OAuth 2.0 showed up in 2012 and basically said, let’s just lean on HTTPS to protect the data in transit like everyone already does for everything else, and drop the per-request signing entirely.

🚨 Jaggy jaggy alert: HTTPS. The padlock in your browser’s address bar. It encrypts everything traveling between your browser and the server, so anyone snooping on the connection, a nosy coffee shop WiFi for instance, sees scrambled nonsense instead of your actual data. OAuth 2.0 leans on this instead of signing every request itself, which is most of why it’s so much simpler than OAuth 1.0.

Simpler for developers, which is exactly why OAuth 2.0 is the one that actually won and is what the rest of this article has been describing the whole time.

But that “every app has somewhere safe to keep a secret” assumption didn’t go away, it just got quietly inherited by OAuth 2.0. And a few years later, that assumption stopped being true for a huge chunk of apps. More on that in a minute.

How it actually works, the short version
Here’s the flow in plain English, using “Sign in with Google” as the example:

You click the button. The app sends you over to Google, not the other way around, you never type your Google password into the third party app’s own form. At Google, you log in (or you’re already logged in) and Google shows you a screen saying “This app wants to see your email and profile picture, allow it?” You say yes. Google then redirects you back to the app with a short-lived code. The app takes that code and, behind the scenes, trades it with Google for an actual token. From then on, the app uses that token to ask Google for the specific things you approved, nothing more.

🚨 Jaggy jaggy alert: scope. A scope is the specific permission being requested, like “read your email address” or “post on your behalf.” Apps are supposed to ask for the narrowest scope that does the job, though in practice a lot of apps ask for way more than they need, because broader scopes are easier to code against and nobody’s really checking.

Download the Medium app
🚨 Jaggy jaggy alert: redirect URI. This is the exact web address Google is allowed to send you back to after login. It’s locked in advance when the app registers with Google, specifically so an attacker can’t trick Google into sending your freshly issued code to some address they control instead.

That whole flow above quietly assumes one more thing: that the app trading the code for a token is a proper backend server, one that can hold onto a permanent, secret password of its own (a “client secret”) to prove to Google “yes, this request really is from the real app.” Which brings us to a problem nobody really had to think about back in 2012.

The problem nobody saw coming, apps that can’t keep a secret
Fast forward a few years and suddenly half the apps in the world are single-page apps running entirely in the browser, or mobile apps sitting on someone’s phone. A SPA’s entire codebase gets shipped straight to the user, anyone can pop open dev tools and read it line by line. A mobile app’s installer can be decompiled by anyone with a weekend and mild curiosity. There is nowhere to actually hide a secret in either case. Whatever you bake in, someone can pull back out, which is a uniquely bad punchline for something literally called a “client secret.”

🚨 Jaggy jaggy alert: client secret. A password-like value, known only to the real backend and Google, used to prove a token request is genuinely coming from the real app and not an imposter. Fine when there’s a server to keep it locked up. Useless the moment the “app” is just JavaScript running on a stranger’s laptop.

The fix is called PKCE, Proof Key for Code Exchange, and it’s pronounced “pixie,” which might be the single most delightful name in all of security engineering.

🚨 Jaggy jaggy alert: PKCE. Instead of a long-lived secret baked into the app, the app invents a random, one-time value right before login starts, the code verifier, and sends over a hashed version of it, the code challenge, when kicking off the flow. At the very end, when trading the code for a token, it has to produce that original, un-hashed value again. If someone intercepts the authorization code mid-flow, it’s useless to them, they don’t have the random value that generated it, and they can’t fake having it either.

🚨 Jaggy jaggy alert: hashing. A one-way math function that turns any input into a fixed-length scrambled output, and crucially, you can’t run it backwards to get the original input out again. Same input always produces the same output, which is exactly what makes it useful here, the app can prove “I know the original value” without ever saying the original value out loud until the very last step.

No permanent secret to leak, because there was never a permanent secret to begin with, just a fresh, disposable value that self-destructs after one use. Genuinely a cleaner idea than the thing it replaced, and these days it’s recommended for basically every app, server backed or not.

The trade-off nobody puts on the slide
Here’s what the “just add OAuth” tutorials gloss over: you’re trading simplicity for control, and that trade costs real engineering effort.

With a plain username and password system, you own the entire flow. One table, one hashing function, done. With OAuth, you now depend on a third party’s uptime, their API changing underneath you, tokens that expire and need refreshing, and an entire second system, your own backend, that has to track who’s logged in even though the actual login happened somewhere else entirely.

🚨 Jaggy jaggy alert: refresh token. Access tokens are deliberately short-lived, often an hour or less, so if one leaks, the damage window is small. A refresh token is a longer-lived, more sensitive token your backend holds onto quietly, so it can go get a fresh access token without making the user log in again every hour. Lose control of a refresh token and you’ve basically lost control of the session, so these need to be stored far more carefully than access tokens.

That’s the actual trade-off. You get: no passwords for you to store or leak, users can revoke access with one click from Google’s side, scoped and limited permissions, and single sign-on convenience. You pay for it with: more moving parts, a dependency on an external provider, token lifecycle management, and a genuinely larger surface area for things to quietly go wrong.

How I actually use this in production
On one platform I worked on, we had Google OAuth as a social login option sitting alongside our own email and password system, with our own JWT issued after either path succeeded.

🚨 Jaggy jaggy alert: JWT. Short for JSON Web Token. This is different from an OAuth access token, even though people use the words interchangeably all the time. A JWT is a signed, self-contained token your own backend creates, that your own backend can verify without a database lookup, basically a sealed, tamper-proof note saying “this is user 482, and I, the server, vouch for that.” OAuth tokens come from Google. JWTs, in this setup, come from us.

That distinction, two different kinds of token doing two different jobs, is exactly where things went wrong for me once.

What actually goes wrong: my real production bug
We had a production authentication bug where users were randomly getting logged out, or logged in as the wrong state, with no clear pattern. Intermittent, hard to reproduce locally, the worst kind of bug.

The root cause, once I actually traced it through production logs, was almost embarrassingly simple: the OAuth token coming back from the social login flow and our own JWT were both being stored under the same cookie key. Whichever one got written last silently overwrote the other. So depending on timing, a user’s session cookie might end up holding Google’s OAuth token instead of our own JWT, or vice versa, and the backend would try to verify it as the wrong type of token entirely, fail, and drop the session.

Nobody designed it that way on purpose. It happened because OAuth tokens and JWTs look similar enough, both are just strings you shove in a cookie, that it’s easy to treat them as interchangeable “auth stuff” instead of two distinct tokens with two distinct lifecycles and two distinct purposes. The fix was straightforward once I actually understood the distinction: give each token its own, clearly named cookie key, so they could never collide again.

The real lesson wasn’t the one line fix. It was that OAuth tokens and your own session tokens are not the same thing wearing a different hat, and treating them as the same thing is exactly the kind of assumption that only breaks in production, under real timing and real concurrent requests, never in your local dev environment where everything happens conveniently one at a time.

Other pitfalls worth knowing before you hit them
A few more ways this goes sideways, things I’ve seen or would actively watch for:

Storing tokens in localStorage instead of an httpOnly cookie, which means any cross-site scripting vulnerability on your site can just read the token straight out and walk away with the user's session.

🚨 Jaggy jaggy alert: httpOnly cookie. A cookie flagged so that JavaScript running in the browser can’t read it at all, only the browser itself can send it along with requests. It’s a deliberate wall against exactly the kind of theft I just described.

Mismatched redirect URIs between what’s registered and what the app actually sends, which just breaks login outright and is a surprisingly common first deploy bug when you move from a local dev URL to a real production domain and forget to register the new one.

Asking for way more scopes than you need, which makes users nervous at the consent screen and gives you a bigger blast radius if your token ever does leak.

Treating “the user has an OAuth token” as “the user is logged in” without actually verifying who they are. OAuth by itself is about authorization, what you’re allowed to do, not identity. Actual login on top of OAuth is a separate layer called OpenID Connect, and conflating the two is a genuinely common source of security bugs.

So, do you actually need it
If you’re building something small and internal, OAuth is probably overkill, a simple username and password system with properly hashed passwords is less to maintain and less to get wrong. OAuth earns its complexity the moment you want users to log in with an identity they already have elsewhere, or you want to act on a third party service on their behalf without ever touching their password. At that point, yes, deal with the token juggling, because the alternative, a world where “give the app your password” is normal, is worse in ways that only show up after something’s already gone wrong.

That’s the real shape of it. Not a login button. A permission system, with real trade-offs, that will absolutely find the one gap in your assumptions if you let two different tokens share a cookie key and call it a day.

Software Maintenance Cost: What to Budget Each Year

Fri, 16 Oct 2026 14:55:47 +0000

What software maintenance actually covers

"Maintenance" is a bucket for four different kinds of work. Knowing which is which makes the budget conversation much easier.

Corrective: fixing bugs found in production.
Preventive: security patches, framework and library upgrades, and clean-up that stops problems before they appear.
Adaptive: changing the system because the world changed, such as a new PHP version, a payment provider update or a browser change.
Perfective: small improvements and requests that make the product better but are not a new project.

New features are usually a separate budget. The trap is letting small feature requests quietly consume the maintenance hours that keep the system safe.

The recurring costs people forget

Beyond developer time, a live system has running costs:

Hosting and infrastructure: servers, databases, storage and bandwidth. My comparison of Laravel hosting options and cost shows how widely this varies.
Third-party services: email, SMS, payments, search, error tracking and AI APIs, often billed by usage.
Domain, SSL and licences for paid plugins or tools.
Backups and disaster recovery, including actually testing a restore.
Monitoring and alerting, so you hear about problems before customers do.
Security work: patching, access reviews and responding to vulnerability notices.

These costs are easy to estimate once you have a list. The surprise usually comes from usage-based services that grow with your traffic.

What drives maintenance cost up or down

I will not quote a percentage that pretends to fit every system. These factors decide where yours lands:

Age and technology. Systems on unsupported frameworks cost more to keep safe. This is the main link to rewriting or refactoring a legacy application.
Test coverage. Tested code is cheaper to upgrade because you can see what broke.
Number of integrations. Every third-party API is something that can change or fail.
Traffic and data volume. More load means more tuning and more infrastructure.
Quality of documentation. Missing knowledge makes every task slower.
Deployment process. Manual releases cost time and cause errors. A clean pipeline, as in my guide to zero-downtime deployments, lowers the cost of every future change.
Required uptime. A system that must always be available needs monitoring and on-call, which costs more.

Three ways to pay for maintenance

1. Retainer. A fixed monthly amount that covers a set number of hours or a defined scope. It suits systems with steady needs and gives you predictable budgeting and fast response.

2. Time and materials. You pay for hours as they are used. It is flexible for quiet systems, but costs are unpredictable and urgent issues compete with other clients' work.

3. Fixed-scope packages. Specific tasks, such as a framework upgrade or security audit, for a fixed price. These are good for planned one-off work on top of a baseline.

Many businesses use a small retainer for monitoring and urgent fixes, plus fixed-scope projects for bigger upgrades.

How to estimate your own maintenance budget

Use this simple worksheet:

List every running cost: hosting, services, domains, licences. Check last month's invoices for each.
Estimate routine hours: security patches, dependency updates, backup checks and monitoring review, per month.
Add a bug-fix allowance based on how often production issues have appeared so far.
Add planned upgrade work for the year, such as a major framework or PHP version.
Add a contingency for the unexpected, and treat it as real money.
Separate new features into their own budget so they do not eat maintenance time.

If the system is brand new and has no history, expect the early months to need more attention while real usage reveals problems. A short audit gives you a better starting point than a guess.

Why support dates decide your upgrade calendar

Two clocks set most of your planned maintenance: the language and the framework.

Language versions each get a limited support window, after which they stop receiving security fixes. PHP publishes these dates on its supported versions page.
Framework versions follow their own policy. Laravel documents which releases still receive bug and security fixes in its release notes and support policy.

What happens when you skip maintenance

Skipping maintenance does not save money. It moves the cost to later and makes it larger:

Known security holes stay open
Upgrades pile up until they become a risky big-bang project
Small bugs turn into outages
Only one person understands the system
Hosting and dependencies freeze because nobody dares touch them

A simple way to report maintenance value

Questions to ask before you agree a maintenance plan

What exactly is included, and what counts as a new feature?
What is the response time for an urgent production issue?
Who monitors the system, and how will I be alerted?
How are backups tested?
What happens to the plan if the system grows?
Can I see a report of what was done each month?

A clear answer to these is a better signal than the lowest price. My thoughts on why experience beats low-cost outsourcing apply here too.

Key takeaways

Maintenance is four kinds of work: fixing, preventing, adapting and improving.
Running costs go beyond developer time: hosting, services, backups, monitoring and security.
Age, tests, integrations and deployment process drive the price more than size alone.
Choose a retainer for steady needs, fixed-scope packages for planned upgrades.
Skipping maintenance makes problems larger and more expensive later.

If you want a clear maintenance budget for a system you already run, book a call and I will review it and tell you what it really needs.

Originally published at aqibjavaid.site.

Reverse Proxy or Transparent WAF? A Checklist for Web Teams

Fri, 16 Oct 2026 07:56:12 +0000

Reverse Proxy or Transparent WAF? A Checklist for Web Teams

Ask five questions before deployment: Can the origin be restricted? Who terminates TLS? Where are logs collected? How will bypasses be detected? What is the rollback path?

A reverse-proxy WAF makes these decisions explicit in the HTTP path. A transparent design may reduce topology changes but can require more network-level coordination. SafeLine’s documented deployment model is a self-hosted reverse proxy, so use the official guide to validate fit.

FAQ

Does SafeLine support both rule-based and semantic protection?

SafeLine documents layered application protection and dynamic protection features. Check the current product documentation for the exact behavior of each release.

Should I test changes before production?

Yes. Use a staged rollout, review logs, and verify legitimate requests before tightening policies.

Where can I check current Pro pricing?

Use the official 700K+ Installs Sale page for current campaign terms.

Is the sale renewable at the same price?

The campaign page says promotional pricing ends with the campaign; renewals return to regular pricing.

What evidence would you use to compare protection modes in your own traffic?

Read the full guide on dev.to:

📖 Reverse Proxy or Transparent WAF? A Checklist for Web Teams

SafeLine Pro 700K+ Installs Sale

Reverse Proxy or Transparent WAF? A Checklist for Web Teams

Fri, 16 Oct 2026 06:23:12 +0000

Reverse Proxy or Transparent WAF? A Checklist for Web Teams

Ask five questions before deployment: Can the origin be restricted? Who terminates TLS? Where are logs collected? How will bypasses be detected? What is the rollback path?

FAQ

Does SafeLine support both rule-based and semantic protection?

SafeLine documents layered application protection and dynamic protection features. Check the current product documentation for the exact behavior of each release.

Should I test changes before production?

Yes. Use a staged rollout, review logs, and verify legitimate requests before tightening policies.

Where can I check current Pro pricing?

Use the official 700K+ Installs Sale page for current campaign terms.

Is the sale renewable at the same price?

The campaign page says promotional pricing ends with the campaign; renewals return to regular pricing.

What evidence would you use to compare protection modes in your own traffic?

Read the full guide on dev.to:

📖 Reverse Proxy or Transparent WAF? A Checklist for Web Teams

SafeLine Pro 700K+ Installs Sale

Reverse Proxy WAF vs Transparent Bridge: A Deployment Decision

Fri, 16 Oct 2026 06:22:39 +0000

Reverse Proxy WAF vs Transparent Bridge: A Deployment Decision

The main difference is how clearly the WAF participates in the application traffic path. A reverse proxy is an explicit HTTP gateway. A transparent bridge is designed to minimize visible topology changes.

Reverse proxy trade-offs

The explicit hop can make routing and policy ownership easier to reason about. It also means the team must plan certificates, upstream connectivity, health checks, and origin restrictions.

Transparent designs

A transparent design may fit networks where changing endpoints is difficult, but it can make troubleshooting and application context more dependent on the surrounding network.

SafeLine documents the reverse-proxy model, application onboarding, and web attack protections. Compare that documented model with your existing topology before choosing a deployment.

FAQ

Does SafeLine support both rule-based and semantic protection?

SafeLine documents layered application protection and dynamic protection features. Check the current product documentation for the exact behavior of each release.

Should I test changes before production?

Yes. Use a staged rollout, review logs, and verify legitimate requests before tightening policies.

Where can I check current Pro pricing?

Use the official 700K+ Installs Sale page for current campaign terms.

Is the sale renewable at the same price?

The campaign page says promotional pricing ends with the campaign; renewals return to regular pricing.

What evidence would you use to compare protection modes in your own traffic?

Read the full guide on dev.to:

📖 Reverse Proxy WAF vs Transparent Bridge: A Deployment Decision

SafeLine Pro 700K+ Installs Sale

Gating Cloudflare Worker Deployments With CI

Thu, 15 Oct 2026 17:16:41 +0000

Automated build systems often deploy code immediately on merge without waiting for external quality gates to finish.

Quick solution:
Configure GitHub branch protection rules to require your test and lint status checks before merging, and ensure your Workers Builds configuration targets that protected production branch as detailed in the Build branches - Cloudflare Workers docs.

name: CI Validation
on:
  pull_request:
    branches: [main]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Tests
        run: npm test

Understanding the Deployment Automation Boundary

When a pull request passes all status checks and merges to the main branch, teams often assume that deployment waits for every upstream validation. In reality, automated continuous integration and platform builds run in decoupled systems. A green test suite or static analysis badge on a pull request does not guarantee that Cloudflare Workers waited for that specific result before deploying the latest script.

Understanding where validation occurs is essential for preventing regressions from reaching users. Pull request checks evaluate proposed changes in an isolated environment. However, platforms like Cloudflare Workers Builds can trigger automatically upon receiving a push to a configured branch, independently of whether external quality gates have finished or passed on that exact ref.

GitHub allows you to enforce requirements via protected branches. According to the About protected branches - GitHub Docs, repository owners can require specific status checks to pass before merging any code. If a required check fails or is missing from the protection rules, a developer can merge code that has not been fully verified.

Integrating SonarQube Cloud Quality Gates

Static analysis tools provide deep insights into code smells, vulnerabilities, and duplications. As detailed in the Introduction to quality gates - SonarQube Cloud, a quality gate defines the pass or fail criteria for a project based on clean code metrics.

A common failure mode occurs when Sonar analysis runs post-merge rather than as a blocking pull request check. If the analysis executes only after code lands on the main branch, it reports the state of the codebase after deployment has already begun. To act as a true quality gate, the Sonar analysis status must be reported back to the pull request and registered as a required status check in your version control platform.

Configuring Workers Builds and Branch Triggers

Cloudflare Workers can build and deduplicate automatically from connected Git repositories. Reviewing the CI/CD - Cloudflare Workers docs and the Configuration - Cloudflare Workers docs helps clarify how build triggers map to repository branches.

Integration Option	When to Use It	Trade-off or Failure Mode	Recommendation
Automatic Branch Builds	Fast iterations and continuous delivery for single-service projects.	Triggers immediately on push; external analysis might still be running.	Use with strict branch protection rules.
Manual or API Deployment	Strict regulatory requirements or complex multi-service release sequencing.	Requires maintaining external CI pipeline tokens and credentials.	Use when deployment must wait for external sign-offs.

Tracing Release Evidence Across Systems

Proper release engineering requires tracing a single release across multiple disconnected platforms. You should inspect the commit SHA of the pull request, verify the exact merge commit hash, confirm the corresponding Cloudflare Workers Build version, check the Sonar branch analysis report, and perform a live route check. If deployment starts immediately on merge, a later main-branch scan is not a pre-deployment gate unless another system explicitly waits for it.

Do not infer correct ordering from two separate green badges. Inspect actual branch-protection rules, Workers Builds settings, and the deployed version directly. Record the exact commit SHA for each event and report any missing gate as a configuration gap in your infrastructure.

Release Verification and Execution Checklist

Use this execution checklist to confirm that your validation steps actually gate your production releases:

Verify branch protection rules: Confirm that pull requests cannot be merged without passing unit test and lint checks.
Inspect status check mapping: Ensure that the exact check names in GitHub match the jobs defined in your CI pipeline configuration.
Check analysis timing: Verify that SonarQube Cloud evaluates the pull request commit rather than only scanning the main branch post-merge.
Audit build triggers: Inspect your Cloudflare Workers Builds settings to ensure production deployments listen only to the protected main branch.
Perform live route testing: Execute a targeted curl command against your production worker route to verify that the newly deployed version hash matches the expected release commit.

From “Why Is This Pod Pending?” to `HTTP/1.1 200 OK`: My NGINX Gateway Fabric Lab

Thu, 15 Oct 2026 15:15:48 +0000

A beginner-friendly Kubernetes networking story about Gateway API, Services, HTTPRoutes, DNS, a few misleading errors, and the joy of finally seeing a successful response.

The mission

The goal of this exercise was simple to say and slightly less simple to do: deploy NGINX Gateway Fabric in my Kubernetes cluster, create a small NGINX web application, and make the application accessible through a Kubernetes Gateway and HTTPRoute.

The intended request path was:

My Kali terminal
      |
      | HTTP request to NodePort :30080
      v
NGINX Gateway
      |
      | HTTPRoute matches path /
      v
Kubernetes Service: frontend-svc:80
      |
      v
Pod: frontend-app (NGINX listens on port 80)
      |
      v
HTTP/1.1 200 OK

Think of it like sending a parcel. The Gateway is the front desk, the HTTPRoute is the instruction that says where the parcel should go, the Service is the stable address for the destination, and the Pod is the actual person receiving it.

This post walks through what I did, the problems I hit, and how I worked through them. The useful lesson was not just getting the page to load. It was learning to test one layer at a time instead of randomly changing things until Kubernetes takes pity on me.

1. Installing the Gateway API building blocks

I started in my nginx-gateway-fabric working directory and applied the standard Gateway API CRDs:

kubectl apply -k config/crd/gateway-api/standard

CRD means CustomResourceDefinition. Kubernetes comes with built-in resources such as Pods, Deployments, and Services. CRDs teach Kubernetes about additional resource types. The Gateway API CRDs add resources such as GatewayClass, Gateway, and HTTPRoute.

I then installed the NGINX Gateway Fabric-specific CRDs and the NodePort deployment for version 1.6.1:

kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v1.6.1/deploy/crds.yaml

kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v1.6.1/deploy/nodeport/deploy.yaml

The NodePort deployment gave me a way to reach the Gateway using a port on a Kubernetes node. In this lab, HTTP was exposed on port 30080.

Problem 1: Kustomize timed out while fetching the CRDs

My first attempt to fetch the Gateway API CRDs directly from GitHub using kubectl kustomize failed with a Git fetch timeout. Because the command failed to produce the manifests, the following kubectl apply command reported that no objects had been passed to it.

That second message was a consequence of the first failure, not a separate mystery.

What I did: I worked with the repository locally and applied the Kustomize directory from the checked-out source:

kubectl apply -k config/crd/gateway-api/standard

Lesson: When a command downloads configuration and then applies it, check the download/rendering step first. If it produces nothing, kubectl apply cannot magically apply it.

2. The Gateway Pod was Pending

After applying the NGINX Gateway Fabric deployment, I checked the Pod:

kubectl get pods -n nginx-gateway

At first, it showed Pending. My first attempt to describe it failed too:

kubectl describe pod nginx-gateway-55f96886f7-j4kzq

The Pod was not found because I had left out its namespace. It lived in nginx-gateway, not the default namespace.

The corrected command was:

kubectl describe pod nginx-gateway-55f96886f7-j4kzq -n nginx-gateway

The important event was:

0/2 nodes are available: 2 node(s) had untolerated taint(s).

What does that mean?

A taint is a rule on a Kubernetes node that says, in effect, “Don't schedule Pods here unless they tolerate this condition.” A toleration is a matching permission in a Pod's specification.

The Gateway Pod was not necessarily broken. Kubernetes was saying it could not find a node where the Pod was allowed to run. I also saw that k8s-worker1 briefly became NotReady before returning to Ready.

What I learned: Check the node status and the Pod's scheduling events before changing the deployment. Taints may be protecting the cluster for a reason, so don't remove them blindly just to make a Pod start.

Useful commands:

kubectl get nodes -o wide
kubectl describe pod nginx-gateway-55f96886f7-j4kzq -n nginx-gateway
kubectl describe node k8s-worker1

Eventually, the Gateway Pod was scheduled to k8s-worker1 and progressed toward running.

3. Creating the frontend application

Next, I created a simple NGINX Pod:

kubectl run frontend-app --image=nginx --port=8080

One subtle detail caused a later problem: --port=8080 does not reconfigure NGINX to listen on port 8080. It records container-port metadata in the Pod specification. The NGINX image still listens on port 80 by default.

I exposed the Pod as a Service:

kubectl expose pod frontend-app \
  --name=frontend-svc \
  --port=80 \
  --target-port=8080

A Service gives a stable network address to a group of Pods. This is useful because Pods can be replaced, and their IP addresses can change. Clients can keep talking to the Service instead of tracking individual Pod IPs.

I checked the Service and its endpoints:

kubectl describe svc frontend-svc -n default
kubectl get endpoints frontend-svc -n default

Initially, the endpoint showed port 8080. That looked consistent with the Service configuration, but it was not consistent with the port NGINX actually used.

Problem 2: The Service pointed to the wrong port

I tested NGINX from inside its own Pod:

kubectl exec frontend-app -n default -- \
  curl -I --max-time 5 http://127.0.0.1:8080/

It failed:

curl: (7) Failed to connect to 127.0.0.1 port 8080

Instead of guessing, I tested the default NGINX port:

kubectl exec frontend-app -n default -- \
  curl -I --max-time 5 http://127.0.0.1:80/

That returned:

HTTP/1.1 200 OK
Server: nginx/1.31.7

There was the clue. NGINX was alive; I had simply sent it to the wrong port.

Fix: I changed the Service's targetPort to 80:

kubectl patch svc frontend-svc -n default \
  -p '{"spec":{"ports":[{"port":80,"targetPort":80}]}}'

Then I checked it again:

kubectl describe svc frontend-svc -n default

The endpoint now pointed to the Pod on port 80.

Lesson: The Service's port is the port clients use to reach the Service. The targetPort is the port on the backend Pod that receives the traffic. Those values can be different, but targetPort must match the application that is actually listening.

4. Building the HTTPRoute

The next step was to tell the Gateway where to send incoming HTTP requests. I created an HTTPRoute called frontend-route.

The important parts of the route were:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: frontend-route
  namespace: default
spec:
  parentRefs:
    - name: nginx-gateway
      namespace: nginx-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: frontend-svc
          port: 80

In plain English:

Attach this route to the nginx-gateway Gateway in the nginx-gateway namespace.
Match requests whose path starts with /.
Forward matching requests to the Service named frontend-svc on port 80.

I checked the result with:

kubectl describe httproute frontend-route -n default

The good news was:

Type:    Accepted
Status:  True

Type:    ResolvedRefs
Status:  True
Message: All references are resolved

These conditions mean the Gateway accepted the route and Kubernetes could resolve its references. They are useful signals, but I still needed to test actual traffic.

Problem 3: The HTTPRoute could not find the backend Service

My first route was in the nginx-gateway namespace and referred to frontend-svc. Kubernetes returned:

BackendNotFound
spec.rules[0].backendRefs[0].name: Not found: "frontend-svc"

I had created frontend-svc in the default namespace. A Service reference in an HTTPRoute is normally resolved in the route's own namespace. Giving the Service the correct name was not enough; the namespace mattered too.

I edited the route YAML and placed the working frontend-route in default, while keeping its parentRefs pointing to the Gateway in nginx-gateway.

That fixed the reference. The route then reported both Accepted=True and ResolvedRefs=True.

Lesson: In Kubernetes, always check both the resource's name and namespace. Two resources can share a name in different namespaces, and a route can be accepted while still having an unresolved backend reference.

5. Testing the Service before testing the Gateway

Before troubleshooting the Gateway, I tested the Service directly. First, I checked that CoreDNS was running:

kubectl get pods -n kube-system
kubectl get svc -n kube-system kube-dns

Then I created a short-lived diagnostic Pod:

kubectl run dns-test --image=busybox:1.36 \
  --restart=Never --command -- sleep 300

kubectl wait --for=condition=Ready pod/dns-test \
  -n default --timeout=60s

I inspected its DNS settings:

kubectl exec dns-test -- cat /etc/resolv.conf

The file showed the cluster DNS Service IP, 10.96.0.10. I then tested DNS resolution:

kubectl exec dns-test -- \
  nslookup kubernetes.default.svc.cluster.local

kubectl exec dns-test -- \
  nslookup frontend-svc.default.svc.cluster.local

Both names resolved. The frontend Service resolved to 10.104.158.51.

Problem 4: One curl test could not resolve the Service name

An earlier temporary curl Pod returned:

curl: (6) Could not resolve host: frontend-svc.default.svc.cluster.local

That was confusing because the DNS Service and CoreDNS Pods looked healthy.

Rather than restarting DNS immediately, I used a separate diagnostic Pod to test the basics. Its nslookup commands successfully resolved both the Kubernetes API Service and frontend-svc. Then I tested the Service itself:

kubectl exec dns-test -- wget -S -O- \
  http://frontend-svc.default.svc.cluster.local/

This returned the default NGINX welcome page and an HTTP/1.1 200 OK response.

What I can conclude: DNS resolution and Service connectivity worked from the diagnostic Pod. The earlier curl test failed, but the evidence did not establish its exact cause. The right move was to reproduce the test and inspect the layers separately rather than assume CoreDNS was broken.

6. The Gateway was programmed, but my first request returned a 500

I checked the Gateway and its Service:

kubectl get svc -n nginx-gateway
kubectl describe gateway nginx-gateway -n nginx-gateway

The Service showed:

TYPE       NodePort
PORT(S)    80:30080/TCP,443:30081/TCP

The Gateway status showed Accepted=True and Programmed=True. That meant the Gateway configuration had been accepted and programmed, but an end-to-end test was still needed.

I checked the route in nginx-gateway and discovered that a second route with the same name existed there. Unlike the working route in default, this duplicate route still referenced a backend Service that was not in its namespace. Its status included BackendNotFound.

I deleted the broken duplicate route:

kubectl delete httproute frontend-route -n nginx-gateway

I kept the working route in default.

My first request to the NodePort returned 500 Internal Server Error. After removing the broken duplicate route, I tried again. The request succeeded.

The logs show that the bad duplicate route existed and that the response changed after I removed it. That makes the duplicate route a strong suspect in the earlier failure, although the response alone does not prove it was the only cause.

Problem 5: Two routes, one name, different namespaces

Kubernetes permits resources with the same name in different namespaces. That is not automatically an error, but it made this troubleshooting session confusing: one frontend-route was correctly configured, while the other had an unresolved backend.

Lesson: When a route behaves unexpectedly, check all routes across namespaces:

kubectl get httproute -A

Don't check only the namespace where you expect the route to live.

7. The final test: success through the NodePort

I checked the node addresses:

kubectl get nodes -o wide

The worker node's internal IP was 10.0.2.4. The Gateway's HTTP NodePort was 30080, so I tested:

curl -i --max-time 10 http://10.0.2.4:30080/

At first, I mistakenly typed the placeholder literally:

curl -i --max-time 10 http://WORKER_IP:30080/

The terminal returned Could not resolve host: WORKER_IP. The shell did exactly what I asked: it treated WORKER_IP as a hostname. Placeholders must be replaced with actual values before running a command.

The final request returned:

HTTP/1.1 200 OK
Server: nginx
Content-Length: 896

And, below the headers, the familiar “Welcome to nginx!” HTML page appeared.

That was the end-to-end confirmation I wanted: a request from my Kali terminal entered through the node's NodePort, reached the NGINX Gateway, matched the HTTPRoute, and was forwarded to the frontend Service and Pod.

Every disk-alert script I had piped into a command this machine does not have

Thu, 15 Oct 2026 13:00:00 +0000

The standard bash alerting pattern ends the same way everywhere you read it: pipe the finding into mail. So I checked what my own box would actually do with that line.

$ type -a mail mailx sendmail
mail not found
mailx not found
sendmail not found

Nothing. No MTA, no mailutils, no fallback. Any alert script written the usual way would have run its check correctly, found the problem correctly, piped the message into a command that does not exist, and told me nothing.

This fails quieter than it sounds. Under set -euo pipefail you at least get a non-zero exit somewhere a cron wrapper might notice. Without it, the pipeline's exit status is the last command's, cron mails you nothing because cron's mail also needs an MTA, and the script looks like it ran fine.

Check for the binary before you rely on it, not after an incident. And on a box that does have mail, exit 0 still only means the message reached a local MTA — on a host with no outbound relay it lands in a spool file under /var/mail that nobody reads. Confirm real delivery once by tailing /var/log/mail.log. Cron's minimal PATH is a second reason to test from inside cron rather than your login shell.

The msmtp relay config and the rate-limiting guard are in the bash email alert walkthrough.

CloudPloy: Deploy anywhere from your AI

Wed, 14 Oct 2026 23:17:09 +0000

CloudPloy: Deploy anywhere from your AI tool.

Add an API key. Your agent deploys from Claude Code, Cursor, or any MCP client. Bring your own Ubuntu/AWS server or provision Hetzner/DigitalOcean/AWS at cost. Flat plan for the control plane; compute at the provider's rate.

Pricing: Free forever (1 small server, 1 app). Starter $9.99 / Pro $19 / Scale $39.

How to get alerted when someone changes your DNS records

Wed, 14 Oct 2026 14:40:51 +0000

TL;DR. Most DNS changes do not take your site down. A new MX record or an edited SPF record usually still resolves, so your uptime checks stay green. To catch these changes, list every record a name should have. Then get an alert when the live answer is different in any way. The original post has the same text with a short FAQ at the end.

A DNS change does not look like an outage

Uptime monitors wait for something to break. A DNS change often breaks nothing you can see.

Here is an example. Someone adds a second MX record to your domain. MX records tell other servers where to deliver your email. Maybe it was a test of a new email provider, and nobody removed it. Maybe someone stole the password to your DNS account.

Each MX record has a priority number, and senders try the lowest one first. If the new record has a lower number than yours, your email now goes to another server first. Your website works, your checks are green, and nobody notices for weeks.

Other changes work the same way:

Someone changes your nameservers at the registrar. Now a different server answers for your whole domain.
Someone edits your SPF record during a migration. SPF lists the servers that may send email as you, and your newsletter tool is no longer on it.
Someone deletes your _dmarc record while cleaning up old records.
Someone changes your CAA record. Now a different certificate authority can issue certificates for your domain.

These changes can leave your DNS lookups and website checks passing.

This happens in real attacks

In January 2019, the US Cybersecurity and Infrastructure Security Agency (CISA) published Emergency Directive 19-01. Attackers had changed DNS records on several US government domains.

They obtained credentials for a DNS account. They changed A, MX and NS records to point to their own servers. Then they could read web and email traffic as it passed through.

CISA's first order to the agencies was simple: check your public DNS records, and make sure each one points to the right place. Start with NS records.

A check like that tells you the records were right on one day. A monitor does the same check every few minutes, so you find out on the day of the change.

Why "does it resolve?" is not enough

Most DNS checks ask one of two questions.

The first is "Does the name resolve?" This catches a deleted record or an expired domain. It does not catch a changed record, because a changed record still resolves.

The second is "Does the answer contain this value?" For example, you enter your mail server, and the check fails if it disappears. This is better. But the check passes as long as your value is somewhere in the answer. So it misses:

an extra MX or NS record next to yours;
an SPF record changed around the part you check;
one A record replaced, when you have several.

To catch these, the check must know every record that should be there. Then it can fail on anything else.

Watch the exact record set

An exact record set is a list of every record that one name should have, for one record type. The check reads the live answer and compares it with your list:

A record in your list but not in the answer is missing.
A record in the answer but not in your list is unexpected.
A changed record shows up as both. The old value is missing and the new value is unexpected.

Any difference fails the check. The order of the records does not matter, and neither does the TTL.

To write the list, start from the records your domain has now. The DNS lookup tool shows them from Cloudflare and Google side by side. Check that every record is one you expect. If the two resolvers disagree, a change is still spreading, so wait until they match.

In Uptimepage, create a DNS monitor and choose exact set as the answer match. Write one record per line, the way a DNS answer shows it:

10 mail.example.com.
20 alt.example.com.

Say someone adds a third MX record. The next check fails, and the result shows what changed:

{
  "domain": "example.com",
  "record_type": "MX",
  "missing": [],
  "unexpected": ["5 mx.attacker.example."],
  "answers": ["10 mail.example.com.", "20 alt.example.com.", "5 mx.attacker.example."],
  "expected_contains": null,
  "matched": false
}

The new MX record has priority 5, lower than yours, so senders try it first.

The check ignores small differences that would only cause false alarms:

Upper and lower case in names do not matter, so 10 Mail.Example.com matches 10 mail.example.com.
The dot at the end of a name is optional.
Numbers and IP addresses are compared by value, so 2001:0db8::1 matches 2001:db8::1.

TXT values must match exactly, including case. In SPF and DKIM, every character counts.

Enter only the record value, without the name, TTL or type. For TXT records, you can paste quoted strings from your zone file, with all parts on one line.

Which records to watch

You do not need this for every record. Start with the ones where a quiet change hurts the most.

Record	Name	What it controls
NS	`example.com`	Which servers answer for your whole domain
MX	`example.com`	Where your email goes
TXT	`example.com`	Your SPF record: who may send email as you
TXT	`_dmarc.example.com`	Your DMARC policy: what receivers do with fake email from your domain
TXT	`selector._domainkey.example.com`	Your DKIM public key, which receivers use to check your email signatures
CAA	`example.com`	Which certificate authorities may issue certificates for you
A, AAAA	your server hostnames	Only if your servers have fixed IP addresses

Your main domain often has other TXT records too, such as verification records from other services. Put all of them in the list. A new one means someone connected a new service to your domain, and you want to know about that.

One monitor watches one name and one record type. Start with six monitors, one for each of the first six rows. Then add one for each other DKIM selector you use, and one for each fixed A or AAAA record set. A hostname with both an A and an AAAA record needs two monitors.

If you set up only one, make it NS. CISA put NS first too. Whoever changes your NS records controls all your other records.

Query your own nameserver

Normally a DNS lookup goes through a resolver, such as 1.1.1.1 or the one from your internet provider. The resolver keeps each answer for a while. This time is called the TTL. If your MX record has a TTL of one hour, the resolver can show the old answer for about an hour after a change. Sometimes it is longer: a resolver that cannot reach your nameserver may keep serving the old answer until it can.

You can skip this delay. Set the monitor's resolver to the IP address of your own nameserver, the one that holds your real records (your authoritative nameserver). Then every check sees your records as they are right now.

There is one exception: NS records. If someone changes your nameservers at the registrar, your own nameserver does not know. It still answers with the old, correct list. So watch NS through a public resolver, such as 1.1.1.1 or 8.8.8.8. A public resolver follows the registrar, like your visitors do. You can check what a public resolver sees for any name before you create the monitor.

When not to use an exact set

Some records change on purpose. An exact set on them only gives you false alarms.

Names behind a CDN or a load balancer get their IP addresses from a pool. The answer can change over time and depending on where you query from. I wrote about this in why DNS returns different IP addresses. An exact A set on such a name would fail all the time. Watch the CNAME record that points to the CDN instead.

SOA records change every time you edit your zone, because their serial number goes up. _acme-challenge TXT records come and go with each certificate renewal. Do not use an exact set for these either.

Keep the list next to your DNS

If you manage DNS with Terraform, put the monitor in the same repository:

resource "uptimepage_target" "mx" {
  name     = "mx example.com"
  interval = 300

  check = {
    type = "dns"
    dns = {
      domain           = "example.com"
      record_type      = "MX"
      expected_records = ["10 mail.example.com.", "20 alt.example.com."]
    }
  }
}

Now a pull request that changes an MX record must change the monitor too. An MX change that nobody reviewed fails the check. You need Terraform provider 0.15.0 or later. Read more in monitoring as code. All DNS options are in the DNS monitor docs.

Where Uptimepage fits

I build Uptimepage, an uptime monitor that checks your services from several regions and alerts you when one fails. The exact set match above is part of its DNS monitor. A DNS change alerts you once the failure is confirmed: by default two failed checks in a row, from more than half of the regions. The result names the missing and the unexpected records, so you see the change without running dig yourself. It is open source under AGPL, so it can also run on your own server.

Whatever you use, look up your NS records today and check that every name in the answer is one you expect.

Now the question for the comments: has a DNS record on one of your domains ever changed without anyone noticing? I would like to hear how you found out, and how long it took.

Why AI Coding Agents Need a Feature Flag Firewall: Inside Spotify Confidence MCP + ThumbGate

Wed, 14 Oct 2026 14:18:03 +0000

Why AI Coding Agents Need a Feature Flag Firewall: Inside Spotify Confidence MCP + ThumbGate

Spotify's experimentation team just announced "A New Chapter for Confidence" (October 14, 2026), launching Confidence Agent, Session Recordings, and official MCP servers for Claude Code, Cursor, and Codex (https://mcp.confidence.dev/mcp/flags).

This is a massive leap forward for agentic software development: instead of a human engineer manually creating flags, setting rollout percentages, and monitoring A/B test sample sizes in a web dashboard, an AI coding agent can do it directly through natural language in their editor.

But giving autonomous AI agents direct write access to production feature flags introduces an entirely new catastrophic blast radius.

The Three Failure Modes of Autonomous Flag Mutation

Premature 100% Rollouts:
An autonomous agent refactors a checkout component or backend service. To "verify" its work, it calls set_rollout to 100% in production before automated integration tests pass or without running a canary slice.
A/B Experiment Pollution:
An agent modifies user variant allocations, sample sizes, or targeting criteria mid-experiment, destroying statistical validity and costing the business tens of thousands in unrecoverable experimentation bias.
Destructive Flag Purges:
During automated dead-code cleanup, an agent purges a legacy feature flag because it cannot find references in one repository, unaware that a mobile client or upstream microservice still relies on the flag schema.

The Architecture: Pre-Action Feature Flag Diodes

At ThumbGate, we built the Pre-Action Infrastructure Firewall for Spotify Confidence MCP.

Before an agent executes any mutation via https://mcp.confidence.dev/mcp/flags, ThumbGate intercepts the tool call in sub-0.5ms and enforces deterministic governance diodes:

[ AI Coding Agent: Claude / Cursor / Codex ]
                  â
                  â¼
       [ Proposed MCP Tool Call ]
       (e.g., set_rollout: 100%, flag: "new-checkout")
                  â
                  â¼
   [ ThumbGate Infrastructure Firewall (<0.5ms) ]
   âââ gate:confidence:rollout_ceiling
   â   ââ Is rollout > 10%? âââº [BLOCK: Require Human Sign-Off]
   âââ gate:confidence:test_proof_required
   â   ââ Tests failing? ââââââº [BLOCK: Require Same-Turn Test Proof]
   âââ gate:confidence:protect_active_experiment
       ââ Active A/B test? ââââº [BLOCK: Experiment Allocation Locked]
                  â
                  â¼ (If Verified & Bounded)
    [ Allowed -> Passthrough to Spotify Confidence API ]
                  â
                  â¼
         [ mcp.confidence.dev ]

Key Pre-Action Diodes Implemented

gate:confidence:rollout_ceiling (Max 10% Autonomous Canary):
Autonomous agents are hard-capped at 10% canary traffic. Expanding beyond 10% requires an explicit operator confirmation token.
gate:confidence:test_proof_required:
Blocks creating or updating feature flags while the repository's CI or local test suite is failing.
gate:confidence:protect_active_experiment:
Protects live A/B tests against autonomous parameter mutation.
gate:confidence:prevent_flag_deletion:
Prevents deleting flags in production environments. Flags must follow an explicit deprecation and archive lifecycle.

How to Protect Your Agentic Tooling

If your engineering organization is adopting agentic workflows, MCP servers, and autonomous coding tools like Cursor, Claude Code, or Codex, never grant unconstrained write access to production infrastructure without a pre-action firewall.

ð Install the ThumbGate Pre-Action Firewall for your team: https://thumbgate.ai/go/pro ($19/mo)
ð Deploy a verified feature-flag safety diode on your production stack this week: Book a Workflow Hardening Diagnostic ($499).

Alternatives to the DevLabs Pull Request Merge Conflict Extension (2026)

Wed, 14 Oct 2026 13:44:34 +0000

title: "Alternatives to the DevLabs Pull Request Merge Conflict Extension (2026)"
published: true
description: "Microsoft's DevLabs merge conflict extension for Azure DevOps is no longer maintained. An honest comparison of the alternatives: resolving locally, your IDE, and a maintained extension."
tags: azuredevops, git, devops, productivity
canonical_url: https://jr-devtools.com/guides/devlabs-merge-conflict-extension-alternatives/

cover_image: https://jr-devtools.com/merge-conflict-resolver/screenshot-editor.png

Originally published on jr-devtools.com.

For years, the free Pull Request Merge Conflict Extension from Microsoft DevLabs (ms-devlabs.conflicts-tab) was how teams resolved conflicts without leaving Azure DevOps. It is no longer maintained. Here's what's going on and what you can use instead.

Disclosure: we make Merge Conflict Resolver, one of the options below. We've tried to keep this comparison fair; corrections are welcome at support@jr-devtools.com. Last checked: October 2026.

Is the DevLabs extension still supported?

No. Its last update was in November 2023, and Microsoft wrote in the extension's Q&A that “there are no current development resources for this extension”. Questions like “Is this even supported anymore?” go unanswered. Since 2024 its reviews average about 1.6 out of 5. The most reported problems:

the tab freezes or keeps “taking longer than expected to load”, and work is lost when it crashes;
non-ASCII characters (ä, é, č, Hebrew, Chinese …) are changed, especially in UTF-8 and ANSI files;
you can only take a whole file from one side, not choose per conflict or keep both;
the resolution lives only in the pull request's merge commit, so reviewers and builds never see it;
large files such as package-lock.json, or many conflicts, make it hang.

It still works for some teams on simple conflicts, so there's no need to remove it in a hurry. But if you run into the problems above, there won't be a fix.

The alternatives

1. Resolve locally with Git (free)

Merge the target branch into your source branch, resolve in your editor, and push. See our step-by-step guide.

Good: free, always works, you can build and test before pushing. Less good: you need the repository cloned and a working Git setup; slow for a two-line fix or for someone else's pull request.

2. Your IDE's Git integration (free)

Visual Studio, VS Code and JetBrains IDEs all have 3-way merge editors. In Visual Studio, check out the source branch, merge the target branch from Git → Manage Branches, and resolve in the merge editor.

Good: comfortable editor with IntelliSense. Less good: same as locally: you need a clone, and it's per developer, not something a reviewer can do from the pull request.

3. Merge Conflict Resolver (free plan, Pro from $12 per organization per month)

Merge Conflict Resolver is an independent, actively maintained extension that adds a Resolve conflicts tab to pull requests, built to fix the problems above:

3-way editor (Monaco, the editor of VS Code) with Accept target / source / both per conflict;
drafts are saved in the browser, so a reload or crash never loses work;
UTF-8, UTF-8 with BOM, UTF-16 and Windows-1252 files and their line endings are written back byte for byte;
edit/delete, renames, binary files, Git LFS and submodules each get a clear choice;
stays responsive with hundreds of conflicts and multi-megabyte lockfiles;
Pro: commit the resolution to the source branch, a summary comment on the pull request, bulk and whitespace auto-resolve.

Good: no clone needed, anyone on the pull request can resolve. Less good: Azure DevOps Services only for now (not Azure DevOps Server); Pro is paid. The free plan covers 10 pull requests per organization per month, and every organization gets a 14-day Pro trial.

Side by side

DevLabs extension	Locally / IDE	Merge Conflict Resolver

Switching from the DevLabs extension

Both extensions can be installed side by side, so you can try before you remove anything. An organization admin installs Merge Conflict Resolver from the Visual Studio Marketplace; there's nothing to configure, and the 14-day Pro trial starts the first time someone opens the tab.

Not affiliated with Microsoft. “Azure DevOps”, “Excel” and “Office” are trademarks of Microsoft.

One empty variable, one error, one wrong answer, exit 0

Wed, 14 Oct 2026 13:00:00 +0000

I wrote four lines to capture a bash error for a guide. One empty variable, tested twice:

$ cat queue-check.sh
#!/bin/bash
count=""
if [ $count -eq 0 ]; then echo "queue empty"; fi
if [[ $count -eq 0 ]]; then echo "queue empty ([[ ]])"; fi

$ bash queue-check.sh
queue-check.sh: line 3: [: -eq: unary operator expected
queue empty ([[ ]])
exit=0

The message blames -eq. The bug is the unquoted $count before it: word splitting removed the empty value, so [ received -eq 0 and failed with status 2, which an if treats as "not true". The [[ ]] line does no splitting, read the empty string as 0 in an arithmetic test, and printed a confident wrong answer. Then the script exited 0.

Strict mode does not help here. count="" is set, so set -u stays quiet, and set -e ignores failures inside an if condition. What stopped it was validating the value first: [[ $count =~ ^[0-9]+$ ]] || exit 1.

Most bash errors work this way. The message names the check that failed, not the cause: a syntax error reported on line 4 for a missing semicolon on line 2, Permission denied after chmod +x on a noexec mount. Nine of them, each reproduced with the check that names the cause, are in Bash Error Messages, Decoded.