Coder Legion - Recent posts tagged sre

Drill the Safe Open: How IANA's 2020 Root Key Ceremony Ran Four Days Late

Mon, 26 Oct 2026 15:36:01 +0000

In February 2020 the safe holding the internet's root signing key would not open. IANA's published, hand-initialed ceremony scans show the custody behind the cryptography: a repair script written before anyone drilled, and a signing that took four minutes and fifteen seconds.

Page 6 of the script is a table with four columns: Step, Activity, Initials, Time. In step 5 a locksmith replaces a combination lock, and the first task on the list reads "a) Drill the lock/safe open." Beside it, in blue pen, someone has initialed the box and written a time: 21:48+1.

The safe is Equipment Safe #1 in the Key Management Facility that IANA runs in El Segundo, California. It holds the hardware security modules that keep the private half of the root zone's key-signing key, the trust anchor that validating resolvers chain every signed DNS answer back to. In February 2020 it would not open. IANA publishes the paperwork from its key ceremonies, scanned with the handwriting left in, so the story of how it got into its own safe can be read page by page, initials and all.

Most writing about the root key ceremony is about cryptography. The paperwork is mostly about custody: who held what, who watched, and what got written down when something went wrong. That February the cryptography took four minutes and fifteen seconds. Getting into the safe took most of four days.

How a ceremony runs

The root zone is signed in quarters, and the key-signing key is brought out of its safe four times a year to sign the keys for the next one. The rule in force in 2020, in IANA's DNSSEC Practice Statement, was that each quarter's ceremony is held "approximately 60 days but no later than 33 days, before the time cycle commences."

The people are set out in roles. A Ceremony Administrator runs the script and an Internal Witness keeps the record. Safe Security Controllers hold the combinations, two people for each of the two safes: the Equipment Safe with the hardware, and the Credentials Safe with the safe deposit boxes. Crypto Officers hold the keys to those boxes. The Practice Statement describes "successive barriers in three tiers, requiring at least seven trusted persons from four different roles," and activating a hardware module "requires three out of seven Crypto Officers to extract activation material from above safe deposit boxes using their physical key." The Crypto Officers are Trusted Community Representatives, picked from statements of interest that IANA solicits from the internet community, and its roster page states the rule that keeps them independent: "Persons considered directly affiliated with PTI, ICANN or VeriSign may not become a Trusted Community Representatives."

The script's preamble says how the room behaves. "CA reads each step aloud prior to its performance." "Upon completion of each step, IW announces the time of completion, records the completion time, and initials their copy of the script." And then the paragraph the rest of this story runs on:

Ceremony participants who notice a problem or an error during the ceremony should interrupt the ceremony. Ceremony participants agree on a resolution before proceeding ... Unplanned events (exceptions) during the ceremony are evaluated, documented, and acted upon. It is the CA's sole responsibility to decide on proper actions after consulting with the IW. In either case, an exception is regarded as an incident, and incident handling procedures are enacted.

"Documented" has a form. The Root DNSSEC Script Exception is a one-page sheet. Its first row asks for the Act, the Step, the Page, and the date and time, and its second says "IW describes the exception(s) and action(s) below," with a box for initials and a box for the time on each row. An exception is filed against a place in the script, the way a bug is filed against a line of code.

The day the dial was due for replacement

The trouble surfaced during maintenance, and the maintenance was itself a ceremony with its own script. Administrative ceremonies are smaller affairs, with no signing, but the same rules apply: a Ceremony Administrator, an Internal Witness, cameras, and a participants table. In that week's tables the participants included a locksmith from an outside lock-and-security company.

The work was to replace the combination dials on both safes, swapping each Kaba Mas X-09 for a new X-10. The day before, at a ceremony to program the safe deposit boxes' lock assemblies, one assembly had misbehaved. That got a sheet too:

The last KMF West Lock Assembly was malfunctioning, and we delayed the bagging of the unit to allow more time for the locksmith to troubleshoot. Once the lock assembly was functioning, we placed it in TEB BB91951419 and continued where we left off in the script.

TEB is a tamper-evident bag, and the bag's serial number goes in the record.

The script for the Equipment Safe, dated Tuesday 11 February, has an Act 2 called "Safe #1 Equipment Maintenance." Step 1, initialed at 21:34, has the administrator and the witness carry "a flashlight, phillips screwdriver, change key tool" into the safe room. Step 2 reads "SSC1_A opens Safe #1 while shielding the combination from the camera," with a note that the controller "will begin by rapidly spinning the dial counter-clockwise in order to charge it." The lock powers itself from the dial. Step 4 is the job they had come to do: "a) Remove the current Kaba Mas X-09. b) Install a new Kaba Mas X-10."

Step 2 has no initials. The exception sheet behind it is filed against Act 2, Step 2, Page 6, dated 2020/02/11 and initialed at 23:14:

SSC1_A was unable to open the safe due to malfunctioning hardware. Escorted out by IW2 to verify combination with other SSC. Locksmith contacted manufacturer and they suggested the issue was a faulty logic board and the dial will require drilling. Ended ceremony to discuss steps going forward.

So the dial that failed was the one they had come to replace, and it failed at the one step that needed it to work. A second ceremony a couple of hours later opened the Credentials Safe without trouble and swapped its X-09 for an X-10.

The next day ICANN's blog announced that the 40th ceremony, "originally scheduled for 12 February 2020 in El Segundo, California, has been postponed." "One of the security mechanisms that protects the contents of the secure safes was found to be malfunctioning." "The issue was identified yesterday, 11 February 2020, during routine maintenance of the Key Management Facility." The post said there would be "no interruption to Domain Name System Security Extensions (DNSSEC) service," that "We maintain a complete replica facility in Culpeper, Virginia," and that the repair was scheduled for Friday the 14th.

The branch was written before anyone drilled

The 11 February script had no step for a safe that would not open. Step 2 simply assumed it would. When it failed, nobody improvised the repair in the room. They filed the sheet, ended the ceremony, and wrote a new script.

The repair ceremony began on the 14th, and its Act 2 opens with a section titled "Confirmation of Equipment Safe #1 Dial Malfunction." Before anyone was allowed to drill, the controller tried the safe again, under the cameras, at 17:26. The step carries a note that "Locksmith may provide additional advice/techniques to attempt to operate the safe dial." Then the script splits:

If the safe opens: Complete the safe log by following the steps below ...

If the lock malfunctions: Continue with the steps below.

If the safe had opened on that attempt, the script would have sent everyone to the safe log and the dial swap, and there would have been no drilling. It didn't open. Step 5 begins "The locksmith performs the following tasks to replace the combination lock," and lists them: "a) Drill the lock/safe open," "b) Remove the current Kaba Mas X-09 Safe Dial," "c) Return the safe door to a secure state by performing necessary repairs," "d) Install a new Kaba Mas X-10 Safe Dial." Under the first task sits a note about breaks: "A ceremony break will be performed as required. All personnel will be escorted out of the Tier 5 (Safe Room) and Tier 4 (Ceremony Room.)" Each lettered task has its own box for initials and time.

Drilling into the safe that guards the root key was written down in advance as a numbered, witnessed, timed step, with a branch for the possibility that it wasn't needed.

What the sheets caught

The repair took two days, and the packet of exception sheets behind its script reads like a log of everything the plan didn't specify. I read four of them, all filed against Act 2, Step 5, Page 6. I didn't read every page of the packet, so four is a floor.

The first, dated the 14th at 21:00, records that an external witness "left the room on a break and stayed outside until the next break," returning at 22:19, and adds: "Due to the length of the ceremony carrying into the next day, other non essential participants came and went as well." The second is one sentence: "Ceremony was paused overnight and resumed in the morning."

The third, at 19:26 on the 15th, is about the building. A security guard used the hand scanner outside Tier 3 to escort an external witness in, and that triggered the alarm. The witness was asked to step out and wait in the conference room, "and the alarm was triggered again when he manually opened the door."

The fourth is about the order of steps:

Steps e, f, g, and h were performed immediately after opening the safe for the first time which occurred between steps a and b.

Steps e to h are the safe log: the controller shows the old log to the camera, takes the pre-printed one from the witness, writes the time and signs where it says "Open Safe," and the witness checks and initials the entry. The script had them after the new dial went on. In the room, they did them the moment the drilled door first opened, and the times in the script's own columns agree: a at 21:48, e to h at 21:54, the new dial at 23:56. Doing the log first was arguably the better order, since it put the opening on paper at once. Nobody rewrote the script to match. The deviation went on its own sheet, and the record shows the steps in the order they happened.

The times in that column all carry a "+1," which the script never defines. I read it as the day after the ceremony began, which puts the drilling at 21:48 UTC on the 15th. Three other records agree with that reading: the overnight sheet is dated the 15th at 00:38, IANA's announcement that the safe was open went out at 22:10 UTC on the 15th, and the participants signed the ceremony off at 00:44 on the 16th. The discovery sheet was initialed at 23:14 on the 11th, so the drilling step was initialed 3 days, 22 hours and 34 minutes after it.

IANA's message to the root-dnssec-announce list read: "We have successfully opened the safe with the failed mechanism, and are in the final stages of remediation, with the repair work and installation of a new lock underway. We are therefore arranging resources to start the KSK Ceremony 40 at 0000 UTC, or shortly thereafter."

Four minutes and fifteen seconds

The signing went the way the script said it would. The signer's log for Ceremony 40 opens with "Starting: ksrsigner /media/KSR/KSK40/ksr-root-2020-q2-0.xml (at Sun Feb 16 02:21:33 2020 UTC)." Twenty-two seconds later the hardware module is activated. At 02:25:48 the log reads "Generated new SKR," a signed key response whose first signature period begins on 1 April 2020, made with the key whose tag is 20326. The log prints the response's SHA-256 hash in hex, and then again as a run of English words: "unwind leprosy fracture savagery tactics speculate Neptune miracle slingshot aftermath …"

From start to signed response is four minutes and fifteen seconds. Against the Practice Statement's rule, the latest date for the second-quarter ceremony was 28 February, so the signing finished with nearly twelve days to spare.

The ceremony's closing act has the administrator read every exception out before anyone signs. Step 1 of the participants' signing reads "CA reads all exceptions that occurred during the ceremony." In step 2 each attendee comes to the table and signs the witness's list, and "All signatures declare that this script is a true and accurate record of the ceremony." The read-back comes first. You hear what went wrong before you put your name to the record.

April: nobody could come

Two months later the failure was the people. Ceremony 41 was set for 23 April 2020, in the first weeks of the pandemic. IANA's published questions-and-answers document says there were no representatives for the El Segundo facility "located close enough to allow unrestricted and safe travel to the facility," and says where they were: "Mauritius, Spain, Russia, Tanzania, Uruguay and east-coast United States."

The plan IANA used kept the custody chain and moved the people. "Four trusted community representatives (TCRs) who fulfil Cryptographic Officer roles in the ceremony have transmitted their safe deposit keys to separate ICANN and PTI staff. These staff will act as surrogates, but the TCRs will participate in the ceremony remotely and monitor their credential's use." The keys travelled sealed: "The TCRs have wrapped their deposit keys with opaque material, and then transmitted them in tamper-evident bags. This bag will not be opened until within the ceremony so that each TCR can witness their key is in the same condition as when they released it." The ceremony would sign three quarters at once instead of one, and "The additional generated signatures will be initially withheld."

The plan they didn't use is in the same document. "The first version of our contingency plan assumed the trusted community representatives would not transmit their safe deposit box keys for performance of the ceremony. In such a scenario, a trained locksmith would drill and replace the locks to three safe deposit boxes during the ceremony in order to retrieve the smart cards needed to activate the hardware security module." Two months after the safe, the first version of the fallback was to drill again.

One part of the April plan broke the rules as they stood, and the document says which: "The only element of our revised approach that was inconsistent with the DPS was signing additional quarters of KSRs during the ceremony. The Policy Management Authority, an internal committee that reviews changes to the DPS, convened on 6 April and approved revisions that allow for signing additional quarters of KSRs as a disaster recovery response." The rule was changed by its owner seventeen days before it was needed, and not by the people in the room on the day.

The Ceremony 41 script also adds a step to its closing act. After reading the exceptions, the administrator "asks any COs who are participating remotely if they have any concerns pertaining to the ceremony or exceptions which may have occurred."

By November the bend was standing policy. The Practice Statement in force in February 2020, the fourth edition, has no variance clause. The sixth edition, dated 4 November 2020, carries section 4.5.4.1, and the eighth edition of 2025 still carries it word for word:

In the event of a declaration of a disaster recovery or business continuity response, the adverse event may necessitate variances to the procedural controls and technical security controls. These variances shall be limited in scope to those necessary for the successful conduct of key signing operations, and according to protocols that best ensures a proper record of the chain-of-custody and limiting sensitive operations to trusted persons.

The clause ends by requiring that variances are "consented by the RZ KSK Operator's management, are documented and communicated." I didn't read the fifth edition, so I can't say which edition first carried it, only that it arrived between February and November 2020.

The record since

IANA's schedule lists 63 held ceremonies from June 2010 to August 2026, counting Ceremony 53's two parts separately. In 2020 and 2021 there were two each, the only full years with fewer than four. Ceremonies 41 and 42 are the only ones that signed more than one quarter, and the 294 days between them is the longest gap in the schedule.

The public archive runs to 65 directories and 940 files, among them 315 video files from the ceremony cameras. Files from administrative ceremonies first appear in the directory for Ceremony 40, the safe repair, and turn up again in 15 later directories. I didn't read what those later administrative ceremonies were for. Everything here comes from IANA's and ICANN's own record. That is the point of reading it, and also its limit, because an outside auditor's view isn't quoted here.

What carries over

None of this is exotic. It is a procedure that expects to be interrupted, and a record that treats the interruption as a first-class entry. Anyone running software that acts where actions can't be taken back, a deployment pipeline or an agent holding production credentials, can copy four habits from it.

An exception there is a typed record: act, step, page, time, initials, and a description written by the witness rather than the actor. When the script doesn't cover a failure, the response in that room was to stop and write a new script, with a branch for each outcome, before taking the irreversible step. When steps happen out of order, the record says so instead of being tidied. And when a rule has to bend, its owner bends it in advance, in writing, and later turns the bend into a clause with limits.

At the back of the Ceremony 40 script is a page for the Internal Witness alone. It is two sentences long: "I hereby attest that the Key Ceremony was conducted in accordance to this script. Any exceptions that occurred were accurately and properly documented."

Sources

IANA, Key Signing Ceremonies, the schedule of held and planned ceremonies (as fetched 7 October 2026).
IANA, ceremony archive, 65 directories of scripts, logs and video.
IANA, Ceremony 40 administrative scripts, annotated scans: AC40_1, 10 February 2020 (lock assembly exception, page 7); AC40_2, 11 February 2020 (Act 2 and the discovery exception); AC40_3, 11 February 2020 (the Credentials Safe); AC40_4, 14 to 16 February 2020 (the drilling, script page 6, and the exception sheets).
IANA, Ceremony 40 script (preamble, closing act, Appendix C) and signer log, 16 February 2020.
Kim Davies, "Root Key Signing Key Ceremony Postponed," ICANN blog, 12 February 2020.
Kim Davies, "[Root KSK Ceremony 40 scheduled for 0000 UTC](https://lists.icann.org/hyperkitty/list/*Emails are not allowed*/message/Z7KNDGDKD5PNZJ52XHP5Y4W3GIYM23H3/)," root-dnssec-announce, 15 February 2020.
IANA, Questions and Answers on COVID-19 mitigations for KSK Ceremony 41, and the Ceremony 41 script, April 2020.
DNSSEC Practice Statement for the Root Zone KSK Operator: 4th edition, 1 October 2016, as published at iana.org/dnssec/icann-dps.txt in February 2020 (Wayback Machine capture, 16 February 2020), sections 4.2.2 and 6.6; 6th edition, 4 November 2020 and 8th edition, 14 April 2025, sections 4.2.2 and 4.5.4.1.
IANA, Trusted Community Representatives (as fetched 7 October 2026).

How this was checked. Typed quotations were checked word for word against the saved text of the source named beside them. The handwritten ones were transcribed from the scanned pages of the annotated scripts and checked against the images; the scans have no text layer. Times are UTC, as the scripts record them, and the "+1" day is read as explained in the text. The intervals (3 days, 22 hours and 34 minutes; 4 minutes and 15 seconds; the margin before 28 February) and the counts from the schedule and the archive listings were computed by script from those sources. Four exception sheets is the number read in the repair ceremony's packet, not a total for the month.

Originally published at vibeagentmaking.com. If you want a tamper-evident, dated record of what an automated system did and what went off-script: pip install chain-of-consciousness or npm install chain-of-consciousness, or try Hosted Chain of Consciousness.

Can AI SRE Really Reduce MTTR? What It Can and Cannot Automate

Fri, 23 Oct 2026 07:58:29 +0000

Most SRE teams do not lose the bulk of an incident because nobody noticed something was broken. In mature environments, alerting is already fast enough to wake the right people, interrupt the right channel, and open the right incident record. The harder question starts after the page: what changed, which dependency is involved, how wide is the blast radius, which dashboard matters, what did we try last time, and what action is safe enough to take right now?

The practical case for AI SRE is narrower than “replace the on-call engineer,” and more useful: reduce the time between alert and evidence-backed hypothesis. Good incident response depends on actionable alerts, prepared on-call teams, up-to-date playbooks, clear coordination, and automation that removes repetitive work so responders can focus on judgment.

So, can AI SRE reduce MTTR? Yes, mostly by compressing investigation, triage, handoffs, runbook selection, and documentation. In practice, the largest MTTR gains usually come from shortening two intervals: alert-to-hypothesis during investigation and mitigation-to-postmortem during recovery review. AI can reduce decision-making drag. It should not be treated as a blanket license for autonomous production changes.

Where incident-response time actually goes

MTTR is often discussed as one number, but responders experience it as a sequence of smaller clocks. Time to detect, time to acknowledge, time to orient, time to investigate, time to mitigate, time to verify, time to communicate, time to resolve, and time to document. Monitoring and paging tools have already improved the first two. The middle of the incident is where time still disappears.

That middle stage is messy because it is not one task. It is a search problem under pressure. Engineers move between dashboards, logs, traces, deploy history, feature flags, ownership maps, recent pull requests, dependency graphs, previous incidents, runbooks, customer reports, and tribal memory. Some of this work is mechanical. Some of it is judgment. The expensive part is that both happen at the same time.

A useful operating metric is the time from alert to first useful hypothesis. Not the first guess. Not the loudest theory in the incident channel. A useful hypothesis is specific, falsifiable, and supported by evidence: “The checkout latency spike started three minutes after deployment x; error volume is concentrated in service y; downstream dependency z is healthy; rollback is likely safe.”

A second clock starts after mitigation. Once the service is stable, teams still have to reconstruct the incident from chat threads, dashboards, timestamps, and memory. If that work takes days, the team loses context and ships weaker corrective actions. AI can preserve the evidence while it is still fresh, making the postmortem less dependent on whoever happened to take notes under pressure.

This is where AI SRE reduces MTTR. It gathers context that humans would otherwise collect manually, ranks likely explanations, and keeps a traceable record of what it found. During the first minutes of an incident, responders usually need the same pieces of context: the relevant runbook, recent changes, logs from the failing path, ownership details, downstream impact, and whether the same pattern has appeared before. AI is useful when it surfaces that context before the incident channel fills with guesses.

What AI SREs can automate today

The near-term value of AI incident response automation is strongest when the work is read-heavy, repetitive, and evidence-based. Claims weaken when vendors imply that any incident can be diagnosed and fixed without human review.

Alert triage

AI can help with alert triage by grouping related alerts, suppressing obvious duplicates, identifying impacted services, suggesting ownership, and connecting a new incident to past incidents. It can also summarize severity signals for the incident commander: which regions are affected, which customer-facing paths are degraded, whether error budgets are burning, and whether the pattern resembles a known failure mode.

The caveat is simple: AI does not rescue bad alert hygiene. If alerts are noisy, cause-based rather than symptom-based, or detached from user impact, the AI has to reason from weak inputs. That usually produces faster confusion, not better triage.

Log and metric investigation

AI is well suited to pulling together logs, metrics, traces, deploy events, and topology data. A human can ask a focused question, and the system can retrieve the relevant slices without requiring the engineer to remember every query language or dashboard path at 3 a.m.

Summarization is not proof. Correlation is not causation. The output has to remain inspectable: links to raw logs, metric windows, traces, deploy diffs, and the reasoning path. If the AI cannot show why a signal matters, the responder still has to redo the investigation manually.

Root-cause analysis

Root-cause analysis automation is most credible when it produces ranked hypotheses, rather than a single magical answer. A good AI SRE should say what it thinks happened, what evidence supports that view, what evidence argues against it, and what validation step should come next.

This is the philosophy HyperProbe is built around. The goal is not to ask engineers to trust an opaque RCA summary. It is to capture diagnostic evidence through read-only probes, preserve the context around the failure, and help the engineer confirm the root cause faster. That distinction matters. In production incidents, “the AI says so” is not an acceptable causal chain.

RCA is also where many AI systems overstate their confidence. Distributed failures rarely announce themselves cleanly. A downstream timeout may be the visible symptom, while the real cause is a slow dependency, a bad config push, a connection-pool limit, a partial deploy, or an unexpected interaction between services. An AI SRE should make that uncertainty explicit and keep alternative hypotheses alive until the evidence eliminates them.

Incident timelines and postmortems

Incident postmortem automation is one of the least controversial uses of AI SRE. The system can capture actions, decisions, timestamps, communications, metric changes, deploy events, and remediation attempts while the incident is still active.

Afterward, it can draft a timeline, summarize customer impact, propose action items, and turn recurring patterns into playbook candidates.

This will not make postmortems automatically good. Humans still need to decide what was causal, which actions to prioritize, and how to write a blameless analysis. AI can remove a lot of reconstruction toil.

Runbook execution

AI can make runbooks more usable. It can find the right runbook, map the current incident to the relevant branch, fill in service names or regions, check preconditions, and propose an action.

In a mature setup, it can also verify postconditions after the action: alert cleared, error rate normalizing, affected pods healthy, latency returning to baseline.

The key is to separate runbook recommendation from runbook execution. Suggesting “restart this worker pool” is not the same as restarting it. Parameterizing a rollback is not the same as executing it across production. AI should earn execution authority workflow by workflow, not inherit it globally.

Where autonomous remediation becomes risky

Autonomous remediation becomes risky when an AI system moves from reading production to changing production. That boundary is operational. A read-only investigation can waste time or lead to the wrong hypothesis. A state-changing action can expand the outage.

The risk increases when an action has a broad blast radius, ambiguous rollback, hidden coupling, incomplete telemetry, customer or revenue implications, or interaction with another live change.

Database failovers, schema changes, region-level traffic shifts, global cache flushes, broad restarts, security-policy changes, and feature-flag flips tied to billing or conversion are not good first candidates for full autonomy.

The right mental model is copilot before autopilot. A safer workflow uses deterministic tools, scoped permissions, policy checks, human authorization, and a durable audit trail before mutating production.

The AI can propose the action, explain the expected effect, list the rollback path, and verify the result. That does not mean it should always be allowed to execute the action by itself.

The hard part is not generating a plausible command. The hard part is knowing whether the command is safe under the current production state, whether it conflicts with another action, whether it can be reversed, and whether the system is healthier afterward.

This is why autonomous remediation needs more than model confidence. It needs bounded actions, rollback semantics, post-action verification, and clear rules for giving control back to humans.

Human-in-the-loop vs. fully autonomous workflows

The practical question is which workflow deserves which level of autonomy.

A read-only AI investigation workflow is the easiest place to start. The AI gathers evidence, summarizes context, ranks hypotheses, and drafts updates, but it cannot mutate production.

HyperProbe fits naturally here because its read-only probe architecture, default PII redaction, and immutable audit logs support evidence capture without turning every investigation into a write-capable automation path.

The next level is human-approved execution. The AI proposes a runbook-backed plan with preconditions, expected impact, rollback steps, and verification checks. A human approves the action, and the system executes through scoped tools.

Fully autonomous remediation should be narrower. It belongs to pre-approved, low-blast-radius, reversible actions with clear success criteria: restart a known stateless worker, scale a saturated deployment within policy, roll back a recent canary, or rotate traffic away from a single unhealthy instance. Even then, autonomy needs scoped identity, RBAC, dry runs, policy gates, rollback, audit logs, and post-action verification.

Teams should not flip a global switch from manual operations to autonomous operations. They should graduate individual workflows. A stateless restart might move to approval-based execution after several successful recommendations. A canary rollback might become automatic only after the team has clear checks for customer impact, deploy state, and rollback success. A database failover may remain human-approved indefinitely. That is not a failure of AI adoption; it is good production engineering.

What teams should realistically expect from AI SRE

Teams should expect AI SRE to reduce investigation drag before it eliminates on-call work. The realistic gains are fewer blind pivots, faster context gathering, more consistent triage, cleaner handoffs, better incident summaries, faster postmortems, and more disciplined runbook use.

They should not expect perfect RCA, universal zero-touch remediation, or a system that can compensate for missing telemetry and stale documentation. AI SRE depends on operational inputs: service ownership, deploy metadata, logs, metrics, traces, runbooks, dependency maps, past incidents, and clear access boundaries. If those inputs are incomplete, the AI will either ask better questions or make weaker guesses.

The best implementations start with a measurement plan. Track time from alert to first useful hypothesis. Track time spent gathering logs and change context. Track how often AI-suggested hypotheses were confirmed, rejected, or partially useful. Track whether postmortems became faster and more complete. MTTR reduction is the outcome, but diagnostic compression is the leading indicator.

Trust matters as much as speed. A fast suggestion that engineers ignore does not reduce MTTR; a cited, reviewable investigation has a better chance of changing responder behavior.

The maturity model: AI-assisted → AI-investigated → AI-recommended → AI-remediated

A useful way to evaluate AI SRE adoption is to treat autonomy as a maturity model, not a binary switch.

AI-assisted

At this stage, AI helps responders read and write faster. It summarizes alerts, explains dashboards, searches documentation, drafts status updates, and answers questions about known systems. It does not perform independent investigation, and it does not take production action. This is low risk and useful, but it will only modestly reduce MTTR if engineers still have to gather all evidence themselves.

AI-investigated

Here, the AI actively collects incident evidence. It pulls relevant telemetry, correlates recent changes, builds an incident timeline, finds similar historical incidents, and produces ranked hypotheses. Engineers still verify the facts, but they start from a prepared diagnostic brief rather than a blank page. This is where many teams will see the first meaningful MTTR reduction.

AI-recommended

At this stage, the AI proposes a remediation plan. The recommendation includes the suspected cause, evidence, runbook reference, expected impact, rollback path, and verification criteria. Humans approve execution. This is the right default for high-value production services because it accelerates action without removing accountability.

AI-remediated

The final stage is limited autonomy. The AI executes only pre-approved, reversible actions inside strict policy boundaries. It verifies the result, rolls back when conditions fail, and records the full trail for review. This can work for narrow classes of incidents, but it should be earned through repeated evidence, not assumed because the model sounds confident.

Conclusion: reduce MTTR by reducing uncertainty

The strongest case for AI SRE is not that it can take production away from engineers. It is that it can remove the investigative drag that keeps engineers from making a confident decision sooner.

MTTR stays high when responders have to rebuild context from scratch during every incident. They need to know what changed, what is affected, which signal matters, which hypothesis is most likely, which action is safe, and how to prove the system recovered. AI can help with that. It can gather evidence, preserve timelines, connect symptoms to recent changes, surface similar incidents, draft postmortems, and recommend the next step with enough context for a human to challenge it.

That is a meaningful reduction in operational load. It gives engineers back minutes during the incident and hours after it. It also improves response quality because the team spends less time searching and more time deciding.

Autonomous remediation should come later, and only for workflows that have earned it. The safer path is to move from AI-assisted to AI-investigated to AI-recommended before allowing AI-remediated actions. Each step should be backed by evidence, scoped access, rollback, verification, and auditability.

For teams evaluating AI SRE, the question should not be “Can this system fix production by itself?” A better question is: “Can this system help our engineers reach the right conclusion faster, with evidence they trust?” That is where MTTR reduction starts.

HyperProbe is built for that evidence-first path: faster investigation, verified root cause, and human-controlled production decisions.

FAQ

Can AI SRE really reduce MTTR?

Yes, but usually by reducing investigation time rather than by fully automating production fixes. AI SRE is most useful when it helps engineers move faster from alert to evidence-backed hypothesis, then preserves the context needed for verification and postmortems.

What part of incident response can AI SRE automate today?

AI SRE can automate or assist with alert triage, log and metric investigation, change correlation, root-cause hypothesis generation, incident timelines, postmortem drafts, and runbook recommendations. These workflows are valuable because they are evidence-heavy and often repetitive.

Can AI SRE automate root-cause analysis?

AI SRE can assist root-cause analysis by producing ranked hypotheses, supporting evidence, counter-evidence, and next validation steps. It should not be treated as a final authority because production incidents often involve incomplete telemetry, hidden dependencies, and multiple plausible causes.

When does autonomous remediation become risky?

Autonomous remediation becomes risky when the action changes production state, has broad blast radius, is hard to roll back, depends on incomplete data, or affects customers, revenue, security, or compliance. These actions should usually require human approval.

What is the safest way to adopt AI SRE?

The safest path is a staged maturity model: AI-assisted, AI-investigated, AI-recommended, and only then AI-remediated for narrow, reversible workflows. Each stage should include evidence, scoped permissions, rollback, verification, and audit logs.

TSB Went Live on a Letter Regulators Called "Statements of Good Intention"; £330M Aftermath

Wed, 21 Oct 2026 12:29:53 +0000

TSB's 2018 move off the Lloyds platform could not be rolled back. The FCA and PRA notices show the go decision rested on a supplier letter, a test run on one of two data centres, and a memo that left the letter's caveats behind.

On 10 April 2018, TSB's board met to decide whether to serve notice on Lloyds Banking Group. Serving it would commit the bank to moving its customers off the Lloyds IT platform it had run on since it was carved out of Lloyds in 2014, and onto Proteo4UK, a platform built for it by its new owner, Sabadell. It would also give up the fallback, a copy of the Lloyds platform that TSB had the option to carve out and keep. That same day, the last outstanding confirmation from the platform's supplier chain arrived.

The board went ahead. On the weekend of 20 to 22 April, TSB moved its customers' accounts across, predominantly in a single event. When the new platform opened, its online and mobile channels failed, and they kept failing. TSB "did not return to a BAU position until 10 December 2018," the FCA found, and between April 2018 and April 2019 it received 225,492 complaints.

There was no way back. The FCA's Final Notice puts it plainly: "after MME it was, technically speaking, essentially impossible to roll back from the changes made and revert to using the LBG IT Platform." MME is the Main Migration Event, that April weekend.

So the go decision was the whole decision, and what it rested on is the story. The record is unusually complete: two Final Notices issued on 20 December 2022, one from the Financial Conduct Authority and one from the Prudential Regulation Authority; a third from the PRA in April 2023, against the bank's chief information officer; and the bank's own 2018 accounts. Read together, they show a cutover with no way back approved on three pieces of evidence: a letter, a test and a memo. None of the three was what it looked like.

Why the bank was moving at all

Sabadell bought TSB in July 2015, and the migration was the point of the purchase. Its offer document promised "efficiency cost savings in IT amounting to approximately £160 million per annum on a pre-tax basis, in the third full year after completion of the Offer," and said those savings would "derive from a full migration" onto Sabadell's Proteo platform. The FCA records that "The strategic rationale for the takeover included the expected financial returns from a full migration," and that "The aim was to achieve migration by the end of 2017."

Sabadell knew how to do this. It "had previously migrated seven banks onto" Proteo, the PRA notes, and its "existing methodology for IT migrations was to use a 'big bang' or single operation data migration approach." This one was different in size, as the FCA notes: "The scale of the migration project was unprecedented," and the Proteo4UK platform "was not yet proven."

TSB's board saw the shape of the risk early. In December 2015 the minutes recorded that "the Board could not be put in a position where they gave up the carve-out option, migration "stalled" or was found to be unworkable leaving the bank in a position without an IT infrastructure." That is, in so many words, the risk the go decision would later take.

The schedule slipped once. In September 2017 the board was told the bank could not migrate in November as planned. It acknowledged that the November date, set two and a half years earlier, had been "deliberately very ambitious", had acted as a "forcing mechanism", and had been "based on very little information". The new "Defender Plan" was approved on 24 October 2017. Then, according to the PRA, "The Migration Programme quickly fell behind the Defender Plan schedule, with the result that critical testing plans and principles had to be deviated from to keep on track for MME."

The letter

The supplier was SABIS, Sabadell's IT subsidiary, which built and ran the platform. On 5 April 2018 it wrote to TSB expressing "confidence as to the migration readiness of the platform." The letter gave "an early report on certain test results (noting that some tests were still to be completed)" and referred to confirmations from SABIS's own suppliers, the "Critical Fourth Parties." Those confirmations "were referenced in (but not appended to) the letter."

Both regulators later read those confirmations and reached the same verdict, in the same words. They were "to some extent forward looking statements of good intention or expectation rather than statements of fact about the completeness of readiness activities undertaken." And "All but one was caveated with a number of outstanding tasks or tests which had not yet been completed."

The letter then travelled up the bank inside a summary. The memo that recommended going ahead carried the chief information officer's attestation, which "asserted that SABIS was ready for MME and, without any further explanation, that Mr Abarca was satisfied that the SABIS Confirmation could be relied upon." The memo "did not annex the letter itself. Nor was the letter included in any of the papers for the Board." The caveats were not in the papers of the board that committed the bank.

The dates make it starker. The last supplier confirmation "was received on 10 April 2018, the same day that the TSB Board was meeting to determine whether to serve the definitive notice of migration on LBG." The tests the letter called outstanding "were completed on 17 April 2018." The board's decisions came on 10, 18 and 19 April, and the migration began on the 20th.

One more change sat underneath. On 9 February 2018, confirming that non-functional testing was complete became the IT function's sole responsibility. The FCA found that this change "does not appear to have been explicitly drawn to the attention of the TSB Board" when it made those April decisions.

The test that ran on half the system

The failure itself was not in the data. The PRA is explicit that "the data migration itself was successful." What failed was the way customers reached the new platform.

Proteo4UK ran in two data centres in an "Active-Active" configuration, with a Global Load Balancer in front deciding which centre served each session. Each centre "had the capacity to run the entirety of the customer sessions on its own." And there was nowhere safe to rehearse: "TSB did not have a pre-production environment on which to conduct testing which would stress the platform without affecting the live production environment."

TSB first decided to performance-test both data centres together, "to ensure that the conditions were the same as expected at MME." After a testing forum on 28 February 2018, it accepted a "counter-proposal" from SABIS to test only one, "due to concerns that the live services would otherwise be unavailable (for example ATMs would not have been available for a period of approximately three to four hours)."

The notice records the reasoning. The configuration of the two centres was assumed to be "symmetric following assurances received," and "if it could pass with half the capacity, it would be able to easily pass using data centres." So the test, too, stood on an assurance. The NFT Final Report and the memo that went with it did not refer "to any risks associated with conducting performance testing in only one data centre."

Then the platform went live, and "following MME it became apparent that both the Global Load Balancer and the Security Session (OpenAM) component of the data centres had not in fact been correctly configured, leading to the issues experienced by customers in trying to access the digital channels."

The notice states those facts and stops. The inference is ours, and it is the operator's lesson of the whole case. A capacity test on one node proves capacity. It cannot prove that two nodes are the same, and it never exercises the component that routes between them. The fault sat in exactly the parts the one-centre test left out: the load balancer, whose job is to divide traffic between the two centres, and the session component, whose configuration was assumed to match. The test was real and it passed. It answered a different question from the one the go decision needed answered.

TSB's incident plan had made a similar assumption about failure itself. The bank "was prepared to deal with "bumps in the road"" but not "a massive failure in the configuration leading to an erratic digital service."

The bill

TSB's 2018 annual report puts the post-migration charges at £330.2 million. The two largest pieces were customer redress and associated costs, £107.3 million, and additional resource and advisory costs, £122.4 million. Against that the bank booked "the estimated recovery of £153.0 million of post migration charges" from SABIS under its two service contracts, "up to the level of liability caps in each agreement." Net of the recovery, the charges came to £177.2 million.

That recovery deserves a sentence of context. SABIS is "a subsidiary of the Banco Sabadell Group," and so is TSB. The £153.0 million moved money from one Sabadell subsidiary to another. The accounts read here do not net the group's position, so they cannot say what the group as a whole lost.

Then there were the costs of getting to the cutover. The same report shows £417.3 million of costs for preparing the migration, including "£249.0 million payable to Sabadell under the Migration Services Agreement." Preparation and aftermath together come to £747.5 million in one year's accounts. That is 4.7 times the annual IT saving the takeover promised. The comparison is a sense of scale, not a payback calculation: the promise was for the third full year after the deal, and nothing read here says whether it was ever met. (Our list of the most expensive software failures in history does not include TSB; on these figures it would not look out of place there.)

The regulators priced the failure more than four years later. The FCA fined TSB £29.75 million and the PRA £18.9 million, £48.65 million in all. That is 14.7% of the gross post-migration charges, or 27.5% of the net. Both fines ran the standard discount chain: the FCA's started at £50 million and came down by 15% for remediation and 30% for settling. The FCA found the breaches "committed negligently", and the PRA found them "neither deliberate nor reckless."

The customers' redress is in the record too. To 7 April 2019, TSB paid £32,705,762 in redress against 225,492 complaints, about £145 per complaint. The complaints equalled about 4.3% of a customer base the notices put at "approximately 5.2 million," roughly one for every twenty-three customers.

The one person in the record

One individual penalty appears in the published record. In April 2023 the PRA fined Carlos Abarca, TSB's chief information officer through the migration, £81,620. That was 15% of his relevant income of £777,356.86, cut by 30% for settling. He owned, by his senior-manager responsibilities, the risk that "migration causes operational instability or a degradation in resilience and poor customer outcomes."

The PRA's notice to him adds a detail. Before TSB, "Mr Abarca was Chief Process and Information Officer at Banco de Sabadell, S.A. ('Sabadell'), and at the same time, the Chief Executive Officer ("CEO") of Sabadell Information Systems, S.A. ('SABIS Spain')." At TSB he was responsible for the bank's "key outsourcing relationship with Sabadell Information Systems Limited ('SABIS UK')." The man whose job was to judge the supplier's word had run its Spanish counterpart.

The PRA drew the general rule from it: "The fact that a firm and its service provider are within the same group does not do away with the need for a careful assessment." Sam Woods, then the PRA's chief executive, put it in one line when the fine was announced: "the PRA has fined Mr Abarca because his management of a key outsourcing relationship fell below the standard we expect."

His fine was 0.17% of the firms' fines and 0.025% of the post-migration charges. The failure the PRA made personal was the management of an outsourcing relationship, down to an attestation that a supplier's letter could be relied upon.

What a readiness claim has to say

Strip the bank out of it and the case is about the grammar of "ready." SABIS's letter was written in the future tense: confidence, expectation, tests still to be completed. The board received it as a conclusion: the memo that carried it asserted readiness and left the letter, and its caveats, behind. Regulators later separated the two tenses for them: statements of good intention on one side, statements of fact about completed activities on the other.

The test made the same move in a different register. That the second data centre matched the first was a claim of fact. It rested on assurances, nobody had tested it, and the test plan treated it as settled.

For a team running any cutover, the lessons are plain. A readiness claim counts as evidence only when it says what was completed, not what is expected. An equivalence assumption is a test that has not been run, and it belongs on the list of open items rather than in the reasoning that closes them. And the less reversible the step, the more the go decision has to rest on evidence of the first kind. TSB's migration could not be rolled back, so its go decision was the last point at which the difference mattered.

The same grammar now turns up in software that reports on itself. An AI agent that says a task is "done" is writing SABIS's letter: a confident summary of activities, some of them still outstanding, written by the party that did the work. The fix is the one the regulators applied after the fact: ask for the statements of fact, attached, and read them before the step you cannot undo.

TSB's board met again on 18 and 19 April. The outstanding tests had finished on the 17th, and the memo in its papers still carried the attestation, not the letter. On 20 April the weekend began.

Sources

Financial Conduct Authority, Final Notice to TSB Bank plc, 20 December 2022 (115 pages), paragraphs 2.1-2.8, 2.24-2.25, 4.48-4.52, 4.179-4.205, 4.257-4.258, 4.267 and section 6.
Prudential Regulation Authority, Final Notice to TSB Bank plc, 20 December 2022 (108 pages), including paragraph 1.9 (Sabadell's offer document) and the penalty annex.
Prudential Regulation Authority, Final Notice to Carlos Abarca, 13 April 2023 (35 pages).
TSB Banking Group plc, Annual Report and Accounts 2018.
Financial Conduct Authority, press release, 20 December 2022; Bank of England, press release, 20 December 2022 and press release, 13 April 2023.

How this was computed. Every ratio above (fines against charges, the £747.5 million, the multiple of the promised saving, the redress per complaint and the CIO's fine against his income) is arithmetic on figures quoted in this piece from the notices and the 2018 accounts, so each can be recomputed from the text. Page and paragraph references are to the PDFs as published. Every quotation was checked against the saved text of its source.

Originally published at vibeagentmaking.com. If you want a tamper-evident, dated record of what a system actually completed, and when: pip install chain-of-consciousness or npm install chain-of-consciousness, or try Hosted Chain of Consciousness.

Google says on-call needs eight engineers. You have three.

Tue, 06 Oct 2026 14:27:45 +0000

TL;DR. Google's SRE book works out that a 24/7 on-call rotation needs at least eight engineers. A team of two or three cannot reach that number, so it has to change what the number is built from: which alerts page a person, which hours are covered, and how fast someone must answer. After that it needs one named person on call at any moment and a short escalation ladder behind them for the page nobody answers. A weekly count of pages shows whether it is working. The original post has the same text with a short FAQ at the end.

"We are not Google"

Google's first SRE book has a chapter called Being On-Call. Readers wrote back. The follow-up book, the SRE Workbook, opens its own on-call chapter with their feedback, and the first item on the list is this:

"We are not Google; we're much smaller. We don't have as many people in the rotation, and we don't have sites in different time zones. What you described in your first book is irrelevant to me."

I build Uptimepage, which pages people when a monitor goes down, so I had to decide what a rotation should look like for a team that small. That reader is right about the numbers. The reasoning behind the numbers still works for three people, and it leads to a different setup.

Where the eight comes from

The book starts from a limit on time:

"we strive to invest at least 50% of SRE time into engineering: of the remainder, no more than 25% can be spent on-call"

Then it does the arithmetic:

"Assuming that there are always two people on-call (primary and secondary, with different duties), the minimum number of engineers needed for on-call duty from a single-site team is eight: assuming week-long shifts, each engineer is on-call (primary or secondary) for one week every month."

Two people on call, each for at most a quarter of the time, makes eight. Run the same arithmetic on a small team, with one person on call at a time and week-long shifts:

Team size	Share of weeks each person is on call
2	50%
3	33%
4	25%
8, with two on call at a time	25%

A team of four reaches Google's limit only by giving up the secondary. A team of three is over the limit before anything breaks, and a team of two is on call every other week.

A schedule cannot fix that, because any rotation of three people still has three people in it. What the team can change is how often the pager goes off, during which hours, and how quickly it has to be answered.

The limit exists because pages are expensive. The same chapter estimates that one incident, with its diagnosis, fix and follow-up, takes about six hours of work, which is why it allows at most two incidents in a 12-hour shift. A person on call every third week can live with that week if it is quiet. So the work for a small team is to make the week quiet, then make sure the rare page reaches someone.

What a team of three can change

Decide what is worth a page

The Workbook is direct about this:

"Engineers shouldn't have to be at a computer and working on a problem within minutes of receiving a page unless there is a very good reason to do so."

Its Table 8-1 gives three examples of a realistic response:

Incident	Response time
Revenue-impacting network outage	5 minutes
Customer order batch processing system stuck	30 minutes
Backups of a database for a pre-launch service are failing	Ticket (response during work hours)

Sort every alert you have into those three rows. Only the first two should reach a phone at night. The rest go to a channel or an inbox that somebody reads in the morning.

An alert has to pass two more filters before it may page. It has to be real: one probe with a bad network path should not wake anyone. And it has to come with something to do. The SRE book's wording is "All paging alerts should also be actionable." A page that can only be watched until it clears teaches people to ignore pages.

Name one person at a time

The usual start for a small team is a shared channel: every alert goes to #alerts, and everyone is in it. During the day this works, because somebody is looking. At 03:00 five people receive the message, and each of them can reasonably think one of the other four has it.

A rotation puts exactly one name on call at any moment. That person knows it is them, and everyone else knows it is not. The channel can still get a copy, but the page goes to the person.

Hand off on a weekday morning, when both people are awake and can talk about what is still open. A handoff at midnight on Sunday means nobody mentions the disk that is filling up. Keep it at the same local time all year too, so the change does not move by an hour when the clocks do.

Split the day before you split the week

The Workbook does not expect one person to carry a full week around the clock:

"In our experience, 24 hours of on-call duty without reprieve isn't a sustainable setup."

Its suggestion fits a small team too: "it would be better for two engineers to split a week of on-call, with one person on-call during the day and one on-call overnight."

With three people that can be two rotations in one schedule. Working hours rotate weekly among all three. Nights and weekends rotate separately, with the order shifted by one, so nobody carries both in the same week.

There is also a cheaper option: leave some hours uncovered on purpose. If your customers are businesses in one region and nothing you run loses data while it is down, a page at 02:00 may only move the fix from 08:00 to 02:30. Decide that as a team, then write it where customers can see it, in your SLA and on your status page. As a customer I would rather read a stated response window during European working hours than a 24/7 promise kept by one tired person.

Do the arithmetic before you choose. A 99.9% target leaves 43 minutes a month. One night outage that waits six hours for the morning uses more than eight months of that budget. If the number you promised cannot absorb that, someone has to be on call at night.

Add a ladder for the unanswered page

One name on call is one point of failure. Phones run out of battery, and a person who was paged twice last night can sleep through the third.

An escalation policy is an ordered list of who gets paged next when a page goes unanswered. Level one pages when the incident opens, and each level waits a set time for an acknowledgement before the next one pages. Acknowledging stops the ladder.

For three people a ladder can be this short:

The person on call. Wait 10 minutes.
The other two. Wait 15 minutes.
Everyone, on the loudest channel each person has.

How long to wait comes from the response time you promised. The SRE book's typical values are "5 minutes for user-facing or otherwise highly time-critical services, and 30 minutes for less time-sensitive systems". If level one has 5 minutes to answer, the wait before level two cannot be 30.

This ladder is the small team's version of Google's secondary. Nobody is formally second on call, and the page still has somewhere to go. Watch how often level two fires. If it fires most weeks, the other two people are on call as well, whatever the schedule says.

Make swaps cheap

The Workbook again:

"No one can promise on Monday that they won't have the flu on Thursday."

On a team of eight a swap is easy to arrange. On a team of three every swap lands on one of two people, so it has to take a minute and leave no doubt about who holds the pager afterwards. That means an override on the schedule itself, for exact dates, which ends without anyone remembering to undo it. A chat message asking a colleague to cover Thursday changes nothing about where the page goes.

Put the shifts in everyone's calendar as well. A person who sees an on-call week next to a dentist appointment three weeks ahead asks for the swap now instead of on the day.

Test that the page arrives

A schedule can name a person the page cannot reach. Their phone is on do-not-disturb, the chat app is muted at night, or the email address was never confirmed. Everything looks covered until the first night page arrives on a silent phone.

So test it. On the first morning of a shift, send a test page to the person on call and have them acknowledge it from the phone they will have at night, with the settings they will have at night. Use at least one channel that is allowed to make noise through do-not-disturb, such as SMS from a number saved as an emergency contact, or a push app with an emergency priority.

GitLab's 2017 outage shows what an untested path costs. The failure alerts for its backup job were sent every night and rejected by the receiving mail server, and nobody knew for months.

Count the pages

Google's teams work to a number: "We target a maximum of two incidents per on-call shift". The Workbook also describes a team with a budget of two paging incidents per shift that had been receiving five for a year. Their answer was a project to reduce the pages, approved by senior management.

A small team needs that count more than a large one does, because each page lands on a third of the team. Once a week, look at how many pages there were, how many of them needed a person, and how long the first acknowledgement took. A page that needed nobody is an alert to fix or delete. A slow acknowledgement is a reason to look at the ladder and the paging channel before it is a reason to look at the person.

What this does not fix

Two people are still on call every other week, and no ladder changes that. What is left is to cover fewer hours or fewer systems and tell customers which, and to pay for the nights. The Workbook's position is short: "On-call usually implies some amount of out-of-hours work. We believe this work should be compensated."

Four is the first team size where one person on call at a time meets the 25% line. If on-call is wearing down a team of three, that is a real argument for the fourth hire, and the table above is how to make it.

Sources

Google, Site Reliability Engineering, chapter 11, Being On-Call.
Google, The Site Reliability Workbook, chapter 8, On-Call.
GitLab, Postmortem of database outage of January 31, 10 February 2017.

Where Uptimepage fits

I build Uptimepage. It checks your services from several regions, and when one goes down it pages whoever is on call.

An on-call schedule holds one or more rotations, each on call at all hours or only in the hours you give it. Daily and weekly handoffs stay at the same local time across daylight saving. Overrides cover exact dates. Each person picks the channels that page them, and the schedule list names anyone on a rotation whom no channel can reach. An escalation policy is a ladder of levels with a wait on each, and acknowledging from the chat or email alert stops it. Each person gets a calendar link for their shifts, and the incident reports include mean time to acknowledge.

One gap to know about: there is no phone-call channel yet. SMS, Pushover's emergency priority, Telegram and the chat apps are there, and if you already use PagerDuty you can put it on a level. The incident docs have the details. It is open source under AGPL, so it can also run on your own server.

Whatever you use, write down the answer to one question. At 03:00 tonight, whose phone rings, and whose rings next if they do not answer?

Now the question for the comments: how does a team of two or three cover nights where you work? I would like to hear from teams that leave some hours uncovered on purpose and say so on their status page.

System Design for DevOps Personnel Is More Than Knowing Kubernetes, AWS or Terraform.

Tue, 29 Sep 2026 05:30:47 +0000

Recently, I interviewed for a Staff Site Reliability Engineer (SRE) role, where I encountered some interesting system design challenges around building a scalable, highly available, cloud-native application.

The discussion reinforced an important lesson: Knowing individual technologies is one thing; understanding how to bring them together to solve real-world distributed systems problems is another.

Here are 5 key technical takeaways from the discussion that I'm revisiting and deepening my understanding of.

1. Cloud Networking: AWS vs. GCP

One interesting discussion was around designing DNS, VPCs and networking for a globally distributed application running on Amazon EKS.

A key distinction:

AWS VPCs are regional.
GCP VPCs are global resources.

For a multi-region AWS architecture, we need to consider separate regional VPCs, public and private subnets, cross-region connectivity using Transit Gateway or other appropriate networking patterns, and Route 53 for global DNS routing.

Key takeaway: Cloud architecture requires understanding the fundamental differences between cloud providers, not just their equivalent services.

2. Designing for Burst Traffic and High-Throughput Ingestion

How would you handle a user uploading 1,000 files in a single action without overwhelming Kubernetes application pods?

Autoscaling with Karpenter and applying rate limits can help, but neither alone solves the problem of sudden bursts in application-level traffic.

A more scalable approach:

Generate S3 presigned URLs through an API.
Allow clients to upload files directly to S3.
Trigger asynchronous processing through S3 events and SQS.
Use KEDA to scale Kubernetes worker pods based on queue depth.

This separates the ingestion data plane from the application control plane and helps absorb traffic spikes without overwhelming application workloads.

Key takeaway: Before scaling infrastructure, ask whether the workload needs to pass through your application in the first place.

3. Event-Driven Autoscaling with KEDA

Traditional Kubernetes HPA commonly relies on resource metrics such as CPU and memory.

However, for asynchronous workloads, these metrics may indicate a problem only after a backlog has already accumulated.

KEDA enables event-driven autoscaling using signals such as SQS queue length, allowing worker replicas to scale according to pending work, including scaling to zero where appropriate.

Key takeaway: Autoscaling should be driven by workload characteristics, not just infrastructure utilization.

4. Container Security and GitOps

Some areas that remain essential for production-grade Kubernetes platforms include:

Multi-stage Docker builds and minimal container images.
Running containers as non-root users.
Read-only root filesystems wherever practical.
Container image vulnerability scanning using tools such as Trivy or Grype.
Workload identity and least-privilege access using IAM Roles for Service Accounts (IRSA) on EKS.
CI/CD pipelines with automated testing, security scanning and image signing using Cosign.
GitOps-based deployments using Argo CD and Helm.

Key takeaway: Security and deployment reliability should be built into the software delivery lifecycle, rather than added after deployment.

5. Observability: From Reactive Monitoring to Proactive Reliability

Monitoring CPU and memory utilization is important, but infrastructure metrics alone don't tell us whether users are experiencing a degraded service.

A more comprehensive observability strategy includes:

The Four Golden Signals:

Latency, Traffic, Errors and Saturation.

Distributed tracing using OpenTelemetry.
Service-level indicators (SLIs) and service-level objectives (SLOs).
Synthetic monitoring to validate critical user journeys.
Actionable alerts based on user impact and error budgets.

Key takeaway: The goal of observability is not simply to detect infrastructure failures, but to understand and prevent user-facing reliability issues.

My biggest takeaway

A Staff-level SRE is expected to think beyond individual tools and commands.

It involves understanding trade-offs, identifying bottlenecks, designing for failure, managing cost, ensuring security and making architectural decisions that support long-term scalability.

There are always new technologies to learn, but the ability to reason about distributed systems and explain why a particular design decision makes sense is equally important.

I'm using this experience to strengthen my system design fundamentals and deepen my understanding of large-scale cloud-native architectures.

Continuous learning is an essential part of engineering growth. Every challenging discussion is an opportunity to get better.

What Kubernetes Chaos Engineering Taught Me: Recovery Isn't Binary

Mon, 14 Sep 2026 16:37:21 +0000

A pod dying and a system recovering are not the same thing. That gap is where most of the interesting failures actually live, and it's what pushed me to build fault-sentinel, a lightweight, Kubernetes-native chaos engineering CLI written in Go.

Kubernetes gives you self-healing by default: a container crashes, the kubelet restarts it; a node degrades, the scheduler moves work elsewhere. What that default behavior doesn't tell you is whether anything using the system noticed. A pod's status can read Running again in seconds while in-flight requests during the gap were silently dropped. Recovery, as reported by kubectl get pods, and recovery, as experienced by whatever depends on that pod, are two different claims.

fault-sentinel exists to test the second one directly, by deliberately breaking things and watching what actually happens, rather than trusting that a clean deployment means a resilient one.

Why build a new tool instead of using Chaos Mesh or Litmus

Established chaos engineering platforms do this more comprehensively than a personal CLI ever could. I built fault-sentinel anyway, for a specific reason: I wanted to understand the mechanics from the inside, not just apply someone else's custom resource and trust the abstraction. Writing the pod-eviction logic directly against client-go means I know exactly what a graceful deletion does at the API level, not just that a CRD exists for it.

This is a personal learning project, not a replacement for production-grade chaos tooling, and it doesn't try to be.

Architecture

graph TD
    CLI[fault-sentinel CLI] -->|client-go| KubeAPI[Kubernetes API Server]
    KubeAPI -->|Pod Termination| PodTerm[Target Workload]
    KubeAPI -->|SPDY Exec| Exec[Container Exec Session]
    Exec -->|tc netem| NetLat[Network Delay Injection]
    CLI -->|/metrics| Prom[Prometheus]
    Prom -->|Dashboards| Grafana[Grafana]

The CLI runs standalone, no persistent daemon, no custom resource definitions. It only holds cluster access for the duration of an active experiment, not as a standing permission.

The three experiments

Pod termination: selects a pod by label selector and deletes it via the CoreV1 API directly, rather than shelling out to kubectl delete, to watch how fast the ReplicaSet controller notices and replaces it.

Network latency injection: uses Linux Traffic Control (tc netem) inside the target container's network namespace to inject artificial delay, with automatic cleanup once the experiment window ends. This requires the NET_ADMIN Linux capability on the target container, which is itself a real trade-off: you're loosening a workload's security posture slightly, purely so it can be a valid subject for chaos testing.

CPU stress: generates bounded compute load across a configurable number of worker goroutines, to observe how a workload behaves under compute pressure.

Proving it, not just building it

I ran each experiment against a live cluster and checked the actual outcome, not just that the command executed without error:

Killing a pod by label selector and confirming the ReplicaSet controller replaced it, watched via kubectl get pods -w
Injecting a 250ms delay into a running pod's network namespace and confirming both the delay and its removal afterward
Running CPU stress and confirming it stayed bounded within the container's own resource limits, rather than affecting the node

The CI pipeline runs golangci-lint, unit tests with Go's race detector, a multi-stage Docker build, and a Trivy vulnerability scan on every push. The race detector matters specifically here: a chaos tool with its own concurrency bugs while testing other systems' concurrency behavior would defeat the entire point.

Observability

fault-sentinel exposes Prometheus metrics on :8080/metrics during every run: chaos_experiments_total (by type and status), chaos_injected_faults_total (by target and fault type), and chaos_experiment_duration_seconds. These let you correlate the exact window of an injected failure against whatever else you're watching, latency, error rates, endpoint convergence, in Grafana.

Security model

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fault-sentinel-role
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "delete"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "get"]

A chaos tool is, by definition, something that deletes pods and manipulates network behavior. That's real trust to grant a piece of software, so RBAC is scoped as narrowly as the tool can function on: list and delete on pods, nothing cluster-wide, nothing touching secrets or unrelated resources.

What this project didn't try to be

This isn't a replacement for Chaos Mesh or Litmus, and I'm not claiming it should run against a production cluster. It has no scheduled or recurring experiments, no web UI, no multi-cluster orchestration. What it does have is a small, complete surface area I understand fully, because I built every part of it myself, which was the actual point.
The lesson that stuck with me most: recovery isn't binary. A pod coming back after being killed isn't the same as the system recovering. You have to watch what happens to in-flight requests and error rates during the gap itself, not just confirm the pod count went back to normal afterward. That's the difference between "it healed" and "it healed in a way that didn't hurt anyone using it."

Repo: github.com/aashiruu/fault-sentinel
Author: Okikiola Ashiru — Cloud & DevOps Engineer

Agent traffic does not grow like human traffic

Fri, 11 Sep 2026 14:36:32 +0000

GitHub published its August availability report this week, and one sentence in it is worth more than the rest of the document: "our shared infrastructure services have not kept up with our month-over-month actions growth and peak load."

Five incidents in a month. The worst, on August 17, peaked at a 56% front-door request failure rate and touched more than 29,000 organizations, with roughly 4.8 million requests failing or crawling. On August 26, more than 20% of Actions run starts failed or were delayed, and that is the one GitHub attributes directly to growth outpacing shared services.

I want to be careful here, because it would be easy to read this as a GitHub problem. I do not think it is. I think it is the first large public example of a capacity model meeting a class of traffic it was never built to forecast.

Human traffic has a denominator

For most of my career, capacity planning had a number I could point at outside the system. Working on logistics APIs, the driver was parcels. Volume rose with commerce, peaked in a season everyone could name months in advance, and grew year over year in a band narrow enough to argue about. We disagreed about the multiplier. We did not disagree about the shape.

Internal systems worked the same way. Request volume tracked headcount, customer count, or transaction count. If you wanted to know what next year looked like, you asked someone in finance what they were forecasting and did arithmetic.

Agent traffic breaks that. The driver is not how many people you have. It is how many loops those people are running, and that number lives in a config file, a workflow matrix, or a retry policy. One engineer changing a concurrency setting on a Friday afternoon can move your load more than a quarter of hiring does. There is no external denominator to anchor the forecast to, which means the forecast is now a function of your own configuration surface.

Retries are the amplifier

The August 6 incident has a detail worth sitting with. A latent bug caused runners to retry revoked jobs instead of picking up valid work. That is a small bug. In a system already under pressure it is an accelerant, because retry traffic arrives exactly when there is no headroom left to serve it.

This is the failure shape I expect to see more of, not less. Agents retry. That is most of what makes them useful. The same property that lets an agent recover from a transient failure lets a fleet of them turn a brownout into an outage, and it does not require anybody to have written bad code. It requires a retry policy that was perfectly reasonable back when one caller had it.

I wrote a few days ago about idempotency being the guardrail that saves you at 2am. This is the other half of that argument, one layer down. Idempotency keeps a retry from duplicating an action. It does nothing about a retry storm consuming the capacity everything else needs.

The damage lands on the shared path

August 17 is the incident I would hand to a platform team. The proximate cause was a traffic surge past load balancer limits, but the reason it reached 29,000 organizations is that the degradation landed on a shared authentication path. Agent traffic did not take down the agent features. It took down issues, pull requests, the APIs, and webhooks.

That is the part worth checking in your own system this week. Not whether your agent workloads have capacity, but what they share with everything else. Connection pools, auth, DNS, a service mesh sidecar, one database primary. The blast radius of agent traffic is the set of things it shares, and most architecture diagrams do not draw that.

What I would actually do

Start with attribution, because you cannot forecast what you cannot count. If your logs cannot tell you what share of load came from an agent, which agent, and acting for whom, then your capacity model is guessing and so is your next postmortem. This is the same instrumentation argument I make about auditability, and it turns out to pay for itself twice.

Then give humans a lane. If agent workloads and interactive traffic share a pool, the agents will win, because they are patient and there are more of them. Separate queues, separate pools, or at minimum a priority that favours the request with a person waiting on the other end of it.

And treat concurrency settings as production configuration. If a change to a workflow matrix can double your peak, it deserves the same review as a schema migration.

One thing I got wrong

I have built capacity models off headcount growth that were confidently wrong, on a system where the real driver turned out to be a nightly batch job somebody had quietly parallelised. The model was fine. The denominator was wrong. I did not find that out during planning. I found it out during an incident, which is the expensive way to learn it.

The lesson I took from that is the one I would apply here. The question is not how much traffic you expect. It is what decides how much traffic you get, and whether that thing sits inside your system or outside it. For agent workloads it is inside. That is new, and it is worth re-planning around rather than scaling into.

Figures above are from GitHub's own availability report for August 2026.

AWS's Own Dashboard Could Not Say It Was Down. Four of Our Monitors Had the Same Bug.

Thu, 27 Aug 2026 19:32:33 +0000

A signal has three states and most code stores two. When “cannot measure” is written with the same value as “fine,” the alarm branch becomes unreachable in exactly the situation the alarm exists for.

On 7 December 2021, the operations teams inside AWS lost the ability to see their own network. Not the ability to fix it. The ability to see it. The company's post-event summary says so in plain language: congestion "immediately impacted the availability of real-time monitoring data for our internal operations teams, which impaired their ability to find the source of congestion and resolve it" (Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region, "Issue Summary" section). While customers refreshed a status page, the people responsible for one of the largest computing regions on earth "relied on logs to understand what was happening" (same document, same section). Even the public dashboard was caught in it: "the networking congestion impaired our Service Health Dashboard tooling from appropriately failing over to our standby region" ("Event Communication" section).

The lesson usually drawn from that day is deployment advice. Monitor your monitors. Host the status page somewhere else. Both are sound, and both miss what makes the incident interesting. The monitoring did not miss the failure. It could not report the failure, because the failure and the reporting path shared a dependency. Those are different defects, and the second one has a signature you can find in your own code without waiting for an outage.

Three states, two values

A monitoring signal has at least three states: measured and bad, measured and fine, and could not measure. Most of the code we audited encodes the third state with the same value as the second. When that happens, every branch that would have raised the alarm becomes unreachable in exactly the situation the alarm exists for. Unreachable in the strict sense: while the sensor is the thing that broke, no input can produce the alarm value.

You can check for this by construction rather than by misfortune. For each signal you own, ask one question. What comes out when this cannot be read, and can a healthy system produce that same value?

We asked that question about our own system

Over one working night we audited four independent monitoring and status signals in a single codebase we operate. All four collapsed "unknown" into "fine," each in its own way.

signal	what "I cannot tell" produced	consequence we measured
a deploy-status probe	a lag warning with an empty count	fired on 100% of runs for 28 days; the "no lag" branch was unreachable
a fleet health dashboard	the verdict `STALLED`	reported a fully up-to-date system as stalled
a simulation's scarcity input	`0.0`	a region with no data read identically to an abundant one
an item-identity lookup	a guess derived from the display name	a garment classified as a liquid container

The 28-day figure is the interval between a format change we can date and the night of the audit. For four weeks a warning fired on every single run, which meant it warned about nothing, which meant nobody read it.

The arithmetic version of the defect is the sharpest, so we shipped it as a runnable script. Take a health score averaged from three weighted inputs, alarm threshold 0.70. Two of the three inputs are dead and report the calm value, 0.0. The one live input is screaming at 0.95. The script prints:

naive sum over all three   : 0.285   -> HEALTHY
renormalised over LIVE only: 0.950   -> ALARM

Identical inputs. And 0.285 is a ceiling, since 0.95 is the largest value the live input can report. With two legs dead and voting calm, the alarm was never rare. It was impossible. Renormalising over the inputs that can actually be seen asks the question the operator is actually asking: given what I can observe, how bad is it?

The one that got it right

The same codebase contained a fifth signal that had the answer built in, and it convinced us the fix is a value rather than a discipline. Its unknown state maps to 0.20, on a scale where its two calm states are 0.10 and 0.05. Nothing else about it is clever. An unrecognised input simply produces a number that no quiet state can produce, so the downstream branch for "something needs a look" stays reachable at all times. It asks nothing of anyone's memory and no change of habit. The distinguishable value does the remembering.

That is the difference between this and one more essay telling engineers to be more careful. The four broken signals were written by people being careful. The fifth was written so that carefulness was not load-bearing.

Honest scope

Our data is one codebase, one night, one investigator, and after the first instance we were primed to see the shape everywhere. The four signals are independent subsystems, which is the sample's strength, but this is a report from our own floor and no survey of anyone else's software.

The check

Skip the moral. Run the audit. For each signal you operate, ask what it emits when it cannot read, and whether a healthy system can emit the same thing. If the answers collide, you do not have a monitoring gap. You have an alarm branch that is unreachable precisely when it matters, and one distinguishable value fixes it.

AWS's operators spent that December morning reading raw logs because their telemetry answered "I cannot see" in a voice indistinguishable from silence. Ours answered "fine." Same bug. The only reason we found ours first is that we went looking.

Reproduction: every number we computed comes from repro_unknown_equals_healthy.py, shipped beside this essay. Run it and disagree with the arithmetic directly.

Sources: Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region (aws.amazon.com/message/12721), sections as cited in-text. All other figures are our own measurements, per the reproduction script and the scope note above.

An audit trail has the same three states as a monitor.

Everything above turns on one value: what a signal emits when it cannot see. The same question applies to the record a system leaves behind, where “no entry” and “nothing happened” are the pair that get confused. Chain of Consciousness is a tamper-evident record written as the work happens rather than reconstructed once somebody asks, so a gap in the record reads as a gap instead of as a quiet success. It does not make a system behave. It makes the difference between unobserved and fine a thing you can check.

Hosted Chain of Consciousness · Verify a record

pip install chain-of-consciousness · npm install chain-of-consciousness

Originally published at vibeagentmaking.com.

Persister Cells: The Bug That Survives Every Retry Isn't Resistant, It's Dormant

Tue, 04 Aug 2026 03:36:09 +0000

Originally published on Vibe Agent Making.

In 1944, the physician Joseph Bigger hit a Staphylococcus culture with penicillin and watched almost all of it die. A tiny remnant survived, which is unremarkable. The interesting part: he regrew the survivors and hit them again, and they died just as thoroughly as the first time. The survivors hadn't become tougher; their offspring were exactly as vulnerable. Whatever let that remnant live wasn't heritable and wasn't a matter of overpowering the drug. Bigger called them "persisters," and the word is precise: they didn't resist the penicillin, they persisted through it. That distinction is the whole story of the most maddening class of bug you'll ever chase: the one that survives every restart, every retry, every careful remediation, and comes back identical, as if you'd done nothing.

The drug needs a moving target

Most antibiotics don't work like poison on a fixed structure. They sabotage growth: beta-lactams jam cell-wall construction (the cell tries to divide and bursts); quinolones corrupt DNA replication. The killing requires growth. A persister cell stops moving: it goes dormant, throttles its metabolism, does nothing, and a cell doing nothing presents nothing for a growth-targeting drug to attack. The 2019 Nature Reviews Microbiology consensus paper calls the targets in non-growing cells "recalcitrant": not defended, just absent from the fight. The drug washes harmlessly over the sleeping handful; treatment ends, they wake, and they rebuild the whole population. The infection relapses, genetically identical and fully susceptible. The survivors are not the strong ones. As Kim Lewis put it in 2007, the tolerance "is not inherited and is reversible. Persisters are not mutants, but rather dormant cells." Strength was never the variable. Activity was.

The shape of the failure tells you what it is

Persistence leaves a fingerprint. Plot survivors against time under treatment: a resistant strain gives a flat-ish line (it grows under the drug), but a persister population gives a two-phase curve, the bulk dying fast (a steep cliff) and then a long, stubborn tail the drug can't finish. That's the biphasic killing curve, and it says: not resistance, dormancy.

The vocabulary is load-bearing. Resistance: a heritable change that lets a cell grow at drug concentrations that would kill its ancestors (it defeats the drug and passes it on). Tolerance: a whole population dies more slowly without growing under the drug (buys time, not victory). Persistence: a special case of tolerance, a subpopulation that survives far better than the rest, the tail of the curve. The one-liner from the 2019 paper: resistant mutants can divide during treatment; persisters can only survive. One fights, the other waits.

Your retry is the antibiotic

When a fault survives every restart and retry, the reflex is to call it "hard", a deep, well-defended bug resisting your fix. Sometimes. Far more often it isn't resistant, it's dormant, and it survives for Bigger's reason: your remediation only hits the part of the system that's active.

The cleanest documented case is anthropics/claude-code issue #23081: a connection drops, retry logic kicks in, and the retry keeps failing the same way because it reuses stale connection state, the HTTP/2 pool, the cached TLS session, the DNS entry. The "fresh" attempt reaches back into the pool and re-serves the poisoned connection. The fix isn't to retry harder; it's to destroy and recreate the client (flush the pool and TLS cache) before retrying. Your retry was the antibiotic; it hit the active path with full force; the dormant connection survived precisely because it was doing nothing when you struck.

The shape is everywhere: the pool that hands back a dead connection after idle time ("intermittent, only after a period of idleness when traffic picks back up"); the lagging replica that's quiet during your fix on the primary and then reconnects and re-seeds stale state; the cached bad config, the queued-but-not-running job, the sleeping cron, the in-memory singleton holding a pre-fix value. Each is a place a fault can be dormant during your remediation. And the tell: the restart that helps but doesn't last. Microsoft's docs on connection-pool exhaustion say restarting "helps clear stale connections... typically a temporary solution rather than addressing root causes", which is the biphasic curve in plain English. A fix that works beautifully for ten minutes and relapses under load isn't flaky; it's a persister waking up.

The misdiagnosis is the whole trap

Resistance and persistence need opposite responses, and mixing them up keeps you fighting forever. When a fix doesn't stick, the instinct is to escalate force: retry harder, restart more aggressively, add retries to the backoff. That's exactly right against a resistant fault, one that actively defeats your fix and spreads under it. But against a persister, escalated force is worse than wasted, it's invisible: force tuned to the active population can't touch what's asleep, by definition. Triple your retries against a stale pool and you re-serve the poisoned connection three times as fast. The persister isn't beating your fix; it's ignoring it. Hammering the active symptom while the dormant pool waits you out is the most common way good engineers lose days to a bug a five-minute cache flush would have ended.

So the question is not "how do I hit this harder?" It's "what part of this system was asleep while I was fixing it?" And there's a sting: the Cell (2014) work links persistence to a higher risk of true resistance emerging during treatment, because the dormant survivors buy the time evolution needs. The software echo is faithful: the cached corruption that sits long enough to become the trusted value; the stale replica that, because it happened to be up, gets promoted to source of truth. Leave the reservoir unflushed and a recurring fault gets baked into structure. Clear it before it sets.

The clinician's discipline

Read the curve before you escalate. Kills-most-then-a-stubborn-fraction-returns is the diagnosis: a persister, not a resistant bug. More force is the wrong instrument.
Enumerate the dormant reservoirs before you re-fix. Explicitly, every time: caches, queues, idle pools, sleeping crons, offline/lagging replicas, persisted session state, in-memory singletons. Anywhere the fault could be asleep during remediation.
Flush, don't hammer. Invalidate caches, drain queues, destroy and recreate the pool, quiesce and rebuild idle replicas. Eradicate the reservoir, don't escalate force on the visible symptom.
Quiesce before you fix (the most clinical move). Stop or drain the dormant population first so it can't re-seed mid-repair: take the lagging replica out of rotation, drain the queue, empty the pool, fix the quiet system, then bring the rest back. Otherwise you apply the same fix four times.
Verify across the relapse window. Don't declare victory at the restart; confirm the fault doesn't return when load resumes and the idle parts wake, the moment the "antibiotic concentration drops." That's finishing the full course instead of stopping when you feel better, which is exactly the behavior that breeds persisters.

Survival isn't strength. The fault that laughs at every retry usually isn't tougher than your fix, it's asleep during it. Stop hitting the part that's awake. Find the part that's dormant, and flush the reservoir.

An AI agent that keeps relapsing into the same failure is rarely fighting your fix; it's reusing dormant state the retry never touched, a cached belief, a stale piece of context, a poisoned memory carried across attempts. You can only find the reservoir if you can read what the agent actually did each time. Chain of Consciousness is that record: a tamper-evident chain of an agent's reasoning, tools, and actions across attempts, so you can see the biphasic curve in its behavior, spot the state that survived the restart, and flush it instead of hammering the part that was already awake. pip install chain-of-consciousness / npm install chain-of-consciousness.

How I stop one bad probe from waking you at 3 a.m.

Wed, 29 Jul 2026 15:36:57 +0000

TL;DR. The most common false alert in uptime monitoring is one probe location having a bad network day. I treated that as a design requirement from day one, so the checking pipeline has two gates: a region only counts as down after it fails the same check twice in a row, and an incident only opens when enough regions agree. A region that goes silent leaves the vote instead of counting as down. One bad location cannot page you. The original post carries the three figures below as live widgets you can drive yourself.

The alert that was nothing

The phone buzzes at 3 a.m. "Your API is down." You get up, open the laptop, and everything is green. The site was fine the whole time. One monitoring server, somewhere far away, had a bad network moment and sent an alert about nothing.

This is the false alert everyone in monitoring knows. It costs you sleep, and then it costs you something worse: the next alert feels less serious. The day a real outage comes, you look at your phone and think "probably nothing again."

I knew this failure mode before I wrote the first line of the scheduler, so it became a requirement, on the same level as "checks must run on time." The rule I started from: a single bad location must never be able to page a customer. The whole checking pipeline, from how probes report to how incidents open, is shaped by that rule.

The shape it took is two gates. A failure has to pass both before anyone gets paged.

Gate one: fail twice, from the same place

One failed check proves very little. Networks lose packets, routers restart, and sometimes a DNS answer arrives a second too late. All of that can make one check fail while your site is healthy.

So a region only counts as down after it fails the same check twice in a row. Not two failures somewhere in the system: two failures from that one region, back to back. A single blip resets to zero on the next good check.

One blip resets the counter. Two failures in a row from the same region earn that region a single vote.

The count is a setting on each monitor. Two is the default. For a very sensitive check you can set it to one, and for a noisy target you can raise it. Checks run on the monitor's own interval, so with a one-minute check the second failure arrives about a minute after the first. That minute buys you a lot of silence for a very small delay.

Gate two: the vote

Passing gate one gives a region exactly one vote. Nothing more.

Say you check from five regions and one of them has a bad ISP day. It fails twice in a row and votes "down." The other four keep passing. One vote against four is not enough, so nothing happens and nobody gets paged.

One region against four cannot page you. Three of five can.

The default rule is a majority: more than half of the reporting regions have to agree. You can change it per monitor. "Any" opens an incident on the first confirmed region, which is useful when you want the earliest possible signal and can accept some noise. "All" waits until every region agrees, for a service that only matters when it is unreachable everywhere. Or you pick a fixed number, like two regions out of whatever you assign.

A monitor checked from a single region behaves the same under every rule. The vote only starts to protect you when you add a second location, and it gets better with a third.

Silence is not failure

Here is the part that took the most care to get right.

The probe regions push their results to the control plane, the brain of the system. The brain never calls out to ask "are you alive?" during a vote. It just counts the results that arrived in the last few check cycles.

That gives silence a clear meaning. A region that lost its own connection cannot send anything, so it simply is not in the vote. It is not a down vote and not an up vote; the region is out until it reports again. The majority recalculates over the regions that still speak. Five regions where one goes dark becomes a vote of four.

A silent region is not a down vote and not an up vote. It leaves the vote until it reports again.

I think this is the only honest reading. A region that cannot reach the brain is telling you nothing about your site. Treating its silence as "down" would turn every probe outage on the monitoring side into a fake incident on yours.

And it works in both directions. Missing data never opens an incident, and it never closes one. An open incident only closes when the down votes fall below the threshold and at least one region shows a real run of passing checks. Recovery needs proof, the same way failure does.

Who watches the watchers

There is one gap left. If every region that covers your monitor goes dark, the vote has nobody in it. Your site could burn down and no incident would open, because no data means no votes.

For that case there is a separate signal, one level below the checks. Every probe sends a small check-in to the brain on its own schedule, a heartbeat that has nothing to do with your monitors. When the last live probe covering a monitor goes stale, you get a different message: "NO DATA: monitoring interrupted, no check results received." It is honest about what it knows: the service cannot see your site right now. It does not claim your site is down, because it has no idea.

When probing returns, you get a "monitoring RESUMED, receiving check results again" note, and the vote picks up where it left off.

One more detail I like. If a large share of all monitors goes silent at the same time, that pattern is almost never a thousand customer problems. It is one problem, mine. In that case the system alarms me and holds the customer notices, so my bad day does not become spam on yours.

What this means for your setup

Three practical points follow from this, whatever tool you run.

Check from more than one region. The vote cannot protect you with a single location. Two regions with the majority rule means both have to agree, which already kills the classic false alert. Three gives you two out of three, which is the right balance for most services.

Leave the confirmation count at two unless you have a reason. It is the difference between "a packet got lost" and "this endpoint is failing."

Pick the rule to match the monitor. A payment API deserves majority or even any. An internal tool nobody uses at night can wait for all. The setting is per monitor, so you do not have to choose one policy for everything.

The goal of all this machinery is boring: when your phone buzzes, it is real. Everything else is plumbing.

Key takeaways

One failed check from one place proves nothing. Confirm it from the same region before that region gets a say.

Give each confirmed region one vote, then require agreement. A majority of reporting regions is a good default.

A silent region must leave the vote, not join the down side. Otherwise every probe outage becomes a fake incident.

Missing data must not close an incident either. Recovery needs a real run of passing checks.

Cover the all-dark case with a separate heartbeat and a message that says "we cannot see your site," not "your site is down."

Two regions kill the classic false alert. Three is the sweet spot for most services.

The three figures above are live widgets in the original post: you can change the confirmation count and the vote rule and watch where the line falls. The whole project is open source, so the scheduler and the incident writer are there to read.

How does your setup handle a probe that goes quiet? I have seen tools count silence as down, and I would like to know if anyone has a better reading than "it leaves the vote."

How to write incident status updates that build trust

Tue, 21 Jul 2026 13:45:03 +0000

TL;DR. During an outage your status page has a second job: holding trust while the fix is still in progress. Write plainly. Say what customers feel first, say what you are doing, and promise a time for the next update. Then keep that promise, even when there is no news. The four stages are investigating, identified, monitoring, and resolved.

The idea of an incident commander did not come from software. It came from wildfires. In 1970s California, a run of huge fires pushed fire chiefs to study why large responses fell apart. The cause surprised them. It was not too few firefighters or too little water. It was communication. Different teams used different words and no one shared a plan. The fix was a simple system with clear roles and clear language, now known as the Incident Command System, and on-call engineers still use a version of it today.

Your status page is the communication part of that system. When something breaks, the repair happens in your code. The trust happens on the status page. This post is about the second part: how to write updates that keep people calm while the first part is still in progress.

The words cost real money

An outage is expensive before you write a single word. In ITIC's 2024 survey of more than 1,000 companies, over 90% of mid-size and large firms said one hour of downtime costs them more than $300,000. For 41% of them, one hour costs between $1 million and 5 million. Uptime Institute's outage research points the same way: about one in five recent outages cost more than 1 million, and more than half cost over $100,000.

You do not need to be a bank for this to matter. If your store makes $2,000 a day, one bad hour during a sale can cost more than a slow afternoon, because sales are not spread evenly across the day. The size of the number changes. The shape of the problem does not.

Here is the part teams forget. Some of that cost is the downtime itself. The rest is trust, and trust is where writing helps. Many providers offer an SLA, a promise to stay up with a penalty if they miss it. The penalty is almost always a service credit, which is money back on your bill. A credit refunds your invoice, not your customer's bad afternoon. Good updates cannot bring the service back. They can protect the relationship while it is down.

The four stages of an incident

Most status pages, including the big public ones, use the same four stages. They come from a shared standard, so a customer who has read one status page already understands yours.

Stage	What it means	What to write
Investigating	You know something is wrong. You do not know why yet.	Name the impact and say you are looking into it.
Identified	You found the cause and are working on a fix.	Say what is wrong in plain words and that a fix is coming.
Monitoring	The fix is live. You are watching to be sure it holds.	Say the fix is in and you are checking that it works.
Resolved	The service is back and stable.	Say it is over, and thank people for waiting.

The order matters. Do not jump to identified because you have a hunch. If you say you found the cause and then change your story an hour later, every update after that is worth less. Move a stage forward only when the facts move with you.

The rules of a good update

Five rules cover almost every message.

Start with impact. Say what the customer feels before you talk about servers or databases. "Some payments are failing" is more useful than "we are seeing elevated 5xx errors on the API gateway." Only the first one means anything to a customer.

Share facts, not guesses. Say what you have confirmed. If something is still unknown, say that too. Writing "we do not yet know the cause" is fine. Inventing a cause to sound in control is not. This rule earns its place under pressure: the Uptime Institute's 2025 report found that about 40% of organizations had a major outage caused by human error in the last three years, and 85% of those came from someone not following a procedure. An incident is exactly when tired people guess.

Promise the next update. Give a real time, like "next update by 15:00 UTC" or "another update within 30 minutes." This one line does more than any apology. It tells people they can close the tab and get on with their day.

Add a timestamp and a time zone. "In 30 minutes" means nothing if the reader does not know when you posted. Every update should carry a clear time.

Keep it plain. Short sentences. No blame, no jargon, no backstory. A worried customer reads fast.

A bad update and a better one

Here is a bad update of the kind you have all seen:

We are aware of an issue and our team is working hard to resolve it as quickly as possible. We apologize for any inconvenience.

It sounds polite and says nothing. What is broken? Who is affected? When will I hear more? A better version answers those questions:

Investigating: some customers cannot log in. Sign-ups and password resets are affected too. Our team is looking into the cause now. Next update by 14:30 UTC.

Same length. Only one of them lets a customer decide what to do next.

How often to post

Set the rhythm in your first message and keep it. Every 30 to 60 minutes is a good starting point for a serious outage. The exact number matters less than the promise. If you said 30 minutes, post at 30 minutes, even when the only news is "still working, no change yet." Silence after a promised time reads as "they have lost control." A boring on-time update beats an exciting late one.

Planned maintenance is the same skill, calmer

Maintenance uses three stages: scheduled, in progress, and completed. The difference is that you have time to write it well in advance. Tell people what will happen, when it will happen, and whether they need to do anything. Mark it in progress while the work runs, and completed when it is done. The tone is calmer because nothing is on fire, but the shape is the same: impact, timing, next update.

The mistake almost everyone makes

It is not the wording. It is the second update.

The first update is easy. The outage just happened, and everyone is paying attention. Then the fix takes longer than expected, the team goes quiet, and the page sits at "investigating" for two hours. From the outside, a frozen status page looks the same as a dead company.

The fix is a habit, not a talent. Decide the next-update time in your first message, set a timer, and post again when it rings. If there is no news, that is the update: no change, still working, next check in 30 minutes. The point of the promise is that people stop refreshing and trust you to come back.

Write the message, then give it a home

You can write all of this by hand during an outage. Most people write worse under stress, which is the worst possible time to start from a blank box. That is why we built a free incident update generator: pick a stage, fill in a few fields, and it writes a clean message you can paste into any status page. It runs in your browser and stores nothing you type.

The message still needs somewhere to live. A status page is where customers look when your product will not load, so it should be honest and easy to reach. If you are choosing an uptime number to promise, the uptime SLA calculator turns a percentage into real minutes, and is 98% uptime good shows how small percentages hide large amounts of downtime. For the page itself, a status page you cannot fake covers how to keep it worth trusting.

Common questions

What should the first incident update say? Say what the customer feels, say you are looking into it, and give a time for the next update. You do not need the cause yet. Waiting for the root cause before posting is the most common mistake.

What is the difference between investigating and identified? Investigating means the cause is not confirmed. Identified means you have confirmed the cause and are working on a fix. Do not move to identified based on a hunch.

Should you apologize in an incident update? One short, honest line is fine. Skip the long apology. A clear next-update time does more for trust than three sentences of sorry.

If you have run incidents: what update cadence actually held up for you under a long outage, and did anyone ever complain that you posted too often? I have never seen that complaint, but I would like to know if it exists.

98% uptime sounds high. It means your service is down 7.3 days a year

Fri, 17 Jul 2026 13:24:32 +0000

Cover photo by the blowup on Unsplash.

I build an uptime monitor, so this question lands in my inbox a lot: is 98% uptime good?

For a public website or a paid API, no. For an internal tool or a side project, it is fine. The difference is one division away.

98% looks like a top grade because school taught us that 98 out of 100 is excellent. Uptime does not grade like school. The whole scale for public services lives between 99% and 100%, and serious targets differ only in the digits after the decimal. On that scale, 98% sits at the bottom.

Turn the percentage into time

The allowed failure at 98% is 2%. Two percent of a year is 7.3 days. Two percent of a 30-day month is 14.4 hours. If your shop makes 2,000 USD a day, 98% uptime means you accept about $14,600 of closed-door time per year and the target still counts as met.

Here is the same math for every common target. The last column prices the downtime for that $2,000-a-day shop.

Uptime	Per day	Per 30-day month	Per year	Lost sales per year
90%	2.4 hours	3 days	36.5 days	$73,000
95%	1.2 hours	36 hours	18.3 days	$36,500
98%	28.8 minutes	14.4 hours	7.3 days	$14,600
99%	14.4 minutes	7.2 hours	3.7 days	$7,300
99.5%	7.2 minutes	3.6 hours	1.8 days	$3,650
99.9%	1.4 minutes	43 minutes	8.8 hours	$730
99.95%	43 seconds	21.6 minutes	4.4 hours	$365
99.99%	8.6 seconds	4.3 minutes	52.6 minutes	$73
99.999%	0.9 seconds	26 seconds	5.3 minutes	$7

Two jumps on this table do most of the work in real contracts. From 98% to 99.9%, the allowed downtime drops from 14.4 hours a month to 43 minutes. From 99.9% to 99.99%, it drops from 43 minutes to 4.3 minutes, and that second jump usually costs about ten times more engineering than the first while saving the shop $657 a year.

The shape of the downtime matters

98% per month is 14.4 hours, but the number says nothing about how those hours land.

Thirty minutes of planned maintenance every night at 03:00 adds up to 98% and most users never notice. One 14-hour outage on the day of your product launch is also 98%. Same score, very different month.

So a single uptime percentage is a summary, not the full story. When someone quotes you a number, also ask about the longest single outage and when it happened.

When 98% is enough

Plenty of systems can live at 98% and nobody gets hurt:

An internal wiki. People retry after lunch.
A staging environment. Downtime there is often planned.
A batch job that builds reports at night. It has hours of slack before anyone reads the output.
A home server on a residential connection. Your power company already decided your uptime for you.

The shared pattern: when these go down, nobody loses money and nobody loses trust. Paying for more nines there is waste.

When it is not

A checkout page, a paid API, a login service. Here 98% fails twice. First the direct cost: 14.4 hours a month of failed requests and support tickets. Second the trust cost, which is larger and slower. A customer who hits your outage twice in one week does not check your uptime report. They remember that your service is the one that breaks.

One more trap: an SLA is not uptime. Uptime is the measured number. An SLA is a contract promise with a penalty, and the penalty is almost always a service credit. If a provider misses its 99.9% SLA, you get part of your bill back. Your customers get nothing.

What to aim for instead

99.9% is the default target for customer-facing services for a reason. 43 minutes a month is enough room for a bad deploy and a couple of small failures, and a small team can hit it without heroics. Above that, each nine costs roughly ten times more and most users cannot feel the difference.

If you want to run your own numbers, I keep a free uptime SLA calculator and an error budget calculator on our site; both work without signup.

What target do you actually run in production, and did you pick it or inherit it? I am curious how many teams measured before they promised.

The status page you can't fake: measured uptime, not published

Wed, 15 Jul 2026 13:51:58 +0000

A status page is the one dashboard a company publishes about its own service. It is also the one place where the company has a reason to look good. That is a problem. If the page can be edited to look better than reality, it stops being useful. So when you build or choose a status page, ask one thing: can someone hide a real outage on it?

The short answer: a status page you can trust builds its uptime bar from real checks, not from the incidents a person chose to publish.

Two layers, and only one is yours to edit

A good status page has two parts. The first part is the measured data: the green and red timeline, the 90-day bar, the uptime percent. The second part is incidents: the notes a person writes to explain what broke, what they are doing, and when it is fixed. On the page they sit next to each other and look the same. They are not the same, and mixing them is where trust gets lost.

The measured data answers one question: what did the checks see? The incident notes answer a different one: what does the team want to say about it? The first is a fact. The second is a story. A status page you can trust lets the team write the story, but keeps them away from the facts.

The story layer also includes the postmortem, the write-up you post after an outage. A postmortem is honesty you add on purpose. You explain what broke and why, because you choose to. The bar works the other way. It shows the failure on its own, whether you write anything or not. You control the story. You do not control the facts.

The trap: a bar that only shows what you published

The common mistake is to build the uptime bar from incidents. It feels natural. You already open an incident when something breaks, so why not color the timeline from incidents too? Now the bar turns red only where an incident exists and is marked public.

The problem comes on the day an outage has no public incident. Maybe no one published it. Maybe a setting was wrong. Maybe the monitor was added to the page after the outage, so its incident was saved as private and never checked again. In every case the timeline shows green over a real red day, and it does this quietly. The uptime number goes up. The customer sees 100 percent over a week they remember as broken.

One version of this trap is easy to build by accident, so it is worth explaining. You decide "is this incident public?" once, at the moment the incident opens, based on whether the monitor was on a public page right then. Then you save that answer and never look again. In the code it looks like a live check. It is really a photo taken one time. Move the monitor onto the page a day later, and its past outages stay hidden, because the photo was taken before the monitor was there. Any yes or no mark that is set once and then trusted forever has this problem.

The fix is a rule, not a switch

The bar should come from the measured data. It needs only one rule to stay calm: wait for confirmation before you count downtime. You do not want one failed check from one place to turn a whole day red, because networks are noisy and one bad checker is not an outage. So you wait for agreement: more than one region failing at the same time, for more than one check in a row. That one rule is enough to keep a short blip from becoming a red day.

One thing matters here: the same rule feeds both your alerts and your uptime bar. If the rule that wakes your on-call person is the same rule that colors the timeline, the two can never tell different stories. The moment you add a second rule just for the bar, it will drift from the first, and the page will disagree with itself. On-call gets paged, but the public history says everything was fine.

Publishing stays where it belongs, on the notes. The team decides whether to write an incident, what to say, and when to post the all-clear. They do not decide whether last Tuesday was down. The checks already decided that.

Test the status page you already have

You can check any status page, including your own, in a few minutes.

Add a monitor to a page after it has already had an outage. Does the history show the outage, or does it start clean from the day you added the monitor?
Take a real incident and unpublish it. Does the bar keep the red day, or does the day turn green?
Make one region fail for one second. Does the whole day go red, or does the bar stay calm?

A page you can trust shows the outage in the first test, keeps the red in the second, and stays calm in the third. A page that fails these is usually not lying on purpose. It just built its timeline on top of what people chose to publish, and that always has holes.

Where this leaves you

This is how I built it into Uptimepage: the 90-day bar and each status light come from confirmed downtime, measured across regions with a confirmation rule, not from what someone chose to publish. Incidents and postmortems are the layer you write by hand. It is AGPL and open source on GitHub, and there is a longer write-up on the internals if you want the Rust side.

You should not be able to fake your uptime. You should not be able to fake it by accident either. The bar is a measurement. Keep it one.

How does your status page compute its uptime bar, from checks or from published incidents? Curious what people run and whether anyone has been bitten by the frozen-flag version of this.

Error budgets, explained: SLOs, burn rate, and when to stop shipping

Mon, 13 Jul 2026 14:13:45 +0000

If you have ever seen a team argue about whether they can ship a risky change, an error budget is the tool that ends the argument with a number. Here is how it works, with the formulas and a simple way to turn it into alerts.

What an error budget is

An uptime target has a second number hidden inside it. Say you promise 99.9% uptime. You are also saying that 0.1% is allowed to fail. Put that 0.1% into real time, and that is your error budget: the downtime you can have before you break the promise.

This changes how you look at downtime. It stops being a mistake to feel bad about and becomes a budget you can spend: on a risky deploy, on a slow service you depend on, or on a migration. When the budget runs low you slow down. When it is healthy you can move fast.

The formula

An SLO is your reliability target, for example 99.9%. The gap to 100% is the failure you are allowed. Multiply that gap by the length of the window, and you get real time you can spend.

error budget = window x (1 - SLO)

A 99.9% SLO over a 30-day month is 2,592,000 seconds times 0.001. That is 43 minutes and 12 seconds. That is the whole budget for the month, shared across every incident, not a fresh 43 minutes each day.

So three nines is not "never go down". It is a 43-minute budget each month. Every extra nine costs about ten times more engineering, for downtime that most users never notice. This is why Google says that 100% is the wrong target for almost every service.

Burn rate is the speed

The total tells you how much you can spend. It does not tell you how fast. Two services can both sit at 99.9% for the month: one loses the budget slowly, the other loses it all in a single bad hour. Burn rate tells them apart.

burn rate = (1 - measured) / (1 - SLO)

A burn rate of 1x spends the whole window exactly. 2x spends it in half the time. Below 1x, you finish the month with budget left. Above 1x, the rate tells you the deadline:

budget runs out in = window / burn rate

At 2x, a 30-day budget is gone in fifteen days. At 14.4x it is gone in about two days. That same 14.4x spends 2% of the budget in one hour, which is the level most fast alerts are set to.

Turning burn rate into alerts

One threshold is not enough. It either alerts too late or it sends too many false alarms. The common fix uses two windows: a long one to confirm the problem is real, and a short one to clear the alert quickly once you fix it. Both have to be burning for the alert to fire. For a 30-day budget, these are the usual settings:

Fast page: 2% of the budget in 1 hour (with a 5-minute short window). This is a 14.4x burn.
Page: 5% in 6 hours (30-minute short window). This is a 6x burn.
Slow ticket: 10% in 3 days (6-hour short window). This is a 1x burn.

The fast page catches a sudden outage. The slow ticket catches a slow problem that would still use up the whole month if nobody looked.

The rule is the point

The math is the easy part. The value comes from a rule you agree on before anything breaks. The rule is simple. When the budget runs out, risky launches stop, and the team works on reliability until the budget grows back. While the budget is healthy, you ship and you take the risk.

One more rule keeps it fair. If you never spend your budget, your SLO is too strict, and you are paying for reliability that no user asked for.

Try the numbers

I turned these formulas into a free, no sign-up error budget calculator. You enter an SLO and your measured availability, and it shows budget spent, budget left, burn rate, and a burn-down chart. There is also an uptime SLA calculator with the full downtime-per-nine table.

How does your team set its SLOs and alert thresholds? I would like to hear what works in practice.

How Intuit Is Rethinking Reliability in the Age of AI-Generated Code

Mon, 06 Jul 2026 13:00:00 +0000

AI is shipping code faster than most engineering teams know how to operate it. At Intuit, that tension is front and center — and Akshay Pratinav, Senior Staff Software Engineer, is helping figure out what to do about it.

Pratinav works on AI-driven incident management and cloud resilience at scale. His focus: making sure that as AI accelerates the pace of development, the systems underneath don't quietly come apart.

The Manual Era Is Over (Mostly)

Not long ago, incident management at Intuit looked like what you'd expect at most large engineering organizations. Engineers correlated alerts from multiple monitoring systems by hand, confirmed whether something warranted an incident, then manually set up a collaboration bridge, opened a Slack channel, and notified the right people. It worked, but it was slow and operationally expensive.

That shifted when Intuit standardized on service health metrics — defined thresholds that, when breached, automatically trigger incident creation, provision a Slack channel, and notify stakeholders. Engineers could start troubleshooting almost immediately instead of spending the first 20 minutes on logistics.

But root cause analysis stayed manual. Engineers were still sifting through logs, metrics, and traces by hand — a tedious process during the worst possible moments.

That's the gap AI is now filling.

"AI is now analyzing metrics, logs, traces, historical incidents," Pratinav said. "Instead of on-call engineers manually searching through thousands of signals, AI is able to summarize incident context and help responders understand what changed and where to begin investigation much faster."

The specific advantage is pattern recognition across disconnected data sources. A recent deployment, an infrastructure configuration change, and an uptick in latency across dependent services may each seem unrelated in isolation. AI can surface that correlation quickly. Humans under pressure often miss it.

A New Category of Production Failure

A couple of years ago, most production incidents traced back to bugs in manually written code. Engineers had time between deployments to understand system behavior before the next wave of changes arrived.

AI has compressed that window significantly.

"The bottleneck has shifted from writing code to operating code safely in production," Pratinav said.

The failure modes have shifted too. Configuration drift is more common — AI-generated infrastructure or deployment configurations that look syntactically correct but are operationally incomplete. Unexpected interactions between independently generated components. Deployment velocity that makes it harder to connect an incident to a recent change.

Pratinav described a specific pattern worth watching. An AI assistant generates a microservice, including the application code and deployment configuration. The code passes unit tests. It passes integration tests. But the generated deployment omits resource limits, uses incorrect autoscaling thresholds, skips resiliency configurations like retries or circuit breakers, or introduces inefficient dependency calls that only become visible under real production load.

"This isn't exactly a coding bug," he said. "It's more of an operational design gap that only surfaces at scale."

Building Systems That Can Fail Gracefully

At Intuit's scale, resilience can't rely on a few engineers who remember the recovery playbook. The platform has to encode recovery patterns and automate repeatable actions.

Pratinav's team built a cloud-native disaster recovery orchestration platform that moves away from manual runbooks — documents that different teams maintained differently, that only certain engineers could execute, and that were error-prone under pressure — toward a declarative recovery model.

"Engineering teams can share their intent for disaster recovery, and the platform executes it," he said.

They validate it through regular game days, with a deliberate goal: make recovery boring. No heroics. No guesswork. Just predictable execution.

One game day exposed a gap they hadn't anticipated. The automated recovery workflow completed successfully from an infrastructure perspective. Compute recovered. The database promoted. Traffic shifted. Monitoring showed core platform components were healthy.

But one downstream dependency hadn't fully initialized before traffic arrived. Every individual component appeared healthy in isolation. The application wasn't ready to serve customer requests.

"Recovering infrastructure is only one part of disaster recovery," Pratinav said. "True recovery means the entire application ecosystem is ready — including dependent services, caches, databases."

The fix: dependency-aware recovery workflows with readiness gates and health validation that verify the full stack is actually ready before traffic moves.

Where AI Is Saving Real Time in Incident Response

Detection has improved most visibly. Standardized health metrics combined with AI correlation have reduced alert noise significantly. Instead of flooding engineers with hundreds of alerts, the system makes meaningful determinations about which alerts are related and which are redundant.

Triage has followed. AI has access to metrics, logs, and traces simultaneously, and can summarize incident context quickly — cutting the time engineers spend collecting information so they can spend more time solving problems.

"That's the big paradigm shift," Pratinav said. "Focus on solving the problem rather than collecting information."

Root cause analysis has accelerated too. Engineers receive prioritized areas to investigate rather than starting from scratch across disconnected dashboards.

Remediation is the next frontier. Most mature organizations already have automation for failover and disaster recovery. Pratinav sees AI-assisted remediation — what he calls an SRE co-pilot — as the natural evolution, helping on-call engineers make difficult decisions and take action faster.

The Alert Storm That Changed How They Think About Scale

One incident shaped how Pratinav's team approaches large-scale operations. A failure in a critical service triggered a cascade of alerts across dozens of dependent services. Every downstream service reported elevated latency and failed requests. Within minutes, the monitoring system had generated a large number of alerts.

Under traditional incident management, those alerts created multiple separate incidents. Different service teams began triaging independently. Multiple bridge calls started. Engineers spent time investigating symptoms instead of the source of the failure.

"We weren't just duplicating alerts," Pratinav said. "We were duplicating engineering effort."

The solution was an intelligent correlation engine that analyzed service dependency telemetry to determine whether multiple alerts traced back to the same underlying event. Instead of separate incidents, the platform collapsed them into a single incident representing the primary failure — with all affected services linked, giving every responder one place to collaborate and one root cause to chase.

The result: fewer parallel investigations, clearer blast radius visibility, and engineers focused on resolution rather than symptoms.

One Practical Step for Engineering Leaders

With AI accelerating deployment velocity, operational risk is growing — not because AI writes worse code, but because more changes are reaching production. Pratinav's team introduced deployment risk assessment to address this directly: instead of treating every deployment equally, each change is scored based on size and scope, service criticality, infrastructure modifications, and historical deployment outcomes.

High-risk changes get flagged early. Operational concerns move earlier in the development cycle — before production becomes the place where problems surface.

But if there's one thing Pratinav would recommend to engineering leaders planning their next quarter, it isn't another AI tool.

"Standardize your operational data before investing in more AI," he said. "AI doesn't create observability — it amplifies it. If your operational data is fragmented, AI scales that fragmentation. If it's standardized, AI becomes significantly more effective."

If every team uses different logging formats, different metrics, and inconsistent alerting, AI has very little useful context to work with. Build the foundation first, then let the AI do its job.

How a Single NAT Gateway Can Silently Kill Your AWS High Availability

Sat, 04 Jul 2026 08:44:42 +0000

A real-world lesson from a production-like AWS lab challenge

The Scenario That Should Scare You

Imagine this: your AWS environment has two Availability Zones, public and private subnets, an Application Load Balancer, Auto Scaling. Your architecture diagram looks solid. Then one Availability Zone goes down, your ALB fails over instantly, your EC2 instances in AZ-B are running fine. But your application is still broken.

Because every private subnet instance, including those in AZ-B, is routing outbound traffic through one NAT Gateway sitting in AZ-A. Which is now unreachable.

You didn't have a highly available architecture. You had the illusion of one.

Understanding the Problem: NAT Gateways Are Zonal

A NAT Gateway is not a regional resource. It lives in a specific Availability Zone.

When you create a NAT Gateway, you place it in a specific subnet, which belongs to a specific AZ. If that AZ goes down, your NAT Gateway goes down with it.

Many teams create a single NAT Gateway to save costs, then route all private subnet traffic across all AZs through that one gateway:

Private Subnet AZ-A → 0.0.0.0/0 → nat-09xxxxx (AZ-A) ✅
Private Subnet AZ-B → 0.0.0.0/0 → nat-09xxxxx (AZ-A) ❌

The private subnet in AZ-B is routing through a NAT Gateway in AZ-A. This is a cross-AZ dependency, and a silent Single Point of Failure.

What I Found in the Lab

The lab presented a VPC with this structure:

Resource	CIDR / Details
VPC	10.0.0.0/16
Public Subnet AZ-A	10.0.128.0/20
Public Subnet AZ-B	10.0.144.0/20
Private Subnet 1A (AZ-A)	10.0.0.0/19
Private Subnet 1B (AZ-A)	10.0.192.0/21
Private Subnet 2A (AZ-B)	10.0.32.0/19
Private Subnet 2B (AZ-B)	10.0.200.0/21

Two NAT Gateways existed: one in AZ-A, one in AZ-B. At first glance, this looked correct.

But when I inspected the Route Tables, the problem was immediately visible. All four private subnet Route Tables had the same entry:

Destination: 0.0.0.0/0 → Target: nat-09xxxxxxxx (AZ-A)

The NAT Gateway in AZ-B existed, but nobody was using it. It was provisioned but completely disconnected from the routing logic. The two private subnets in AZ-B were silently depending on the NAT Gateway in AZ-A for all outbound internet traffic.

Why This Happens

There are two common causes:

1. Cost-cutting gone wrong
Teams create one NAT Gateway to reduce costs, then forget that high availability requires one per AZ. A NAT Gateway costs approximately $0.045/hour plus data transfer charges. Running two instead of one adds roughly $32/month, a small price compared to the cost of an outage.

2. Infrastructure drift
The architecture was correct at some point, then someone modified the Route Tables manually, or via a flawed IaC change, and the second NAT Gateway became orphaned without anyone noticing. No alerts, no errors, no warnings. Everything looks fine until AZ-A goes down.

This is what makes this particular SPOF so dangerous: it is completely invisible during normal operations.

The Fix: One NAT Gateway Per AZ, One Route Table Per Private Subnet

The solution is straightforward: each private subnet must route its outbound internet traffic through the NAT Gateway in its own Availability Zone.

Correct routing after the fix:

Private Subnet 1A (AZ-A) → 0.0.0.0/0 → nat-AZ-A ✅
Private Subnet 1B (AZ-A) → 0.0.0.0/0 → nat-AZ-A ✅
Private Subnet 2A (AZ-B) → 0.0.0.0/0 → nat-AZ-B ✅
Private Subnet 2B (AZ-B) → 0.0.0.0/0 → nat-AZ-B ✅

Step 1 — Identify which NAT Gateway belongs to which AZ

Go to VPC → NAT Gateways, click each NAT Gateway and check the Subnet field, this tells you which AZ it belongs to.

Step 2 — Fix the Route Tables for AZ-B private subnets

Go to VPC → Route Tables
Find the Route Table associated with Private Subnet 2A (AZ-B)
Click Edit routes
Change 0.0.0.0/0 from nat-AZ-A → nat-AZ-B
Save changes
Repeat for Private Subnet 2B (AZ-B)

Step 3 — Verify

All four private subnet Route Tables should now point exclusively to the NAT Gateway in their own AZ. If AZ-A goes down, AZ-B is completely self-sufficient.

Getting It Right From the Start: Terraform

If you're provisioning your VPC with Infrastructure as Code, which you should be, here's how to enforce this pattern correctly with Terraform from day one.

ˋhcl

NAT Gateway in AZ-A

resource "aws_eip" "nat_a" {
domain = "vpc"
}

resource "aws_nat_gateway" "nat_a" {
allocation_id = aws_eip.nat_a.id
subnet_id = aws_subnet.public_a.id

tags = {

Name = "nat-gateway-az-a"

}
}

NAT Gateway in AZ-B

resource "aws_eip" "nat_b" {
domain = "vpc"
}

resource "aws_nat_gateway" "nat_b" {
allocation_id = aws_eip.nat_b.id
subnet_id = aws_subnet.public_b.id

tags = {

Name = "nat-gateway-az-b"

}
}

Route Table — AZ-A private subnets

resource "aws_route_table" "private_a" {
vpc_id = aws_vpc.main.id

route {

cidr_block     = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.nat_a.id

}

tags = { Name = "private-rt-az-a" }
}

Route Table — AZ-B private subnets

resource "aws_route_table" "private_b" {
vpc_id = aws_vpc.main.id

route {

cidr_block     = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.nat_b.id

}

tags = { Name = "private-rt-az-b" }
}

Associations — AZ-A

resource "aws_route_table_association" "private_1a" {
subnet_id = aws_subnet.private_1a.id
route_table_id = aws_route_table.private_a.id
}

resource "aws_route_table_association" "private_1b" {
subnet_id = aws_subnet.private_1b.id
route_table_id = aws_route_table.private_a.id
}

Associations — AZ-B

resource "aws_route_table_association" "private_2a" {
subnet_id = aws_subnet.private_2a.id
route_table_id = aws_route_table.private_b.id
}

resource "aws_route_table_association" "private_2b" {
subnet_id = aws_subnet.private_2b.id
route_table_id = aws_route_table.private_b.id
}
ˋ

The beauty of this approach: the correct pattern is enforced by design. Each AZ has its own NAT Gateway, its own Route Table, and explicit associations. Infrastructure drift becomes impossible, any change goes through code review.

The Broader Lesson: Designing for Failure

AWS high availability is built on one fundamental principle:

Assume everything will fail. Design so that the failure of any single component does not bring down the entire system.

A NAT Gateway is a component. An Availability Zone is a failure domain. When you route cross-AZ traffic through a single NAT Gateway, you create an invisible dependency that violates this principle, and the worst part is that everything looks fine until the moment it isn't.

The AWS Well-Architected Framework's Reliability Pillar specifically calls for eliminating Single Points of Failure. A shared NAT Gateway is a textbook SPOF, easy to miss precisely because the architecture looks correct at first glance.

Key Takeaways

A NAT Gateway is zonal, it belongs to one specific Availability Zone
Routing all private subnet traffic through a single NAT Gateway creates a hidden Single Point of Failure
The fix: one NAT Gateway per AZ, one Route Table per AZ
Use Terraform to enforce this pattern by design and prevent infrastructure drift
The cost of two NAT Gateways (~$32/month extra) is nothing compared to the cost of an outage

This article is part of my AWS Solutions Architect Associate (SAA-C03) preparation series. I document real hands-on lab experiences, architecture challenges, and lessons learned along the way.

Follow along for more practical AWS architecture and Infrastructure as Code content.

Building an uptime monitor in Rust: one binary, two databases, 130K checks/sec per core

Tue, 30 Jun 2026 17:20:57 +0000

I spent the last few months building Uptimepage, an open-source uptime monitor and status page written in Rust. This post is the build story: the decisions that shaped it, the parts I rewrote, and the numbers that came out the other side.

The whole thing ships as one self-contained binary of about 23 MB, plus Postgres and ClickHouse. You can docker compose up and self-host it, or use the hosted tier. Source is AGPL-3.0 on GitHub.

Why one binary

A status page is a small surface with a lot of moving parts behind it: a scheduler, probe workers, an HTTP client, a time-series writer, an incident detector, an alerting fan-out, a web UI, a JSON API. The usual answer is a handful of services and a message bus between them.

I went the other way. One process, one binary, the same image whether it runs the control plane or a remote probe. No queue to operate, no version skew between services, no "which container is wedged" at 3am. The cost is that you have to be careful about what shares a thread and what can stall what. Most of the engineering below is about keeping those boundaries clean inside a single process.

Stack: Rust 1.95 (edition 2024), Tokio, Axum 0.8, Askama for compile-time HTML templates, HTMX 2 for partial swaps so there is no SPA framework to ship. The API stays the single source of truth because every UI mutation hits the same /api/v1/* endpoint a script would.

Two databases, on purpose

Monitors are low-cardinality relational data that gets mutated by API calls: targets, regions, channels, incidents, plans. That is Postgres. Check results are append-only, high-cardinality, and almost always queried by time range. That is ClickHouse.

Trying to force one of those into the other is where uptime monitors usually fall over. Putting billions of check results in Postgres turns every dashboard query into a sequential scan. Putting your relational config in ClickHouse means fighting its update model forever. So: Postgres 18 for the world, ClickHouse 26.3 for the firehose. Both run their migrations at process startup, so there is no separate migrator to forget.

The HTTP client I did not want to write

The first version used a popular high-level HTTP client. It worked, but a monitor is a weird HTTP workload: you connect once per target per interval, you never reuse the connection, and you care about the timing of each phase more than the body.

So I dropped down to hyper and hyper-util with rustls and built a connector that times DNS resolution, TCP connect, and the TLS handshake as separate numbers, then runs the request over hyper::client::conn and aborts the connection task the moment the body is read. Each result carries dns_ms, connect_ms, tls_ms, and ttfb_ms as distinct columns, which is what makes "it got slow but it is the DNS, not your server" a thing the dashboard can actually say.

The rewrite paid for itself. On a single core the client sustains around 130K checks/sec at saturation, roughly 7.7 microseconds per check. That was a 44 to 56 percent throughput gain over the old path. A chunk of it was a url::parse call hiding in the redirect policy that cost 7.5 percent on its own and just vanished. Two cores get to about 153K. Scaling goes sub-linear past four cores because of shared HTTP/2 connection state and the pool mutex, which is a fine problem to have for this workload.

These numbers come from a laptop and a loadtest binary, so I treat them as regression detection, not capacity planning. The headline in-process run pushed 252K requests/sec sustained at p99 393ms. The design goal is around 50K concurrent in-flight checks per node with under 50ms p99 of per-check overhead on top of the network.

One heap, not a timer per target

The naive scheduler spawns a timer task per monitor. That falls apart at fleet size: thousands of tasks, thousands of wakeups, memory that grows with the number of targets.

Instead there is a single driver task that owns one BinaryHeap<Reverse<Due>>, a min-heap keyed by the next due Instant for the whole fleet. Memory stays flat in fleet size. Each target gets a deterministic jitter offset hashed from its UUID so a thousand monitors on a 60s interval do not all fire on the same tick. Generation and sequence counters mean a re-scheduled target supersedes its stale heap entry instead of double-firing. The registry refresh that pulls config from Postgres runs on its own task with exponential backoff, so a Postgres hiccup never stalls dispatch: the scheduler keeps running on what it last knew.

Failure isolation inside one process

Because everything is one process, one bad target cannot be allowed to take down the rest. Three patterns do most of that work. Per-host circuit breakers trip when a host keeps failing, so it fails fast with circuit_open instead of tying up a worker on a timeout, then probes half-open after a cooldown. A per-tenant host throttle bulkhead caps how many checks can be in flight against one host at once; over the cap, a check is recorded as throttled and degraded rather than piling on, and it never pages. And singleflight on RDAP collapses domain-expiry checks for the same domain across many tenants into one upstream probe, with sticky last-good state so a flaky registrar does not flip the monitor red.

The worker pool itself is a task-per-dispatch gated by a semaphore sized to a max-concurrency setting, with an SSRF guard filtering resolved IPs before any connect.

Modeling time series in ClickHouse

The check_results table is a MergeTree ordered by (org_id, target_id, region, timestamp). The org_id leads the sort key so each tenant gets a sparse-index slice, but it is deliberately kept out of the partition key (partition is by day) so we do not end up with millions of partitions.

The columns are where the storage savings live:

Timestamps are DateTime('UTC') with CODEC(DoubleDelta, ZSTD(1)). Check intervals are near-constant, so DoubleDelta crushes the gaps to almost nothing.
Numeric phase columns (duration_ms, dns_ms, connect_ms, tls_ms, ttfb_ms, response code and size) use CODEC(T64, ZSTD(1)).
region and agent_id are LowCardinality(String), status is an Enum8.

Retention is per row. There is a ttl_days column with TTL timestamp + toIntervalDay(ttl_days), stamped from the org's plan at write time. A plan that buys longer history needs zero schema change: new rows just carry a bigger number.

On top of the raw table sit two AggregatingMergeTree materialized views: a per-minute rollup kept 30 days and an hourly rollup kept 13 months, both holding quantilesState for p50/p95/p99 and per-status counts. Reads route by range: anything inside 30 days hits the minute rollup, older ranges hit the hour rollup, and raw reads are capped at a 90-day span. A dashboard asking for "last 24h p99 latency" never touches a raw row.

Regional probes without a second brain

You can run probes in multiple regions, but I did not want each region to be its own little system. So an agent is the same binary in agent mode, running as a stateless probe with no database, no web, no alerting. Adding a region adds execution capacity, never a second control plane.

An agent pulls its region's config from the control plane with ETag/304 handling, serves its last-known config if the control plane blips, and pauses if its token is revoked. It ships results back in batches that reuse one UUID across retries so a lost ack cannot double-count. Region and agent identity are derived server-side from the bearer token, never sent in the payload, so a probe cannot claim to be somewhere it is not. A separate long-poll loop handles interactive "check now" so a button press in the UI runs on a real remote probe within milliseconds.

Region is the partition dimension end to end, so every read can slice by region: per-region latency series, per-region incident scope, "down in Singapore, up in Helsinki."

Incidents as a follower, not a gatekeeper

The incident detector is a background task that follows the check_results stream and writes into the Postgres incidents table. The rule I held to: it never touches the hot write path, never gates check execution, and never produces alerts directly.

The detection itself is boring on purpose: two or more consecutive unhealthy results with no open incident opens one, two or more consecutive healthy results closes it. A small flap threshold absorbs single-result blips. The cross-tenant walk is keyset-paginated so memory stays bounded as tenants grow, and a unique index on open incidents resolves the race when two ticks try to open the same one: only the winner pages.

Opening or resolving fires a non-blocking signal to the escalation engine, which does repeat-until-acknowledged paging on a per-monitor cadence across about fourteen transports (Slack, generic signed webhooks, Telegram, PagerDuty, ntfy, Pushover, Discord, email, and more), with sharded per-incident locks so a reconcile and an inbound signal can never double-page the same episode. Channel secrets are sealed at rest with AES-GCM and never echoed back.

Automation as a first-class surface

Because the API is the single source of truth, the rest came almost for free:

A self-describing OpenAPI 3.1 spec with Swagger UI.
An official Terraform provider so you can manage monitors and notification channels as code.
An MCP server so an LLM client can query your monitors and incidents over OAuth 2.1, scope-gated and audited, with per-action confirmation on the few write tools.

Where it is

Uptimepage is live, free to start with no card, and AGPL-3.0 open source. The core is not paywalled: checks, status pages, subscribers, the API, and every alert channel are in the free tier. It monitors itself and serves its own status badges from the running binary, which is the most honest dogfood I could think of.

Site and hosted tier: https://uptimepage.dev
Source: https://github.com/uptimepage/uptimepage
More build notes: https://uptimepage.dev/blog

Happy to answer anything about the Rust internals in the comments.

Controlled Burns for Organizations

Tue, 09 Jun 2026 01:52:37 +0000

Controlled Burns for Organizations

What the Forest Service knows about change that consultants don't.

In late winter, somewhere on the Coronado National Forest, a crew with drip torches walks a ridgeline. The relative humidity is above forty percent. The wind is backing down the slope. A firebreak was cut yesterday. Pump trucks are staged at three access points. At 10:14 a.m., the burn boss gives the word, and the first mixture of gasoline and diesel drips onto pine needles.

Within an hour, the crew is watching a fire they started.

The U.S. Forest Service runs about 4,500 of these a year. Around seven escape. That is less than one percent.

Change management has been talking about "burning platforms" for three decades. The phrase comes from the 1988 Piper Alpha disaster in the North Sea — workers jumped into freezing water because staying on the rig meant certain death. Daryl Conner formalized the metaphor in his 1992 book Managing at the Speed of Change, and consulting firms imported it to justify manufactured urgency in organizational transformations. The trouble is that a burning platform is a wildfire. A disruption that happens to you. The discipline you actually need is the one those crews practice on the ridgeline: a disruption you choose.

The suppression paradox

In 1905, Theodore Roosevelt signed the Transfer Act that created the U.S. Forest Service, and the new agency inherited a simple mandate: put out every fire as fast as possible. In 1944, the Smokey Bear campaign launched, building on the 1942 Cooperative Forest Fire Prevention effort; by the 1950s "Only You Can Prevent Forest Fires" had become one of the most effective public-awareness campaigns in American history. The forests seemed safer. The fire numbers fell. The agency had done its job.

Before Roosevelt and before Smokey Bear, people on this continent had been setting fire to these forests for roughly ten thousand years. The Karuk in the mid-Klamath set fire to oak savannas and mixed-conifer stands on cycles that ran roughly three to ten years depending on the resource and the location. These were not pyromaniacs. They were maintenance crews operating at civilizational scale. Low-intensity surface fire cleared undergrowth, encouraged forage species, killed pathogens, and — most importantly — kept the fuel load low enough that when a lightning strike eventually started a fire, it stayed on the ground instead of climbing into the canopy.

Suppression doctrine stopped that maintenance. The forests didn't become safer. They became dense and choked, stacked with decades of unburned fuel. The number of small fires fell. The size of the fires that did occur grew. The term of art for this is the "fire deficit" — the gap between the acreage that should have burned under a reference regime and what actually burned under suppression.

The deficit is paid back with interest. The 2020 California fire season alone burned more than four million acres. The 2022 New Mexico megafires, the 2023 Maui fires, the 2025 Los Angeles fires — these are not separate events. They are installments on a bill that has been accruing since 1905.

Suppression doesn't eliminate fire. It concentrates and defers it.

What prescribed fire actually is

The "less than one percent" figure is what gets people's attention. But the real insight is what prescribed fire does over time. A 2024 study in Fire Ecology (Brodie et al., Fire Ecology 20:17) found that combinations of mechanical thinning and prescribed fire were still measurably reducing wildfire severity roughly twenty years after the initial thinning treatment and ten years after the most recent prescribed burn. This is not a one-time intervention. It is a schedule.

The 2022 Black Fire in New Mexico became a megafire by area — about 131,000 hectares — but only about four percent of that burned at high severity. Researchers attributed the low severity, in large part, to prior fuels-reduction treatments including prescribed fire and previous wildfire footprints. The fire happened. The catastrophic damage didn't.

Crews don't pick burn days at random. They work within a narrow envelope defined by temperature, humidity, wind direction, slope, and time of year. The Southeast historically had the most generous burn windows in North America; climate change is now compressing that window as rising temperatures and declining moisture reduce the number of safe burn days. Meanwhile, California has a declared goal of treating about 400,000 acres (~162,000 hectares) of land per year with beneficial fire by 2025 — and has fallen drastically short. The constraint is no longer "can we burn." It is "can we find enough trained burners, enough weather windows, and enough legal cover to burn at the rate the forest needs."

Which brings us to the number most worth dwelling on.

The spotfire asymmetry

A spotfire is what happens when an ember — a single burning fragment lofted by convection or wind — lands outside the burn perimeter and starts a small new fire. Oklahoma State Extension's NREM-2903 fact sheet, drawing on field data across hundreds of burns, reports that with relative humidity between twenty and eighty percent, the probability of at least one spotfire occurring is 21.2 percent. Roughly one burn in five.

Now read that inside the other number. Spotfires happen on one burn in five. Escapes happen on fewer than one in a hundred. A spotfire, in other words, almost never becomes an escape. The crews expect spotfires. They staff for spotfires. A spare engine idles at the firebreak precisely because the burn boss assumes that sooner or later, today, an ember will skip the line.

The fact sheet also reports something more interesting than the aggregate. Below forty percent relative humidity, spotfire probability jumps to 41.3 percent. Above forty percent, it drops to 3.8 percent. A ten-fold swing across a single threshold. Fire behavior is non-linear, and the crews know exactly where the cliff is. They burn on the safe side of it.

This is the operational insight that most change programs never internalize. Most organizations treat small negative consequences — a team pushback, a confused customer, a missed rollout deadline — as signals to abort. The prescribed-fire discipline treats them as signals that the system is working as designed. The spotfire is not a failure of the burn. It is a prediction the burn made, caught, and contained.

A system that cannot absorb its own routine spotfires is a system forced to choose between stagnation and catastrophe. A system that expects spotfires, staffs for spotfires, and contains spotfires gets something different: routine.

The mapping

Many things in the fire model have a clean analogue in organizational life, and a few don't. The rough correspondence looks like this.

Fuel load — the decades of unburned biomass — is the accumulated dysfunction an organization carries. Dead projects nobody will kill. Processes whose original reason nobody remembers. Middle-manager resentments from reorgs that never got closure. Every suppressed decision is a handful of pine needles on the forest floor.

Ignition is the deliberate small-scale change. A pilot. A sandbox team. A time-boxed reorg of one department. A production chaos test. Netflix's Chaos Monkey is the purest software example: it deliberately kills production services on a schedule precisely so the organization's response muscle stays conditioned. Google's SRE error budget is the same discipline — a non-zero tolerance for failure that lets engineers push changes without triggering catastrophic risk-aversion.

Pre-positioned crews, firebreaks, and the humidity window are everything an organization does before the change starts. Rollback plans. Communication scripts drafted a week in advance. An executive sponsor who will provide air cover when the first confused Slack message lands. The budget contingency that means a project can absorb a two-week slip without existential risk. Toyota's andon cord — any worker's right to stop the production line — is pre-positioned spotfire containment, elevated to cultural rule.

The spotfire is the unexpected consequence. An employee raises an objection nobody anticipated. A customer segment reacts badly to a feature. The change encounters a dependency nobody mapped. In a suppression culture, this gets treated as evidence the change should be aborted. In a prescribed-fire culture, this gets written down in the after-action review and used to calibrate the next burn.

The wildfire is what happens when all of the above has been deferred too long: forced bankruptcy restructuring, regulator-mandated breakups, mass exodus of senior talent. The fire happens to the organization rather than being conducted by it. Boeing's 737 MAX cascade, following two decades of cost-driven deferrals of engineering investments and a corporate posture that treated pilot retraining as a commercial constraint rather than a safety question, is a modern case study in organizational fuel accumulation.

And the burning platform — the thirty-year-old metaphor that change consultants sold so hard — is the moment the wildfire arrives. By the time you're standing on it, your options have collapsed to "jump or die."

Why we don't do this

If prescribed fire works, and the evidence for it is decades old and accumulating, why don't we burn more? A 2024 paper in Frontiers in Ecology and the Environment (Williams et al.) catalogued the obstacles: liability law — a burn boss is personally exposed if an escape causes property damage — air-quality regulations that flag prescribed-burn smoke even while preventing worse smoke from future wildfires, narrow weather windows, public opposition, and a severe shortage of trained burners. Every obstacle is rational on its own. The sum of them is a system that can't do the thing it knows it should do.

The organizational obstacles map cleanly. Liability and regulatory risk become legal exposure, HR exposure, and the career risk to the executive who sponsors a failed pilot. Smoke — the visible byproduct that neighbors complain about — becomes the external communication failure, the press article about layoffs, the customer who churns during the migration. The trained-burner shortage becomes the most underrated constraint: most organizations have no internal change-craft. Every change initiative gets run by people who are running their first change initiative. The first prescribed burn a new crew conducts is dangerous. The hundredth is routine. Organizations that treat change as a series of one-off projects never get to the hundredth.

Where the analogy breaks

Forests are not organizations, and the metaphor breaks in at least three places worth naming.

First, and most severely: forests are slow systems. Fuel accumulates over decades and the window between burns is measured in years. Organizations are fast. A software team ships code every day; a sales team resets every quarter; a company reorganizes every few years. The cadence of "burns" in a healthy organization runs at minimum weekly and probably daily. If you read "prescribed burn" and think "big annual reorg," you have inverted the prescription. The burn unit, for most of organizational life, is small and frequent.

Second: forests have reference conditions. Ecologists can reconstruct, from tree-ring scars, what the fire regime looked like in the ponderosa pine belt in 1700, and they use those records to calibrate targets. Organizations don't have reference conditions. There is no objectively correct rate of organizational change against which to measure fuel-load. This is a real limitation; it means the prescribed-fire discipline in organizations has to be grounded in principles — expect spotfires, pre-position containment, practice the craft — rather than calibrated to targets.

Third: the capability gap is larger than it looks. The U.S. Forest Service has been running controlled burns, tracking their outcomes, and training burn crews for generations. Their success rate sits on top of institutional knowledge a company cannot replicate in a quarter. Treating this as a how-to is already a mistake; it is more like a direction of travel.

There is a small mercy in how the analogy breaks, which is that the Forest Service already knows. A 2020 paper from researchers at the Rocky Mountain Research Station, Organizational Learning from Prescribed Fire Escapes (Black, Hayes & Strickland, Current Forestry Reports 6:41–59), applies high-reliability-organization theory — a framework originally developed to study nuclear plants, aircraft carriers, and hospital operating rooms — to understand why those seven-or-so annual escapes happen. The metaphor flows both directions. The fire-ecology community learned change-management theory from us. The least we can do is learn fire-ecology discipline back.

What the practice looks like

An organization that takes this seriously does a few concrete things.

It defines burn windows deliberately. Not "whenever we feel like it" and not "whenever a crisis demands it." Post-product-launch. Post-quarter-close. The six weeks after an acquisition closes, before the newly combined teams have set in their new patterns. These are the organizational equivalents of the humidity window. Most companies have natural cadences that would make excellent burn windows if anyone were watching for them.

It pre-positions containment before ignition, not after. Rollback plans, not rollback hope. An executive sponsor briefed on every plausible objection, not an executive sponsor who learns about the change from a complaining Slack thread. A budget line item for the spotfires that will occur, not a scramble when they do.

It reframes unexpected consequences. A pilot team encounters a dependency nobody knew existed. In suppression culture, this is evidence the pilot should be cancelled. In prescribed-fire culture, it is a discovery — the thing we just learned from a spotfire we contained. The after-action review is the central ritual. What did we learn. Where was the windbreak weak. Who needs a drip torch next time.

And it builds a burn association. This is what the Forest Service calls the informal networks of landowners, agencies, and trained burners who pool equipment and expertise across burn seasons. The organizational analogue is a community of practice around change-craft — alumni of past pilots who can be borrowed for new ones, a playbook that accumulates rather than a deck that restarts.

Maintenance, not transformation

Return to the ridgeline. The burn crew is finishing up. Someone is tamping out a line of smoldering duff at the edge of the firebreak. Someone else is checking that the spotfire they caught an hour ago has been fully worked. The captain is filling out a form that will become, eventually, part of an annual report. Tomorrow they will do this again on a different ridge.

For the Karuk, and for the dozens of other peoples whose fire stewardship shaped the ecosystems we now think of as natural, this is not a change initiative. It is not a transformation. It is just the work — the seasonal, unglamorous, necessary work of keeping a forest a forest. The reason Euro-American management doctrine kept inventing more dramatic metaphors for change — burning platforms, disruption, revolution — is that it had forgotten there was a quieter practice underneath.

Most of what change management wants is already available. It has just been misfiled under transformation when it belongs under maintenance. The companies that figure this out will not win because they ran a heroic reorganization. They will win because they ran a few thousand small burns that nobody wrote a book about.

Seven of those burns escaped. The other four thousand four hundred and ninety-three did exactly what they were supposed to do.

Sources: Brodie et al., "Forest thinning and prescribed burning treatments reduce wildfire severity..." Fire Ecology 20:17, 2024 (doi:10.1186/s42408-023-00239-7); Williams et al., "Overcoming obstacles to prescribed fire in the North American Mediterranean climate zone," Frontiers in Ecology and the Environment, 2024; Oklahoma State Extension NREM-2903; Black, Hayes & Strickland, "Organizational Learning from Prescribed Fire Escapes," Current Forestry Reports 6:41–59, 2020 (doi:10.1007/s40725-020-00111-w); GAO blog "Fighting Fire with Fire"; Aiken & Keller, "The irrational side of change management," McKinsey Quarterly, April 2009; Conner, Managing at the Speed of Change, 1992; the 1905 Transfer Act; the 1944 Smokey Bear campaign; Karuk fire-stewardship cycles drawn from ethnoecological literature including Skinner et al. and Karuk Tribe DNR publications.

Spotfires aren't bugs — they're predictions the burn made. Concretely: Chain of Consciousness turns every agent action into a signed record — a content-hashed claim with a timestamp, the model version, the tool calls made, the outcome, and a rating. Those records pin to a public provenance chain. When a spotfire lands — a customer hits unexpected behavior, a downstream dependency breaks, a rating dips — you open the trace and see which deployment introduced it, what prior claims the agent made, and which reviews dissented. The after-action review is already written, because every burn wrote one as it happened. See a live chain · Verify an agent's rating · pip install chain-of-consciousness

Originally published at vibeagentmaking.com.

Terraform Drift Detection and Recovery on Google Cloud: Plan, Import, State, and GitHub Actions

Thu, 28 May 2026 07:18:44 +0000

Hi! so this is my fourth terraform artefact. I'd like to say that creating infrastructure with Terraform is one thing, but using Terraform when reality changes outside the code is another thing.

For the fourth artifact in my Terraform x Google Cloud portfolio, I wanted to explore a more operational topic:

Terraform drift detection, import, and state recovery.

In this project, i will not be building a large infrastructure platform. This project is about understanding what happens when:

someone changes infrastructure manually
Terraform state no longer matches real infrastructure
an existing resource needs to be imported
a resource is accidentally removed from state
drift needs to be detected automatically

The artifact repository:

terraform-gcp-drift-import-recovery

The main goal:

Understand Terraform failure modes, not only Terraform provisioning.

Why I Built This Artifact

My previous Terraform artifacts focused on building and delivering infrastructure.

The portfolio journey looked like this:

Artifact 1 — Terraform GCP Foundation
Artifact 2 — Production-Lite GCP Web Platform
Artifact 3 — Terraform CI/CD with GitHub Actions and WIF
Artifact 4 — Drift Detection, Import, and State Recovery

The first three artifacts answered:

Can I create infrastructure?
Can I structure it properly?
Can I expose it securely?
Can I automate Terraform plan and apply?

This fourth artifact answers a different question:

What happens when Terraform is no longer the only actor changing infrastructure?

That question matters because real cloud environments are rarely perfect.

Someone may change a firewall rule manually.

Someone may create a resource directly from the console.

Someone may remove a resource from Terraform state by mistake.

Someone may perform an emergency change and forget to update the code.

This artifact is about detecting, understanding, and recovering from those situations.

What This Project Demonstrates

This project demonstrates:

Terraform remote state
baseline GCP infrastructure creation
manual drift simulation
terraform plan drift detection
terraform plan -detailed-exitcode
terraform plan -refresh-only
terraform state list
terraform state show
terraform state pull
terraform state rm
terraform import
importing manually created GCP resources
recovering after state removal
scheduled drift detection with GitHub Actions
GitHub issue creation when drift is detected
Workload Identity Federation
no service account JSON key

The lab intentionally uses simple Google Cloud resources:

VPC
subnet
firewall rule
service account

That is deliberate.

The objective is not infrastructure complexity.

The objective is Terraform behavior.

Why I Created a Separate Repository

I created this as a separate repository instead of adding it into my production-lite web platform repo.

Reason:

This lab intentionally creates abnormal infrastructure situations.

It includes:

manual changes outside Terraform
state removal
resource import
drift simulation
drift detection
recovery scenarios

Those activities should not be mixed into a main application platform repository.

This project is a controlled failure-mode lab.

Repository Structure

The repository is structured like this:

terraform-gcp-drift-import-recovery/
├── README.md
├── .gitignore
├── versions.tf
├── providers.tf
├── backend.tf.example
├── main.tf
├── variables.tf
├── outputs.tf
├── terraform.tfvars.example
├── locals.tf
├── environments/
│   └── dev.tfvars
├── imports/
│   ├── imported-network.tf.example
│   └── imported-firewall.tf.example
├── docs/
│   ├── architecture.md
│   ├── drift-detection-lab.md
│   ├── import-lab.md
│   ├── state-recovery-lab.md
│   ├── scheduled-drift-detection.md
│   ├── operations-runbook.md
│   ├── verification.md
│   ├── design-decisions.md
│   └── release-process.md
├── scripts/
│   ├── simulate-drift.sh
│   ├── create-manual-import-resources.sh
│   ├── cleanup-manual-import-resources.sh
│   └── bootstrap-wif-github.sh
└── .github/
    ├── workflows/
    │   ├── terraform-plan.yml
    │   └── drift-detection.yml
    └── ISSUE_TEMPLATE/
        └── drift_report.md

The structure separates:

baseline Terraform code
manual drift scripts
import examples
state recovery documentation
scheduled drift detection workflow
GitHub issue template

Baseline Infrastructure

The baseline infrastructure is intentionally small.

It creates:

custom VPC
subnet
firewall rule
service account

The firewall rule is the main drift target.

Why?

Because firewall rules are easy to modify manually and easy for Terraform to detect.

For example, Terraform may define:

allowed port: tcp:80

Then someone manually changes it to:

allowed ports: tcp:80,tcp:8080

That difference is drift.

Remote State Design

This project uses a GCS backend for Terraform state.

I added two GitHub repository variables for the state configuration:

TF_STATE_BUCKET = terraform-gcp-production-lite-tfstate
TF_STATE_PREFIX = terraform-gcp-drift-import-recovery/dev

Instead of committing a real backend.tf, the GitHub Actions workflow writes the backend config dynamically.

The workflow step looks like this:

- name: Write Terraform backend config
  shell: bash
  env:
    TF_STATE_BUCKET: ${{ vars.TF_STATE_BUCKET }}
    TF_STATE_PREFIX: ${{ vars.TF_STATE_PREFIX }}
  run: |
    set -euo pipefail

    if [[ -z "${TF_STATE_BUCKET}" ]]; then
      echo "TF_STATE_BUCKET repository variable is required." >&2
      exit 1
    fi

    if [[ -z "${TF_STATE_PREFIX}" ]]; then
      echo "TF_STATE_PREFIX repository variable is required." >&2
      exit 1
    fi

    cat > backend.tf <<EOF
    terraform {
      backend "gcs" {
        bucket = "${TF_STATE_BUCKET}"
        prefix = "${TF_STATE_PREFIX}"
      }
    }
    EOF

This keeps the backend configuration flexible across environments.

It also avoids hardcoding my state bucket and prefix directly in the workflow logic.

Lab 1 — Simulating Drift

The first lab simulates drift manually.

After deploying the baseline infrastructure, I modify the firewall rule outside Terraform.

Example:

export FIREWALL_RULE_NAME="$(terraform output -raw firewall_rule_name)"

gcloud compute firewall-rules update "${FIREWALL_RULE_NAME}" \
  --allow=tcp:80,tcp:8080

At this point:

Terraform code says: tcp:80
Google Cloud says: tcp:80,tcp:8080

That is drift.

Terraform code, Terraform state, and real infrastructure are no longer aligned.

Lab 2 — Detecting Drift with Terraform Plan

After simulating drift, I run:

terraform plan -var-file=environments/dev.tfvars

Terraform detects that the real firewall rule no longer matches the desired configuration.

The plan proposes to restore the firewall rule back to the Terraform-defined state.

That is the simplest drift detection mechanism:

Run terraform plan.
If Terraform proposes unexpected changes, investigate.

Lab 3 — Using `-detailed-exitcode`

For automation, I used:

terraform plan \
  -input=false \
  -no-color \
  -detailed-exitcode \
  -var-file=environments/dev.tfvars \
  -out=tfplan

The important behavior:

exit code 0 = no changes
exit code 1 = error
exit code 2 = changes detected

This is useful because a CI/CD workflow can use the exit code to determine whether drift exists.

In this artifact:

0 means no drift
2 means drift detected
1 means Terraform error

Lab 4 — Refresh-Only Planning

I also documented refresh-only planning:

terraform plan \
  -var-file=environments/dev.tfvars \
  -refresh-only

This is useful when I want to inspect how real infrastructure differs from Terraform state.

Normal plan answers:

What would Terraform change to make infrastructure match the code?

Refresh-only plan answers:

What has changed in real infrastructure compared to Terraform state?

Both are useful, but they answer different operational questions.

Lab 5 — Recovering from Drift

If the manual change was not approved, the recovery is simple:

terraform apply -var-file=environments/dev.tfvars

Terraform restores the resource to the desired configuration.

In this case, it removes the manually added port 8080.

Then I can confirm that the environment is clean:

terraform plan \
  -var-file=environments/dev.tfvars \
  -detailed-exitcode

Expected result:

exit code 0

That means Terraform sees no pending changes.

Lab 6 — Importing Existing Infrastructure

The next scenario is different.

Instead of changing an existing Terraform-managed resource, I create a resource manually outside Terraform.

Example:

gcloud compute networks create manual-import-network \
  --project="${PROJECT_ID}" \
  --subnet-mode=custom

This VPC exists in Google Cloud, but Terraform does not know about it.

To bring it under Terraform management, I need two things:

Terraform resource configuration
terraform import

Example Terraform configuration:

resource "google_compute_network" "imported_network" {
  name                    = "manual-import-network"
  auto_create_subnetworks = false
  routing_mode            = "REGIONAL"
}

Then import:

terraform import \
  -var-file=environments/dev.tfvars \
  google_compute_network.imported_network \
  "projects/${PROJECT_ID}/global/networks/manual-import-network"

After import:

terraform state list
terraform state show google_compute_network.imported_network
terraform plan -var-file=environments/dev.tfvars

The objective is to reach:

No changes.

That means the Terraform configuration and imported real resource are aligned.

Lab 7 — State Inspection

Terraform state is the mapping between Terraform resource addresses and real infrastructure objects.

Useful commands:

terraform state list

terraform state show google_compute_firewall.allow_http_internal

terraform state pull > state-backups/before-recovery.json

I included state inspection because drift and import are hard to understand without understanding state.

Terraform state answers:

Which real resource is this Terraform resource address connected to?

Lab 8 — State Recovery After Accidental State Removal

This lab simulates a state mistake.

First, I back up state:

mkdir -p state-backups

terraform state pull > state-backups/before-state-rm.json

Then I remove a resource from state:

terraform state rm google_compute_firewall.allow_http_internal

Important:

This does not delete the real firewall rule.
It only removes the mapping from Terraform state.

Now Terraform no longer knows that the real firewall rule belongs to the configuration.

If I run:

terraform plan -var-file=environments/dev.tfvars

Terraform may try to create the firewall rule again.

But the resource already exists in Google Cloud.

The correct recovery is to import it back:

terraform import \
  -var-file=environments/dev.tfvars \
  google_compute_firewall.allow_http_internal \
  "projects/${PROJECT_ID}/global/firewalls/${FIREWALL_RULE_NAME}"

Then verify:

terraform state list
terraform plan -var-file=environments/dev.tfvars

The target result:

No changes.

Lab 9 — Scheduled Drift Detection with GitHub Actions

The final part of this artifact is scheduled drift detection.

The workflow runs:

workflow_dispatch
schedule: every day at 23:00 UTC

The workflow performs:

checkout repository
set up Terraform
authenticate to Google Cloud using WIF
write backend.tf from repository variables
terraform init
terraform validate
terraform plan -detailed-exitcode
upload drift plan artifact
create GitHub issue if drift is detected

The key step:

terraform plan \
  -input=false \
  -no-color \
  -detailed-exitcode \
  -var-file=environments/dev.tfvars \
  -out=tfplan > drift-plan.txt

Then the workflow evaluates the exit code:

if [[ "${EXIT_CODE}" -eq 0 ]]; then
  echo "No drift detected."
  exit 0
elif [[ "${EXIT_CODE}" -eq 2 ]]; then
  echo "Drift detected."
  exit 0
else
  echo "Terraform plan failed."
  exit 1
fi

This is intentional.

If drift is detected, the workflow does not fail as a broken pipeline.

It creates a GitHub issue.

Why?

Because drift is not always a technical failure.

Sometimes drift is an intentional emergency change that needs to be reconciled.

The workflow should create visibility, not immediately destroy or revert something.

GitHub Issue Creation When Drift Is Detected

When Terraform returns exit code 2, the workflow creates a GitHub issue:

gh issue create \
  --title "${ISSUE_TITLE}" \
  --body-file drift-issue-body.md \
  --label "drift,terraform,incident"

The issue tells the operator to:

download the drift plan artifact
review whether the drift was intentional or accidental
revert accidental drift with Terraform apply
update Terraform code if the change was intentional
close the issue only after plan returns exit code 0

This turns drift detection into an operational workflow.

Not just a command.

Authentication with Workload Identity Federation

Like my previous Terraform CI/CD artifact, this project does not use a service account JSON key.

GitHub Actions authenticates to Google Cloud using Workload Identity Federation.

The flow is:

GitHub Actions OIDC token
  -> Google Workload Identity Provider
  -> Terraform drift CI/CD service account
  -> Google Cloud APIs

This keeps the workflow keyless.

No long-lived Google Cloud service account key is stored in GitHub.

Why I Do Not Auto-Apply Drift Fixes

It may be tempting to do this:

detect drift
automatically run terraform apply

I intentionally did not do that.

Reason:

Not all drift is accidental.

Some drift may come from:

emergency production fix
approved console change
another migration process
temporary incident workaround

The right first response is:

detect
notify
review
decide
then reconcile

Automatic remediation can be dangerous if the workflow does not understand why the drift happened.

So this artifact creates a GitHub issue instead.

What This Artifact Taught Me

The biggest lesson:

Terraform is not only a provisioning tool.
Terraform is also a control system.

What I Would Do Differently in Production

For a production environment, I would improve this further with:

least-privilege custom IAM role
Slack or email notification
deduplication of drift issues
severity classification
separate environments
policy-as-code checks
cost impact review
state backup automation
manual approval before remediation

But for this artifact, the core objective is complete:

simulate drift
detect drift
understand state
recover state
import resources
automate drift visibility

7 Signs Your Release Process Is More Fragile Than You Think

Mon, 11 May 2026 06:37:01 +0000

Most of the release processes look good. Pipelines are green, deploys happen all the time, and nothing is on fire. When something breaks, you only see the cracks, and suddenly you’re trying to answer questions that your tooling doesn’t help with.

DORA metrics are speed and stability. But they don't tell you if rollback works, if approvals live outside of Slack, or if the artifact you tested is the one in production. You can hit all four DORA metrics and still be one bad deploy away from a bad day.

Here are seven signs your process is more fragile than you realize. Individually they seem small. But they don't hold up well under pressure together.

1. "Who deployed this?" takes more than a minute to answer

This is asked almost as soon as a serious incident occurs. If you have to go through CI logs, scroll through Slack, or hope someone remembers to answer your questions, your release process has no operational memory.

The data is out there somewhere, but it’s not structured as a release record; it’s scattered across pipeline runs, commit messages, conversations. Running a CI does not capture the actual deployment intent, and so engineers open multiple systems during incident calls to reconstruct the course of events.

2. The rollback lives in a doc, not in the system

A healthy, well-defined rollback scenario is of utmost importance. Most teams talk about having a rollback plan, but hardly anyone’s ever put one to the test during a real incident.

Here’s what really happens: you scramble to find the old tag, kick off a pipeline, cross your fingers that the environment hasn’t changed, and try to remember all those finicky parameters. Every single step is a chance for things to break. As a post-mortem on approval-gated releases said it directly: “rollback instructions existed in documents, but not in executable policy.”

So you end up staring into that gap, right when pressure’s at its worst. And here is where you start to improvise chaotically.

3. Deployments come down to whoever’s got access and enough urgency to use it

Production access until there is an explicit process usually comes down to who can deploy and needs to do it quickly. That feels like efficiency on a normal day. In a risky change it gets murky: was this checked? this the correct artifact? who is responsible for the go decision?

If you’ve got repo access with deploy permissions, merging code is often synonymous with deploying it. That’s not really a decision, that’s just the way things happen when CI is doing the deployments.

From what I can tell this only happens when something has gone wrong and suddenly no one knows exactly who owns the release.

4. Your deployment metrics are good because you’re measuring the wrong thing

Pass/fail is the wrong indicator here. A release that makes it through on the third try after two reruns is not the same as one that went through cleanly the first time.

The rerun ratio is the most overlooked stat by most teams. Successful releases with high rerun rates are symptoms of brittle steps, flaky dependencies, and environment inconsistencies (even when the terminal status is green). One of the most underdiagnosed CI/CD challenges is environment inconsistency, as it doesn’t show up in pass/fail rates until a regression hits production.

If your deployment telemetry does not distinguish between first-run success and ultimate success, you are missing a reliability signal that is important.

5. Staging and production run different artifacts

If you rebuild per environment instead of promoting the same artifact, there’s no guarantee that what passed QA is what ends up in production. Dependencies can resolve differently, configs can drift, and suddenly you’re debugging a bug you’ve never seen in staging.

The fix is simple in theory: build once, promote the same artifact everywhere. But many CI setups rebuild by default, because that’s how the pipeline is wired.

Most of the time, no one notices until a regression makes it obvious.

6. A CI platform outage also freezes your ability to deploy a hotfix

If deployments run through your CI, a platform issue can take out both builds and releases at the same time. The moment you most need to ship a fix is the moment you can’t.

I can say that this isn’t rare. Looking at a set of failed or delayed releases, the root cause was often not bad code but a fragile setup: wrong variables in CI, missing approvals, and unclear artifact versions.

When your release layer depends entirely on your build layer, their availability becomes the same.

7. Deployment knowledge concentrates on one or two people

What I mean is that the team's ability to ship depends on specific individuals being present and available.

You’ll agree that this is the least dramatic sign. Yet I’m convinced it’s the most damaging long-term. Teams usually notice it when the "deployment person" takes a holiday and suddenly nobody is confident releasing anything. Or when someone leaves and takes the full process knowledge with them.

Deployment governance isn't just an operational concern. It's a scaling and team health concern.

Why do these signs appear together?

None of these is a crisis on its own. That’s why they tend to stick around for so long.

Together, they point to a release process optimized for the easy case: normal days, routine deploys, no incidents. The root cause is the same across all seven. CI and CD are different responsibilities. One is about validating code, the other is about governing how it moves. But most setups try to handle both with a single system designed for only one of them.

If you've gone deeper on the CI vs. CD distinction, the pattern becomes clear quickly. The coupling feels efficient early and becomes fragile at scale. Build maintainers end up carrying production risk decisions they don't own. Deployment logic accumulates inside pipelines built for something else.

As one measurement framework puts it: standardization doesn't remove complexity. It makes complexity measurable. And once it's so, you can improve it deliberately instead of reacting to isolated incidents.

How many did you recognize?

At first, you’ll most probably notice one or two. That’s nothing unusual; every team goes through this before major issues show up. But when you start seeing four or five, here’s where cracks are about to show. Things still run, but it’s clear the process is bogged down and getting heavier than it should be. Hit six or seven, and it’s trouble. The next incident is probably going to be costly for your team. Not just from any bug, but because all that extra process is slowing everything to a crawl. The faster you notice this pattern, the better.

What to do about it

The good news is that you don’t have to dismantle your pipeline to fix any of this.

The main change here is splitting up the purpose of CI and the purpose of releasing. CI’s whole job is to build, test, and create an artifact. And it does that just fine. Everything after that (what version goes where, who approved it, and what happens if it fails) is a whole other responsibility that needs its own structure.

So, when you put this in practice, you end up with four key things:

build the artifact once, then push it to each environment instead of rebuilding at every stage
make approvals a real, tracked step, not just a message in Slack
test your rollback process before you face an actual incident
keep your release history somewhere safe, so it doesn’t disappear on your CI failure

You don't actually need a large platform. This thin layer between build and release, which owns the release decision, captures the audit trail and handles rollback as policy rather than improvisation, removing most of the ambiguity without much overhead.

I’ve seen this pattern enough times to know the structure problem only reveals itself under pressure. Everything is fine until it isn't. And when something breaks at a bad time, the question is not whether you can fix the bug, but whether you can understand what happened quickly enough to do something about it.

How I built a self-hosted PaaS on AWS from scratch - no Docker, no Kubernetes

Sat, 09 May 2026 16:48:55 +0000

Every time I pushed to GitHub, the ritual was the same: SSH into the server, git pull, restart the process (hoping the port was free), and pray nothing broke. Logs were scattered, rollbacks were manual, and TLS felt like an afterthought. That works for a small demo, but it gets old very quickly. I wanted push-to-deploy: commit, push, wait for health checks, and get a live URL.

I knew platforms like Render and Fly.io solved this elegantly - but I wanted to understand how. Not at the API level. At the level of what actually happens when a push triggers a deployment.

So I built Forge: a small self-hosted PaaS. A GitHub push becomes a running process on my own AWS infrastructure, with health checks, logs, routing, TLS, rollback, and metrics. No Docker daemon. No Kubernetes cluster. Just Linux processes, cgroups, namespaces, Caddy, SQLite, Terraform, and Ansible.

However, Forge became much more than «run my app after a push». It forced me to learn about VPCs, Linux process isolation, state machines, TLS routing, CI/CD, observability and failure recovery.

The architecture was not designed perfectly upfront. It was shaped by real problems - and that is what this article is actually about.

Check it out at Forge's Github.

What Forge is
Infrastructure
The deploy lifecycle
The problems that shaped Forge
Observability
What I would do differently

What Forge is

Forge is a small self-hosted PaaS. A GitHub push becomes a running process on my own AWS infrastructure, with health checks, logs, routing, TLS, rollback, and metrics. There is no Docker daemon and no Kubernetes cluster.

At a high level, the system has one Go control plane, one or more C worker agents, and Caddy as the edge reverse proxy. The control plane receives webhooks, validates manifests, stores state, schedules tasks, and updates routes. The worker builds and runs applications.

The control plane is written in Go. It exposes a REST API, processes GitHub webhooks, schedules deployments, and manages a Caddy reverse proxy via its Admin API. State lives in SQLite in WAL mode - Write-Ahead Logging, which allows concurrent reads without blocking writes, making it a good fit for a single-node system with frequent status updates. Secrets are encrypted at rest with AES-256-GCM.

The worker agent - forge-agent - is written in C11. It runs on a private worker VM, polls the control plane for tasks, and executes builds and processes. The build runner is also C11: it sets up cgroups v2 limits and Linux user, PID, and mount namespaces before running any user-supplied command.

Apps declare everything in a forge.yaml:

name: myapp
runtime: python3.11

build:
  commands:
    - python3 -m venv .venv
    - . .venv/bin/activate && python -m pip install -r requirements.txt

run:
  command: . .venv/bin/activate && uvicorn app:main --host 0.0.0.0 --port $PORT
  port: 8000

resources:
  memory: 256M
  cpu: 0.5

health:
  path: /health
  interval: 10s
  timeout: 3s
  retries: 3

Infrastructure

The infra journey was not linear. The original plan was Oracle Cloud's Always Free tier: two Ampere A1 cores and 24 GB of RAM at zero cost, forever. In practice, free A1 instances were consistently unavailable in the regions that mattered - capacity errors on every attempt, for days. When instances did provision, ARM64 build toolchain issues and inconsistent networking behaviour on the free tier made the setup brittle enough to abandon. The experiment taught me that «free» and «reliable» tend to be orthogonal properties.

The next step logical step was to try AWS Free Tier: t3.micro instances, 12 months, no cost. That worked - until it didn't. The free tier has no NAT gateway. The worker, architecturally and in terms of security, should sit in a private subnet with no public IP, which means it needs a NAT gateway to reach GitHub for clones and package registries for builds. A NAT gateway costs $0.048/hour before data transfer (in May 2026) - about $35/month just to exist - a number that surprised me. That single line item pushed the project from «free experiment» into «this has a real monthly bill» (which, as a student, is not nothing). Add two t3.micro instances, two 30 GB gp3 volumes, two public IPv4 addresses, and a Route 53 hosted zone, and the number lands around $65/month for an always-on setup. Not expensive in absolute terms, but meaningfully more than zero - and a good reminder that private subnets are not free architecture.

The AWS layout is deliberate. The control plane is a t3.micro in a public subnet - it accepts HTTPS from the internet and serves as the single entry point. The worker is a t3.micro in a private subnet with no public IP. A NAT gateway gives it outbound access for Git clones and package downloads.

Security groups enforce the topology:

The control plane accepts 80/443 from the internet, and SSH, Prometheus, Alertmanager, and Grafana only from my admin CIDR.
The control plane API on port 8080 is only open inside the VPC.
The worker accepts nothing from the internet - SSH, the metrics exporter on 9108, and app ports 20000–39999 are open only from the control-plane security group:

resource "aws_security_group" "worker" {
  name   = "forge-worker"
  vpc_id = aws_vpc.forge.id

  ingress {
    description     = "Application ports from Caddy/control plane"
    from_port       = 20000
    to_port         = 39999
    protocol        = "tcp"
    security_groups = [aws_security_group.control_plane.id]
  }
}

Caddy runs on the control plane and proxies HTTPS traffic to private worker ports. The worker is never directly reachable from the internet.

Terraform provisions the VMs and network. An Ansible playbook builds the Forge binaries on the remote host, renders env files from templates, installs systemd units, configures Caddy, Prometheus, Alertmanager, and Grafana, and validates service health after every restart.

The deploy lifecycle

A deployment starts when GitHub sends:

POST /api/v1/webhook/github

The control plane:

Verifies X-Hub-Signature-256 with HMAC-SHA256.
Checks the repo and branch against allowlists.
Validates the commit SHA format.
Clones the repo and parses forge.yaml - unknown fields are rejected via gopkg.in/yaml.v3 with KnownFields(true), so misconfigured deploys fail before any worker task is created.
Creates a pending deployment in SQLite.
The scheduler picks an online worker based on available CPU and memory headroom.
forge-agent claims a build task, and the build runner executes the build commands inside cgroups v2 limits and Linux namespaces. It is not Docker or Firecracker, but it exercises the primitives directly. In production mode, if namespace isolation is unavailable, the build fails closed.
The agent starts the app process with $PORT injected, then health-checks http://127.0.0.1:$PORT/health.
Once health passes, the control plane updates Caddy's route via the Admin API - without restarting the Caddy service.

If health checks fail, the new process is killed. If a previous deployment was running, Forge restores the previous Caddy route.

The problems that shaped Forge

The interesting part of building a deploy platform is not the happy path. It is the sequence of failures that reveal what the system actually needs.

DNS and TLS were not instant. ACME (the protocol that Let's Encrypt and Caddy use to issue certificates automatically) validation sometimes failed because some resolvers/ACME vantage points still saw the old DNS state after others had already updated. The Cloudflare proxy mode added further confusion: Caddy thought the record pointed to Cloudflare, not to my instance. The deeper issue was that Caddy should not issue certificates for arbitrary hostnames - only for domains that Forge knows have a running deployment. This led to implementing a /api/v1/tls/ask endpoint that Caddy's On-Demand TLS calls before issuing any certificate. If the hostname is not a known running deployment, the certificate request is rejected.

CI is also a production actor. GitHub Actions deploys failed because the security group allowed SSH only from my local admin IP. The ephemeral runner had a different IP. Then the correct IP was allowed, but the SSH key was wrong. Then the key was right but passphrase-protected, which does not work cleanly in an automated pipeline. Each failure pointed to the same conclusion: CI/CD is not just scripting - it is an infrastructure actor that needs the same treatment as any other system: proper credentials, scoped permissions, and explicit access. That led to cleaner thinking around OIDC, IAM roles, temporary ingress, and deploys that run only after CI has passed.

Green Ansible did not mean healthy Forge. A playbook would complete with all tasks marked ok or changed, and Forge would not actually be usable. Grafana needed the API to be ready before the admin password could be reset programmatically. Remote Go builds failed on checksum mismatches due to version inconsistencies. Small VMs ran out of memory during heavier builds because there was no swap configured. The lesson: provisioning is not just installing packages. It is verifying that the resulting system behaves correctly, which means adding real readiness checks after every service restart, not just checking that the process started.

The database said running; reality disagreed. This was the most important and frustrating runtime bug. A deployment could be marked running in SQLite while the worker process was dead, the agent had stopped heartbeating, or the VM had been restarted. Forge had state transitions but no reconciliation. Adding a scheduled reconciliation loop - checking running deployments against actual agent heartbeats, re-syncing Caddy routes on control-plane restart, and handling stale records - changed the system from a state machine into something closer to a control loop.

Early versions assumed apps would bind to fixed ports like 8000, which caused collisions. Forge now allocates a host port from 20000-39999 and injects it as $PORT. The run.port value in forge.yaml is now best understood as a manifest/default fallback; the platform owns the actual runtime port.

Security hardening came from uncomfortable edge cases. An empty admin token must never authenticate - but naive string comparison with an empty string might allow it. GitHub webhook signatures need to be validated with constant-time comparison to avoid timing attacks. The repo and branch allowlists need to exist and be non-empty - an unconfigured Forge should not accept arbitrary webhook payloads. The agent runs as an unprivileged forge user, not root. None of this is exciting to implement. All of it is necessary.

Private repos exposed a design limitation. Repo credentials were initially stored per owner/repo. A GitHub classic PAT is scoped to a user account - not to a single repository. Storing the same token three times for three repos in the same org was operationally wrong. Forge now supports owner-level credentials with per-repo override: a credential for myorg covers all of that org's repos unless a more specific entry exists.

Observability

Forge exposes Prometheus metrics from two sources. The control plane serves its own metrics directly on :8080: current deployment/task status counts, and agent heartbeat status. The worker side is trickier - the C agent serves Prometheus text (CPU, memory, running process count, heartbeat timestamps) over a Unix socket using a minimal HTTP-compatible response. forge-exporter is a small Go bridge that reads that socket and translates it into a standard Prometheus HTTP endpoint on :9108, so Prometheus can scrape it without exposing a TCP listener from the agent itself. This keeps the agent lean and focused on process management, without exposing a TCP HTTP listener from the C agent itself.

The metrics exposed:

forge_deployments_total{status}: deployment rows grouped by current status.
forge_tasks_total{status}: task rows grouped by current status, across build, run, and stop tasks.
forge_agents_online: agents with a recent heartbeat.
forge_agent_cpu_used, forge_agent_memory_used_bytes, forge_agent_memory_capacity_bytes, forge_agent_processes, and forge_agent_last_heartbeat_seconds: worker-level metrics exported by the agent.

The worker metrics are not per-app or per-deployment yet. The agent exposes them over a Unix socket, and forge-exporter bridges that socket to HTTP on :9108, so Prometheus can scrape it. Grafana has a small dashboard for deployment state, online agents, and worker memory.

The alert I check first, arguably very simple and the most important:

- alert: ForgeNoOnlineAgents
  expr: forge_agents_online == 0
  for: 2m
  labels:
    severity: page

If this fires, the control plane has no workers to schedule on. Pending deployments will stay pending indefinitely. No other metric captures that specific failure mode.

Grafana sits behind the control-plane security group, accessible only from the admin CIDR. Alertmanager is installed and wired into Prometheus, but the default receiver is still a placeholder; the next step is connecting it to Slack, Discord, PagerDuty, or another notification channel.

Observability was added incrementally, not upfront. But it changed how I debug. The difference between «something is wrong» and «the agent on worker-1 stopped heartbeating 4 minutes ago» is the difference between guessing and knowing.

What I would do differently

Start with the agent in Go, then move the low-level isolation parts to C. The C agent is the most technically interesting part of Forge — direct cgroup writes, raw syscalls, /proc parsing. It is also the part that is hardest to iterate on quickly. Starting in Go would have accelerated the initial development; the C build runner is the right place to draw that boundary. Debugging a hand-rolled HTTP client while also debugging namespace setup is educational, but painful.

Design the store interface earlier. SQLite WAL is a great choice for single-node Forge - zero external dependencies, concurrent reads, durable writes. But the store interface leaked SQLite-specific details into the server layer earlier than it should have. A cleaner repository interface from the start would have made testing easier and kept the option of replacing the backend open. If I ever wanted multi-region or leader election, I would want that boundary to be cleaner.

Use remote Terraform state from day one. Local state works until you need to apply from a different machine, recover from a failed apply, or share infrastructure with a CI pipeline. The migration cost is low but the friction of doing it mid-project is real.

Finally, I would be clearer about the observability boundary from day one. Forge currently has worker-level metrics, which are enough to see whether a node is busy, but not enough to attribute CPU or memory to a specific app. Per-app resource accounting would be a separate feature, not something the current metrics magically provide.

Conclusion

Forge is not trying to beat Kubernetes. It is a learning project that exposed the real mechanics behind deployment platforms: the state machines, the reconciliation loops, the TLS edge cases, the security groups, the cgroup limits, the rollback logic, the alerting gaps.

The most useful thing it taught me is that the interesting engineering in a deploy platform is not the deploy itself - it is everything that happens when the deploy goes wrong.

Clone it, write a forge.yaml, open a PR, and watch your own infrastructure deploy it.

Forge Live Demo.

I got tired of writing runbooks after incidents. So I'm building something.

Thu, 23 Apr 2026 12:06:49 +0000

Working on a microservices-based platform, I kept running into the same frustrating cycle:
Something breaks → we figure out the fix in real time → someone says "we should document this" → nobody does → same thing breaks 3 months later.
Even when runbooks existed, they were either too generic ("restart the pod") or specific to whoever wrote them and completely useless to the next on-call person.
So I'm building RunbookAI — you describe your stack once (AKS, EKS, Node.js, PostgreSQL, whatever) and it generates incident playbooks grounded in SRE best practices. Stack-aware, not copy-pasted templates.
Very early stage — just launched a waitlist today.
My question to the Dev.to community: What's the worst runbook experience you've had? Either nonexistent, outdated, or just completely wrong? Would love to understand the real pain before I build the wrong thing.
Waitlist: runbookai.in

The Butterfly Effect in Networked Agents

Tue, 31 Mar 2026 16:06:33 +0000

Imagine your bank's customer support is entirely run by AI agents. A triage agent classifies your issue and routes you to specialized agents — billing, fraud, technical support — with an escalation agent handling complex disputes.

Your bank upgrades the triage agent. The new model is smarter — edge cases that used to get misrouted to billing now correctly go to the fraud team. Every test passes. Accuracy improves. The upgrade is a success.

Within a week, the fraud agent is drowning. Its volume doubles, and half the new queries are ambiguous cases that it was never calibrated for. False positives spike. Legitimate customers get their accounts frozen. The downstream escalation agent, designed for rare, complex disputes, is flooded with angry customers who can't access their money. Resolution times triple.

Every agent is doing exactly what it was designed to do. No agent is broken. The system is broken.

Welcome to the butterfly effect in networked agents — a small improvement in one agent silently reshaping the operating conditions of every agent downstream. Not by sending wrong data, but by sending a different distribution of correct data.

Why This Is Different From Traditional Software

In traditional software, when you change the behavior of an upstream service, you think about the downstream impact. You version your API. You communicate breaking changes. You run integration tests.

But this isn't a breaking change. The triage agent's API didn't change. Its output schema is identical. The downstream agents receive the same data types in the same format. What changed is the statistical profile of the requests flowing through the system — and we have no tooling to version, communicate, or test for that.

Traditional monitoring doesn't help either. The fraud agent's error rate went up, so you investigate. Its model is fine. Its prompts are fine. You A/B test new prompts, retrain on fresh data, and tweak thresholds. But it still won't match its previous performance, because it was dealing with simpler, pre-filtered cases before. The problem isn't the fraud agent. The inputs changed, not the agent.

This isn't a new problem, strictly speaking. Data drift has been a known challenge in machine learning for years — models degrade when their input distribution shifts away from their training data, whether due to upstream changes or seasonal shifts in user behavior. But historically, it stayed low-priority. ML-based systems were mostly self-contained. A recommendation engine, a fraud detector, a search ranker — each operated within a single cluster in companies, fed by data pipelines that teams controlled. Building and deploying ML systems required specialists who could train models and maintain infrastructure. ML systems were rarely chained together, so distributional shifts were rarely caused by upstream ML system updates.

Agents change the equation entirely. The barrier to building and deploying an agent is a fraction of what it takes to build traditional ML software. You don't need a team of ML engineers to train and deploy a model — you need a prompt and an API key. Every company can now embed agents in every component of its software. And once agents are everywhere, networking them becomes inevitable. When agents from different companies are chained together, the input distribution of every downstream agent is shaped by the behavior of upstream agents it doesn't control, can't monitor, and may not even know about. Data drift goes from an internal ops concern to an inter-organizational reliability problem — and the tooling hasn't caught up.

This is the fundamental asymmetry: their improvement is your regression. The triage agent got better at its job, and that made every downstream agent worse at theirs. Not because anything broke, but because the downstream agents were implicitly calibrated to the old distribution, and nobody knew that was a dependency.

Now, Imagine This Across Companies

The call center example happens inside one organization. That means one team can, in theory, detect the problem, trace the cause, and recalibrate the chain — even if it takes weeks.

Now imagine the agents are owned by different companies.

The triage agent is a third-party service your bank subscribes to. The fraud detection agent is from another vendor. The escalation routing goes through a customer experience platform built by a fourth company. Each company maintains its own agent, model, and deployment schedule.

The triage vendor upgrades their model. Your fraud vendor's agent starts underperforming. Your fraud vendor investigates, finds nothing wrong, and blames your data. You escalate to the CX platform vendor, which sees degraded metrics but has no access to the fraud or triage models. The triage vendor, meanwhile, is celebrating improved accuracy numbers and has no idea their upgrade destabilized two downstream systems they've never heard of.

Nobody can see the full chain. Nobody has the access, the context, or the incentive to diagnose a cross-boundary distributional shift. The problem festers until someone manually connects the dots — or until it shows up in quarterly customer churn numbers, triggering a post-mortem that takes months.

This is where the butterfly effect becomes truly dangerous. Inside one company, it's an operational headache. Across companies, it's an invisible failure with no owner.

What the Industry Needs

We spent decades building reliability infrastructure for traditional software composition — type systems, semantic versioning, contract testing, and dependency management. For networked agents, we have nothing equivalent. Here's what needs to exist.

Distribution-aware contracts. Agent-to-agent interfaces need to specify not just data types but expected input distributions. "This agent expects 80-90% of inputs to be likely fraud cases" is a distributional contract. If the upstream agent's behavior shifts that distribution outside the agreed range, it should be treated as a breaking change — even if the schema is identical.
Dependency-aware deployment. When you upgrade one agent, the deployment pipeline should automatically identify which downstream agents are affected and trigger re-evaluation. This is the agent equivalent of "which services depend on this one?" — except the dependency isn't in the API contract, it's in the statistical properties of the data flowing between them.
Continuous distributional monitoring. Every agent should track the statistical profile of its inputs over time — volume, composition, feature distributions. When the profile shifts beyond a threshold, alert. This is data drift monitoring applied to agent communication, and it should be as standard as uptime monitoring.
Automated recalibration pipelines. When a distributional shift is detected, downstream agents should have automated pipelines to re-run evaluations, adjust thresholds, update few-shot examples, or trigger fine-tuning. Manual recalibration doesn't scale — especially across organizational boundaries.

What You Can Do Today

The industry-level solutions don't exist yet. But if you're building agent systems, there are practical steps you can take now.

Monitor input distributions, not just output quality. If the volume or composition of queries hitting an agent shifts significantly, that's a signal — even if no individual query looks wrong. Track category distributions, query length distributions, and topic clustering over time.
Treat upstream agent upgrades as deployment events for your entire chain. When you upgrade one agent, re-run evals on every downstream agent. Don't just test the agent you changed. Test the system.
Build recalibration into your ops process. Have a runbook for "upstream distribution changed." Know in advance which few-shot examples, thresholds, and fine-tuning data need updating, and how to update them quickly.
Design agents to be robust to distributional shift. If your fraud agent completely falls apart when its input distribution shifts from 90% fraud to 50% fraud, it's too tightly calibrated. Build in headroom. Test against varied distributions, not just the current one.

The Road Ahead

We're at an inflection point. The industry is building hierarchies of agents — within companies and across them — and deploying them into high-stakes workflows without the reliability infrastructure to support composition.

The butterfly effect in networked agents isn't a theoretical concern. It's the inevitable consequence of chaining systems that are each calibrated to their current operating conditions, without accounting for the fact that those conditions are shaped by every other agent in the chain. One upgrade anywhere changes operating conditions everywhere downstream.

We solved software composability once. We'll solve it again. But the first step is recognizing that the failure mode is different this time. It's not crashes and errors. It's a silent distributional shift that degrades the entire system while every individual component reports healthy.

The teams that recognize this early and build for it will ship reliable agent systems. The teams that don't will spend months debugging invisible failures that no single team caused, and no single team can fix.

If you're building multi-agent systems or thinking about agent reliability, I'd like to hear how you're approaching this. The problem is real, the tooling isn't, and it's a gap worth closing together.

Modern systems generate petabytes of telemetry data, but most teams are still guessing what matters

Wed, 22 Oct 2025 01:05:56 +0000

Why Metrics Might Be Holding Your Observability Strategy Back

Metrics have been the foundation of observability for decades. But what if that foundation is cracked?

Ang Li, director of engineering at Observe, argues that metrics are a relic of outdated infrastructure constraints. They were necessary when systems couldn't handle large volumes of log data. Now, they might be doing more harm than good.

"Metrics force you to predefine what you think will matter before an incident even happens," Li explains. "If you guess wrong, you're effectively blind."

The Problem with Predicting Problems

The core issue isn't just philosophical. It's architectural.

Metrics require teams to decide upfront which labels, aggregation windows, and dimensions matter. But modern distributed systems are unpredictable. A metric might tell you API latency spiked, but it won't tell you the spike only affected requests from a specific region with a particular data pattern.

That context gets stripped during roll up. The user, request, payload, deployment, or dependency that triggered the failure disappears. You're left with a number that says something is wrong, but not why.

Li recalls troubleshooting out-of-memory errors at Observe. Engineers stared at pages of garbage collection metrics without clear insight. "Having individual request traces and logs is still critical to identify the culprit requests that brought down the system," he says.

How Cloud Economics Changed the Game

The technical landscape has shifted dramatically. Cloud infrastructure now makes it economically viable to store and analyze raw telemetry data at scale.

Observe's architecture centers on treating all telemetry as structured event data. They use Amazon S3 for storage with 10x compression, providing low-cost long-term retention for petabytes of data. Above that sits what they call the O11y Knowledge Graph, which transforms raw data into recognizable entities like customers, shopping carts, pods, and containers.

This approach required solving three key engineering challenges: scalable ingestion that adjusts to dynamic volume peaks, flexible transformation pipelines that model different system aspects, and intelligent query parallelization that handles 10GB the same way it handles 10TB.

"The customer should not care whether they are querying 10GB of data or 10TB," Li says. "The results should return in similar time frames."

Making Raw Data Analysis Viable

Processing raw telemetry instead of pre-aggregated metrics sounds computationally expensive. But several optimizations make it practical.

Columnar databases like Snowflake enable higher compression ratios than row-based systems. This reduces both storage costs and query scanning overhead. Observe also skips indexing by default, which is why search-based tools like Splunk become prohibitively expensive.

Aggressive query caching reuses existing results. Dashboard refreshes only re-query the changed portions, saving significant computational costs.

The Role of Metrics Going Forward

This isn't about eliminating metrics entirely. They still have value for infrastructure-level monitoring where detailed information is unavailable, like CPU utilization and OS metrics. They're efficient for capacity planning and catching broad systemic issues.

The shift happens when you need to understand why something broke in complex distributed systems.

"The real answer is combining metrics for high-level indicators and observability for deep contextual investigation," Li explains. "You need to recognize where you need richer data to maintain reliability at scale."

What This Means for Engineering Teams

Moving from metrics-first to direct data analysis changes daily workflows. Instead of jumping between dashboards and guessing which metric might be relevant, engineers can start with the actual data behind the behavior.

This makes troubleshooting more investigative and less reactive. Teams spend less time maintaining dashboards and more time understanding system behavior.

But it requires discipline. Raw data collection demands good instrumentation practices. Schema drifts can break downstream pipelines or alerting rules. Important log and trace messages need protection against accidental changes through unit tests.

AI as the Investigative Partner

AI is becoming crucial for navigating raw telemetry data. But not in the way you might expect.

"The power of AI lies in the capability to explore the data in steps to narrow down the root cause," Li says. AI can launch an initial query to aggregate results, then drill deeper based on what it finds, repeating the process iteratively.

Human engineers could follow the same routine, but it's time-consuming and difficult to execute rigorously. AI replicates the investigative process a skilled engineer would take during an incident.

For trustworthiness, Observe presents every step of the reasoning path. Engineers can modify queries and conduct their own analysis to confirm AI findings.

The Migration Question

Legacy observability vendors struggle with this shift because their rigid data models don't fit the diverse schema of raw data. They lack capabilities for advanced analytics like joins and window functions over massive datasets.

Li's advice for engineering leaders considering a migration? "Think back to a time when you were troubleshooting a tough production issue, only to realize the most relevant raw logs weren't captured because of a cost-saving decision."

Modern cloud economics have eliminated that tradeoff. The question isn't whether you can afford to store raw telemetry data. It's whether you can afford not to.

Splunk unveiled AI agents that can debug your code, triage incidents, and monitor infrastructure.

Tue, 09 Sep 2025 13:30:33 +0000

Splunk's AI-First Future: What Developers Need to Know About .conf25 Announcements

Splunk .conf25 marked a turning point for how developers and engineers will interact with observability and security tools. The company announced three major product updates that shift from traditional monitoring to AI-powered autonomous operations. Here's what technical teams need to understand about these changes.

The Cisco Data Fabric: Turning Machine Data Into AI Fuel

The centerpiece announcement is the Cisco Data Fabric, which addresses a fundamental problem: most machine data sits unused in silos. This framework transforms logs, metrics, and telemetry into what Splunk calls "AI-ready actionable intelligence."

For developers, this means several practical changes. The new Time Series Foundation Model (TSFM) can analyze patterns in your application logs and infrastructure metrics to predict failures before they happen. Instead of reactive debugging, you get proactive insights about potential issues.

The system works across edge, cloud, and on-premises environments without requiring you to centralize all data in one location. This federated approach matters because moving petabytes of data for analysis isn't economically feasible for most organizations.

Splunk is releasing the TSFM as an open model this fall, available on platforms like Hugging Face. You'll be able to fine-tune it with your specific data patterns, creating custom models that understand your application's unique behavior.

Security Gets Agentic AI Treatment

The security announcements introduce two new packages: Splunk Enterprise Security Essentials and Premier. Both include AI agents that handle routine SOC tasks autonomously.

The Triage Agent evaluates and prioritizes security alerts automatically, even for low-volume edge cases that traditional rule-based systems miss. For development teams, this means fewer false positive alerts disrupting your workflow.

The Malware Reversal Agent can analyze malicious scripts line by line and extract indicators of compromise. This speeds up incident response when your applications encounter threats.

Perhaps most relevant for DevOps teams is the AI Playbook Authoring feature. You can describe response procedures in natural language, and the system converts them into functional SOAR playbooks. This eliminates the tedious work of manually scripting incident response workflows.

Observability Becomes Predictive

The observability updates focus on three areas: proactive detection, AI infrastructure monitoring, and business impact correlation.

AI-Directed Troubleshooting automatically analyzes incidents and surfaces root causes in both Splunk Observability Cloud and AppDynamics. When your application has performance issues, the system identifies the underlying problem without manual investigation.

For teams building AI applications, new monitoring capabilities track the health, security, and cost of AI infrastructure. You can monitor LLM performance, token usage costs, and agent behavior. This addresses the "tokenomics" challenge that Kamal Hathi mentioned.

The unified observability approach combines application performance data with business metrics. You can correlate code deployments with business outcomes like checkout completion rates or user engagement metrics.

Integration Strategy: Multi-Vendor Reality

Splunk emphasizes that these capabilities work across vendor ecosystems. The Snowflake integration demonstrates this approach, you can run SPL-like queries against Snowflake data directly from Splunk without data movement.

This matters for development teams using diverse toolchains. You're not locked into Cisco infrastructure to benefit from these AI capabilities. The system connects to Amazon S3, Apache Iceberg, Delta Lake, and Microsoft Azure storage systems.

What This Means for Your Development Workflow

These announcements signal a shift from reactive to predictive operations. Instead of waiting for alerts to fire, AI agents will identify potential issues and often resolve them automatically.

The practical impact depends on your role. If you're a developer, expect fewer interruptions from false alerts and faster root cause identification when real issues occur. Site reliability engineers will spend less time on routine incident response and more time on strategic improvements.

The challenge is trust. As Hathi acknowledged, enterprise customers need confidence in AI decision-making. Splunk addresses this with extensive logging and human oversight capabilities. AI agents recommend actions, but humans retain approval authority for critical decisions.

Implementation Timeline and Considerations

The rollout happens in phases. The Cisco Data Fabric and Snowflake integration launch September 8. Security and observability features follow September 9.

Deployment complexity varies by organization size and existing toolchain. Splunk designed these as turnkey solutions, but integration with legacy systems requires planning.

Training requirements are minimal for end users since the AI handles most complexity. However, administrators need to understand how to configure agent behavior and approval workflows.

The Bigger Picture

These announcements reflect broader industry trends toward autonomous IT operations. Traditional monitoring tools struggle with the scale and speed of modern applications. AI-powered systems can process information and respond at machine speed.

The key question is whether organizations are ready to delegate operational decisions to AI systems. Splunk's approach provides a middle ground—AI handles routine tasks while humans focus on strategic decisions.

For development teams, this evolution promises fewer late-night pages and more time for feature development. But it also requires new skills around AI system oversight and configuration.

The future Splunk envisions isn't fully autonomous operations, but AI-augmented human teams. Whether this vision matches reality depends on how well these systems perform in production environments over the coming months.