Recent posts tagged sre

In February 2020 the safe holding the internet's root signing key would not open. IANA's published, hand-initialed ceremony scans show the custody behind the cryptography: a repair script written before anyone drilled, and a signing that took four mi...

Most SRE teams do not lose the bulk of an incident because nobody noticed something was broken. In mature environments, alerting is already fast enough to wake the right people, interrupt the right channel, and open the right incident record. The har...

TSB's 2018 move off the Lloyds platform could not be rolled back. The FCA and PRA notices show the go decision rested on a supplier letter, a test run on one of two data centres, and a memo that left the letter's caveats behind. On 10 April 2018, TS...

> TL;DR. Google's SRE book works out that a 24/7 on-call rotation needs at least eight engineers. A team of two or three cannot reach that number, so it has to change what the number is built from: which alerts page a person, which hours are covered,...

Recently, I interviewed for a Staff Site Reliability Engineer SRE role, where I encountered some interesting system design challenges around building a scalable, highly available, cloud-native application. The discussion reinforced an important less...

A pod dying and a system recovering are not the same thing. That gap is where most of the interesting failures actually live, and it's what pushed me to build fault-sentinelhttps://github.com/aashiruu/fault-sentinel, a lightweight, Kubernetes-native ...

GitHub published its August availability report this week, and one sentence in it is worth more than the rest of the document: "our shared infrastructure services have not kept up with our month-over-month actions growth and peak load." Five inciden...

A signal has three states and most code stores two. When “cannot measure” is written with the same value as “fine,” the alarm branch becomes unreachable in exactly the situation the alarm exists for. On 7 December 2021, the operations teams inside...

Originally published on Vibe Agent Makinghttps://vibeagentmaking.com/blog/persister-cells/. In 1944, the physician Joseph Bigger hit a Staphylococcus culture with penicillin and watched almost all of it die. A tiny remnant survived, which is unremar...

> TL;DR. The most common false alert in uptime monitoring is one probe location having a bad network day. I treated that as a design requirement from day one, so the checking pipeline has two gates: a region only counts as down after it fails the sam...

> TL;DR. During an outage your status page has a second job: holding trust while the fix is still in progress. Write plainly. Say what customers feel first, say what you are doing, and promise a time for the next update. Then keep that promise, even ...

Cover photo by the blowuphttps://unsplash.com/@theblowup on Unsplash. I build an uptime monitor, so this question lands in my inbox a lot: is 98% uptime good? For a public website or a paid API, no. For an internal tool or a side project, it is fin...

A status page is the one dashboard a company publishes about its own service. It is also the one place where the company has a reason to look good. That is a problem. If the page can be edited to look better than reality, it stops being useful. So wh...

If you have ever seen a team argue about whether they can ship a risky change, an error budget is the tool that ends the argument with a number. Here is how it works, with the formulas and a simple way to turn it into alerts. What an error budget is...

AI is shipping code faster than most engineering teams know how to operate it. At Intuit, that tension is front and center — and Akshay Pratinav, Senior Staff Software Engineer, is helping figure out what to do about it. Pratinav works on AI-driven ...

A real-world lesson from a production-like AWS lab challenge The Scenario That Should Scare You Imagine this: your AWS environment has two Availability Zones, public and private subnets, an Application Load Balancer, Auto Scaling. Your architecture...

I spent the last few months building Uptimepagehttps://uptimepage.dev, an open-source uptime monitor and status page written in Rust. This post is the build story: the decisions that shaped it, the parts I rewrote, and the numbers that came out the o...

Controlled Burns for Organizations What the Forest Service knows about change that consultants don't. In late winter, somewhere on the Coronado National Forest, a crew with drip torches walks a ridgeline. The relative humidity is above forty perc...

Hi! so this is my fourth terraform artefact. I'd like to say that creating infrastructure with Terraform is one thing, but using Terraform when reality changes outside the code is another thing. For the fourth artifact in my Terraform x Google Cloud...

Most of the release processes look good. Pipelines are green, deploys happen all the time, and nothing is on fire. When something breaks, you only see the cracks, and suddenly you’re trying to answer questions that your tooling doesn’t help with. DO...