Coder Legion - Recent posts tagged monitoring

A week of green runs, and the pipeline was waiting for me

Tue, 27 Oct 2026 16:02:11 +0000

On 1 September at 10:57 UTC the monthly ingest ran. It opened issue number 3 of the magazine, picked the vertical from the source registry, wrote 19 candidate sources into Postgres with stage = 'discovery', and opened a GitHub issue telling me it was my turn.

Every morning after that, the daily job ran and succeeded. Eight consecutive green runs, from the first of the month to the eighth. The article count for September stayed at zero the whole time.

The gate was doing its job

The daily job does not decide anything by itself. It asks the database which stage to run, and the answer depends on a view:

SELECT id, issue_id, source_url, tier, independent
FROM signals
WHERE stage = 'verify' AND (tier = 1 OR tier = 2 AND independent);

Nothing satisfies that view until I sit in Supabase Studio and tag the sources by hand. Tier 1 is a primary source, a regulator or the organisation that ran the thing. Tier 2 counts only when it is independent of whoever it describes, which rules out a consultancy writing up its own project as a case study.

That judgement is mine on purpose. It is the editorial bar of the magazine, and I did not want a model deciding what counts as proof of anything.

So the decision function found a draft issue with no verified signals, returned niente, and printed its reason to stderr:

advance: signal in attesa di verifica in Studio

Then it exited zero, which is correct. There was nothing for it to do.

What my alarms actually watch

I have three, and I was fairly pleased with them.

A cron that fails opens a GitHub issue on its own. A cron that never starts is caught by a daily watchman that asks the GitHub API when each schedule last fired, because a schedule that GitHub has quietly disabled produces no failure to report. And because a watchman inside the same failure domain is not a watchman, that job also checks in with a Sentry monitor that alarms on silence from outside GitHub.

Between them they cover a job that breaks and a job that never runs. Neither of those happened. What happened was a job that ran, succeeded, and did nothing, for seven days, because it was waiting for a person who had not noticed.

The information was already there

The part I find annoying is that nothing was hidden. The reason is written every single morning, in the run log, in Italian, on stderr. The job knows exactly why it is idle and says so out loud.

What it does not know is how long it has been saying it. niente on the second of the month and niente on the eighth are the same string, and the only thing that separates them is a week of my attention going somewhere else. No alarm reads that log line, and no counter turns it into a duration.

Green means the machine is healthy. I had been reading it as "the work is moving", and those are different claims.

How it ended

On the eighth I opened Studio and tagged 11 of the 19 sources: two from the regulator as tier 1, nine analyst reports as tier 2 and independent. I left out a vendor case study, which fails the independence test by construction, and seven security bulletins whose scraped body is only a title.

After that the pipeline did the rest in a few minutes. Generate wrote the piece from eight sources, embed rebuilt the chunks, export produced the two markdown files, and the judge on the pull request failed me for weak attribution, which is a story for another post.

The September issue went out a week late. Nothing had broken, and every run had been green.

So I do not need a fourth alarm. I need the daily job to count the mornings it has spent answering niente, and to say that number where I actually look.

Online Says It's Connected, HeartBeat Says It Works: Reading Two Status Signals Instead of Guessing

Tue, 27 Oct 2026 01:03:42 +0000

title: "Online Says It's Connected, HeartBeat Says It Works: Reading Two Status Signals Instead of Guessing From Silence"
published: true
tags: iot, mqtt, radar, monitoring
canonical_url: https://249263431.github.io/60GHz-Radar-SDK/guide/2026/10/27/online-heartbeat-radar-health.html
description: "Two fields in a mmWave radar's MQTT output — online and heartBeat — look like the same health check and are not. One proves the link; the other proves the sensor. And neither supports a timeout."

cover_image:

Two fields in a radar's MQTT output get mistaken for each other constantly, and mistaking them is how a health monitor ends up trusted when it should not be.

They answer different questions:

online answers "is the link up?"
heartBeat answers "is the radar itself working?"

	`online`	`heartBeat`
Question answered	Is this device reachable?	Is this device operating correctly?
Reported values	`"0"` offline, `"1"` online	`"0"` abnormal, `"1"` normal
Layer	MQTT connection and broker session	The radar's own operating state
Reported when	On state change only	On state change only
Available on	Sleep radar, fall radar, presence radar	Sleep radar, fall radar

The presence radar publishes its online state on a dedicated topic — /RadarHP/<DeviceID>/sys/online/post; the sleep and fall radars publish online as an attribute in the same property payload as everything else. Note the asymmetry: there is no heartBeat in the presence radar's protocol documentation. The other two report it like this:

{"version": "1.0", "method": "post", "params": {"online": "0"}}
{"version": "1.0", "method": "post", "params": {"heartBeat": "0"}}

Why this distinction is not academic

Consider what has to be true for any heartBeat message to reach your code: the radar has power, it is associated with the Wi-Fi access point, its MQTT session is established, and you are subscribed to its topic. A delivered heartBeat therefore already proves the link is working.

From that, one conclusion follows immediately:

heartBeat cannot report a connectivity failure. Physically and logically, it cannot — a radar that has lost its link has no way to deliver that news.

The reverse holds for online. An online: "0" message is not written by the radar that went offline; it is generated by the broker when the session it was holding stops answering. The radar contributes nothing to that judgement.

So neither field overlaps with the other, and neither can substitute for the other:

If you only watch online, you will see "connected" while the radar sits there reporting a fault about itself.
If you only watch heartBeat, every network outage looks like nothing at all.

The trap: silence is not evidence

Both fields are documented as reported on state change, not on a timer. That single sentence is where most monitoring code goes wrong.

Occupancy. The presence state is event-driven. A room whose state does not change for three hours produces no occupancy messages, and that is correct behaviour — not a fault.
Heartbeat. A radar that boots normally and stays normal has no reason to send heartBeat at all, because no state change occurred.
The one periodic attribute. In the sleep and fall protocols, the only attribute the documentation marks as periodically reported is workDuration (4 bytes, value range 0x3C–0xffffffff, unit seconds) — and it is marked "not yet available" in the same line. workDurationReportFrequency carries the same note. Do not build a keepalive on either of them.

So the tempting pattern — "if I have not heard from the device in 60 seconds, it must be gone" — has no foundation here. Absence of messages is the normal state of a working installation.

What you can do instead

`online`	`heartBeat`	Conclusion	Action
1	1	Link up, radar reports normal	—
1	0	Link up, radar reports a fault	Raise the higher-severity alarm; this is the case only `heartBeat` can surface
0	—	Session dropped, as judged by the broker	Investigate the link; the radar's own state is unknown
no messages	no messages	Most likely nothing changed	Nothing can be concluded

Treat the two as different classes of event with different severities. A lost link is an infrastructure ticket; a radar reporting itself abnormal is a hardware ticket, and it is the one that should wake someone up.

Two rules you must not break

1. Do not disable online or heartBeat with limit_set. The Wi-Fi protocols expose a reporting-suppression command: limit_set / limit_get, where 1 means "suppress this attribute" and 0 means "allow it". It exists for the attributes that report frequently — someoneExists, motionStatus, movementSigns, humanDistance, humanPosition — and it is a legitimate way to cut traffic and noise. Both online and heartBeat appear in that list. Suppress them and you remove the only two signals that stand between you and a timeout guess.

2. Do not try to poll heartBeat. Every neighbouring attribute in those documents — initFinish, workDuration, productMode — carries the line "this attribute supports querying" followed by a get example. heartBeat does not. The only ? it ever appears with is inside limit_get, and that question means "is reporting suppressed for this attribute", not "what is its current value". There is no documented way to ask the radar for that value on demand.

That leaves one workable approach for the things you genuinely need on demand: query the attributes that do support it. If you must confirm a radar has finished starting up, initFinish (0 not complete, 1 complete) is queryable. heartBeat is not, and no amount of polling will change that.

A minimal health monitor

Subscribe, merge the sparse payload, and raise the two classes separately:

import json, time
import paho.mqtt.client as mqtt

TOPIC = "/Radar60SP/<MAC>/#"

state = {}
last_any = {}


def on_message(client, userdata, msg):
    try:
        payload = json.loads(msg.payload.decode())
    except ValueError:
        return
    params = payload.get("params", {})
    if not params:
        return

    # Only the fields that changed are present.
    for k in ("online", "heartBeat", "initFinish"):
        if k in params:
            state[k] = params[k]

    last_any["ts"] = time.time()

    # Two different failures, two different severities.
    if params.get("online") == "0":
        raise_alarm("link", device=msg.topic, severity="infra")
    if params.get("heartBeat") == "0":
        raise_alarm("sensor", device=msg.topic, severity="high")

Note the shape: nothing here fires on elapsed time. Every action is triggered by an arriving value.

If you still want a timeout as a last resort, the threshold must come from measurement, not habit. Log the longest interval your real installation goes without any message during normal operation, then add margin to that. A number you made up will either fire constantly or never fire.

Measure these at commissioning

Two things the documentation does not state, and that you should record per installation rather than assume:

Does the radar announce heartBeat: "1" on power-up? A device that boots healthy may never have a reason to report it. Power one up on the bench with a subscriber attached and write down what actually arrives, and how long after power-up it arrives.
Does online: "1" get republished after a reconnect? If it does not, your consumer must treat the absence of a change as "still online" rather than as unknown.

Both belong in the same commissioning note as the empty-room baseline from Using Distance and Position Output to Debug a Radar Installation.

Maintenance looks like a failure

One consequence of all this that catches people out: the sleep and fall protocols support firmware update over the air — ota_set with otaModule (0 Wi-Fi module, 1 radar module, and the command must be sent before every update), otaInform giving a firmware URL and port, and downloadProgress reporting 0–100. On success the module restarts and then reports its new version number.

During that window the device is legitimately unreachable. If your monitoring treats a missing device as an alarm, a firmware update will page someone. Put planned maintenance in a suppression window, and use the version report afterwards — the presence radar's /RadarSP/<DeviceID>/sys/version/post topic, or the firmwareVersion attribute — to confirm the update actually landed.

The update server is one you host yourself; the documentation states plainly that you build the web server the firmware is fetched from.

Two controls worth knowing about

Reporting suppression (limit_set / limit_get): set 1 to suppress an attribute, 0 to allow it. Use it on the high-frequency attributes to cut traffic and clutter. Leave online and heartBeat alone.
Per-attribute QoS (control_qos in the presence protocol, qos_set / qos_get with Qos_post in the sleep and fall protocols): the default is QoS 1, where 0 is QoS 0, 1 is QoS 1 and 2 is QoS 2. The presence protocol additionally exposes per-attribute keys — Qos_online, Qos_movementSigns, Qos_breathRate, and so on. This is worth knowing because it contradicts the usual assumption that QoS is decided only by the subscriber.

Checklist

[ ] online and heartBeat handled as two separate signals with two separate severities
[ ] No logic concludes anything from the absence of messages
[ ] limit_set used, if at all, only on high-frequency attributes — never on online or heartBeat
[ ] No polling of heartBeat; anything needed on demand read from attributes that document query support
[ ] Power-up behaviour of heartBeat measured and recorded
[ ] Reconnect behaviour of online measured and recorded
[ ] Any timeout threshold derived from measured worst-case silence, not assumed
[ ] Firmware updates placed in a suppression window, and verified afterwards by version report

Where to go next

Getting the stream in the first place: Reading Radar Data in Your Own Application
The broker underneath it: Apache Artemis setup
Keeping several devices straight: Running Several Radars in One Home
Keeping the history: Storing Radar Data

This is not a medical device. A status field tells you what a radar reports about itself; it does not guarantee what the radar detects. For anyone at real risk, monitoring is an addition to a care routine, never a substitute for one.

How to get alerted when someone changes your DNS records

Wed, 14 Oct 2026 14:40:51 +0000

TL;DR. Most DNS changes do not take your site down. A new MX record or an edited SPF record usually still resolves, so your uptime checks stay green. To catch these changes, list every record a name should have. Then get an alert when the live answer is different in any way. The original post has the same text with a short FAQ at the end.

A DNS change does not look like an outage

Uptime monitors wait for something to break. A DNS change often breaks nothing you can see.

Here is an example. Someone adds a second MX record to your domain. MX records tell other servers where to deliver your email. Maybe it was a test of a new email provider, and nobody removed it. Maybe someone stole the password to your DNS account.

Each MX record has a priority number, and senders try the lowest one first. If the new record has a lower number than yours, your email now goes to another server first. Your website works, your checks are green, and nobody notices for weeks.

Other changes work the same way:

Someone changes your nameservers at the registrar. Now a different server answers for your whole domain.
Someone edits your SPF record during a migration. SPF lists the servers that may send email as you, and your newsletter tool is no longer on it.
Someone deletes your _dmarc record while cleaning up old records.
Someone changes your CAA record. Now a different certificate authority can issue certificates for your domain.

These changes can leave your DNS lookups and website checks passing.

This happens in real attacks

In January 2019, the US Cybersecurity and Infrastructure Security Agency (CISA) published Emergency Directive 19-01. Attackers had changed DNS records on several US government domains.

They obtained credentials for a DNS account. They changed A, MX and NS records to point to their own servers. Then they could read web and email traffic as it passed through.

CISA's first order to the agencies was simple: check your public DNS records, and make sure each one points to the right place. Start with NS records.

A check like that tells you the records were right on one day. A monitor does the same check every few minutes, so you find out on the day of the change.

Why "does it resolve?" is not enough

Most DNS checks ask one of two questions.

The first is "Does the name resolve?" This catches a deleted record or an expired domain. It does not catch a changed record, because a changed record still resolves.

The second is "Does the answer contain this value?" For example, you enter your mail server, and the check fails if it disappears. This is better. But the check passes as long as your value is somewhere in the answer. So it misses:

an extra MX or NS record next to yours;
an SPF record changed around the part you check;
one A record replaced, when you have several.

To catch these, the check must know every record that should be there. Then it can fail on anything else.

Watch the exact record set

An exact record set is a list of every record that one name should have, for one record type. The check reads the live answer and compares it with your list:

A record in your list but not in the answer is missing.
A record in the answer but not in your list is unexpected.
A changed record shows up as both. The old value is missing and the new value is unexpected.

Any difference fails the check. The order of the records does not matter, and neither does the TTL.

To write the list, start from the records your domain has now. The DNS lookup tool shows them from Cloudflare and Google side by side. Check that every record is one you expect. If the two resolvers disagree, a change is still spreading, so wait until they match.

In Uptimepage, create a DNS monitor and choose exact set as the answer match. Write one record per line, the way a DNS answer shows it:

10 mail.example.com.
20 alt.example.com.

Say someone adds a third MX record. The next check fails, and the result shows what changed:

{
  "domain": "example.com",
  "record_type": "MX",
  "missing": [],
  "unexpected": ["5 mx.attacker.example."],
  "answers": ["10 mail.example.com.", "20 alt.example.com.", "5 mx.attacker.example."],
  "expected_contains": null,
  "matched": false
}

The new MX record has priority 5, lower than yours, so senders try it first.

The check ignores small differences that would only cause false alarms:

Upper and lower case in names do not matter, so 10 Mail.Example.com matches 10 mail.example.com.
The dot at the end of a name is optional.
Numbers and IP addresses are compared by value, so 2001:0db8::1 matches 2001:db8::1.

TXT values must match exactly, including case. In SPF and DKIM, every character counts.

Enter only the record value, without the name, TTL or type. For TXT records, you can paste quoted strings from your zone file, with all parts on one line.

Which records to watch

You do not need this for every record. Start with the ones where a quiet change hurts the most.

Record	Name	What it controls
NS	`example.com`	Which servers answer for your whole domain
MX	`example.com`	Where your email goes
TXT	`example.com`	Your SPF record: who may send email as you
TXT	`_dmarc.example.com`	Your DMARC policy: what receivers do with fake email from your domain
TXT	`selector._domainkey.example.com`	Your DKIM public key, which receivers use to check your email signatures
CAA	`example.com`	Which certificate authorities may issue certificates for you
A, AAAA	your server hostnames	Only if your servers have fixed IP addresses

Your main domain often has other TXT records too, such as verification records from other services. Put all of them in the list. A new one means someone connected a new service to your domain, and you want to know about that.

One monitor watches one name and one record type. Start with six monitors, one for each of the first six rows. Then add one for each other DKIM selector you use, and one for each fixed A or AAAA record set. A hostname with both an A and an AAAA record needs two monitors.

If you set up only one, make it NS. CISA put NS first too. Whoever changes your NS records controls all your other records.

Query your own nameserver

Normally a DNS lookup goes through a resolver, such as 1.1.1.1 or the one from your internet provider. The resolver keeps each answer for a while. This time is called the TTL. If your MX record has a TTL of one hour, the resolver can show the old answer for about an hour after a change. Sometimes it is longer: a resolver that cannot reach your nameserver may keep serving the old answer until it can.

You can skip this delay. Set the monitor's resolver to the IP address of your own nameserver, the one that holds your real records (your authoritative nameserver). Then every check sees your records as they are right now.

There is one exception: NS records. If someone changes your nameservers at the registrar, your own nameserver does not know. It still answers with the old, correct list. So watch NS through a public resolver, such as 1.1.1.1 or 8.8.8.8. A public resolver follows the registrar, like your visitors do. You can check what a public resolver sees for any name before you create the monitor.

When not to use an exact set

Some records change on purpose. An exact set on them only gives you false alarms.

Names behind a CDN or a load balancer get their IP addresses from a pool. The answer can change over time and depending on where you query from. I wrote about this in why DNS returns different IP addresses. An exact A set on such a name would fail all the time. Watch the CNAME record that points to the CDN instead.

SOA records change every time you edit your zone, because their serial number goes up. _acme-challenge TXT records come and go with each certificate renewal. Do not use an exact set for these either.

Keep the list next to your DNS

If you manage DNS with Terraform, put the monitor in the same repository:

resource "uptimepage_target" "mx" {
  name     = "mx example.com"
  interval = 300

  check = {
    type = "dns"
    dns = {
      domain           = "example.com"
      record_type      = "MX"
      expected_records = ["10 mail.example.com.", "20 alt.example.com."]
    }
  }
}

Now a pull request that changes an MX record must change the monitor too. An MX change that nobody reviewed fails the check. You need Terraform provider 0.15.0 or later. Read more in monitoring as code. All DNS options are in the DNS monitor docs.

Where Uptimepage fits

I build Uptimepage, an uptime monitor that checks your services from several regions and alerts you when one fails. The exact set match above is part of its DNS monitor. A DNS change alerts you once the failure is confirmed: by default two failed checks in a row, from more than half of the regions. The result names the missing and the unexpected records, so you see the change without running dig yourself. It is open source under AGPL, so it can also run on your own server.

Whatever you use, look up your NS records today and check that every name in the answer is one you expect.

Now the question for the comments: has a DNS record on one of your domains ever changed without anyone noticing? I would like to hear how you found out, and how long it took.

Multi-tenant PageSpeed monitoring under $100: what agencies actually buy

Sat, 10 Oct 2026 07:37:47 +0000

The purchase is rarely another free checker. It is scheduled portfolio coverage, shared history, and flat organisation caps that survive twenty client sites.

An agency can open ChatGPT, paste a PageSpeed Insights URL, and get a plausible fix list before lunch. That loop feels like monitoring until the account manager asks for last month’s LCP trend across twenty WordPress clients and nobody has a shared schedule, a named alert owner, or a history that survives holiday leave. Multi-tenant PageSpeed monitoring under $100 is not a cheaper way to run one more free check; it is the purchase of portfolio coverage that still fits a three-digit ops line.

What agencies actually buy in that band is flatter than the vendor shortlist suggests. They buy pooled site and test caps under one organisation subscription, workspaces that keep client brands separate, and reporting an account team can open without pinging the engineer who wrote the cron. The pricing tables and tier caps live on our blog in Affordable Multi-Tenant Performance Monitoring Under $100/Month. The day-to-day shape of one dashboard across many client sites is in Multiple Client Sites, One Dashboard. What follows here is the buying story: what lands on the purchase order when the brief says “keep Core Web Vitals coverage under a hundred dollars.”

What agencies mean by multi-tenant PageSpeed monitoring under $100

In agency language, multi-tenant means one customer account that holds many client organisations, sites, and schedules without minting a new invoice line per domain. Under $100 means the list price for that organisation-shaped monitor fits a three-digit monthly ceiling before you add flagship real-user monitoring (RUM) seats. The job is scheduled lab coverage on money URLs, CrUX context where PageSpeed Insights publishes it, budgets and alerts with named responders, and history long enough for a quarterly business review.

That purchase is easy to confuse with adjacent tools. A free PageSpeed Insights run is a diagnostic, an AI chat that proposes CSS changes is a fix assistant, and an uptime product with a small page-speed monitor allowance is infrastructure health with speed as a sidecar. Agencies that say they want multi-tenant PageSpeed monitoring under $100 usually want the portfolio job, not another one-off score.

If the roster is still under roughly ten priority URLs and one engineer already owns automation, DIY can still hold for a while. Once account managers need shared history and the URL list grows every campaign week, the under-$100 question becomes whether the maths is flat at the organisation or multiplies per monitored slot. That is the fork most agencies hit before they ever open a pricing page.

AI fix loops are not the same purchase as a portfolio schedule

The confusion shows up often on social threads: someone ships an agent that reads Lighthouse findings and opens a pull request, then treats that as “we have monitoring.” Fix loops are useful on sites you can change, yet they are not a substitute for knowing which client homepage went amber on mobile three Tuesdays ago while the developer who owns the agent was offline. The first job remediates; the second proves the roster stayed inside budget when nobody was watching the chat.

A scheduled portfolio monitor answers a different question. It keeps URL inventories current, runs PageSpeed Insights (or equivalent) on a cadence the retainer can defend, stores history the account team can export, and routes budget breaches to someone other than the person who wrote the workflow. The AI loop still belongs in the stack as a remediation layer on sites you deploy, yet it does not replace the system of record for twenty client origins. Buying under-$100 multi-tenant coverage while also keeping an AI fix assistant is a sensible layering story, but buying only the assistant and calling the retainer “monitored” is how QBR slides become screenshots again. The SOW reads cleaner when both jobs are named, so the account team knows which tool answers which question.

What actually lands on the purchase order

When the buy is honest, the purchase order or statement of work names jobs, not logos. Logos without owners still leave the retainer on hero labour. In our experience the under-$100 multi-tenant buy usually includes:

a flat organisation subscription with published site and monthly test caps,
enough workspaces to separate client brands without a second vendor,
scheduled lab runs on agreed money URLs (not “every URL we ever shipped”),
shared history and exports the account team can open,
budgets and alerts with named owners,
a clear non-goal: this is not deep RUM on every brochure domain.

What usually does not land on that same line item: unlimited regions and devices per URL, enterprise RUM seats, or a promise to monitor forty properties when the published site cap is twenty-five. Those gaps belong in a later tier or a thinner URL list, not in silent overages. Saying the non-goals out loud is cheaper than explaining a surprise invoice mid-engagement.

Procurement still needs plain wording for the row. That is a different essay from the purchase shape itself; for translating milliseconds into finance language, see our Hashnode notes on procurement budgets and selling Core Web Vitals monitoring without calling the line item “latency.” Here the decision is simpler: either you are buying organisation-scoped scheduled coverage under a three-digit ceiling, or you are still assembling DIY layers and should say so in the SOW. Ambiguous wording is how “monitoring” gets approved as a free checker and then fails the first multi-client review.

Flat organisation caps versus per-site maths

The under-$100 question collapses to how capacity is counted, and the short table below is the decision frame agencies actually use when three vendor demos arrive in the same week.

Pricing shape	How capacity is counted	Fits under $100 when…	Breaks when…
Per monitored URL / slot	Each URL × device × region consumes a slot	One client, deep lab work	Twenty clients each need homepage + checkout
Per-site SaaS seat	Billing scales with domains	Portfolio stays tiny	Retainer count grows faster than tool budget
Flat organisation cap	Sites, tests, workspaces pool under one subscription	Roster grows inside published limits	You exceed site or test caps and need the next tier
DIY scripts + API	Cash cost is low; labour is not	One engineer owns the pipeline	Quota, reporting, and on-call become unpaid platform work

Agencies that stay under $100 with multi-tenant coverage almost always pick the flat organisation row and accept published caps. Per-slot tools can still win for a single high-traffic property; they are a poor match for “monitor the whole WordPress roster for under a hundred.” DIY wins on cash until the labour row on the same budget is honest. If you cannot name that labour owner, you are not really choosing DIY; you are deferring the purchase. Apogee Watcher’s Professional tier is built as that flat-cap shape under the $100 line at list pricing, with Agency as the published step when site or workspace caps bite. Current numbers live on the pricing page, and the buyer story does not change when a list price moves by a few dollars: you are still choosing organisation caps versus per-site multiplication.

When DIY still wins the budget

DIY PageSpeed Insights API scripts, Lighthouse CI, sheets, and a workflow tool still win the purchase when three conditions hold together: the portfolio is small enough that one person can keep URL lists accurate, nobody expects branded client PDFs from the monitoring layer itself, and the team already treats automation ownership as billable engineering time rather than free evenings.

DIY stops winning the under-$100 comparison when any of these repeat: campaign URLs ship and never join the sheet; the digest dies on leave; account managers rebuild slides from CSV; “nightly for everyone” became “nightly for whoever we remembered” under API quota. At that point a flat-cap portfolio monitor is usually cheaper than the labour you were pretending was free, even when the SaaS line sits at $79 rather than $0. The cash line looks worse; the retainer maths usually looks better. DIY still fits as a layer after the portfolio monitor is in place (Lighthouse CI on templates you ship, manual PageSpeed Insights in teaching conversations), because the purchase under $100 is for the jobs DIY stopped covering reliably, not a mandate to delete every script.

FAQ

Can an agency monitor forty client sites under $100/month?

Often not on a single flat-cap Professional-shaped plan if the published site limit is twenty-five. Realistic under-$100 strategies put priority origins on the paid monitor and leave long-tail pages on free diagnostics, or move to the next published tier when the roster honestly exceeds the cap. Pretending forty equals twenty-five is how retainers over-promise.

Is an AI agent that fixes Lighthouse issues the same as multi-tenant monitoring?

No: an agent remediates findings on sites you can change, while multi-tenant monitoring schedules checks, stores history, and alerts across many client organisations whether or not you control the repository. Most agencies need both jobs named separately rather than collapsing them into one “AI monitoring” line.

What should we put on the purchase order so finance understands the buy?

Organisation-scoped scheduled PageSpeed / Core Web Vitals coverage, included site and test caps, retention, seats, and reporting are the rows that usually clear finance. A three-digit portfolio monitor is a different purchase from a four-hundred-dollar RUM seat on one flagship property. The SOW reads better when it states what is out of scope as well as what is included.

Does multi-tenant PageSpeed monitoring replace uptime tools?

No: uptime and SSL watches answer “is the site reachable,” while portfolio PageSpeed monitoring answers “are LCP, INP, and CLS drifting on money URLs.” Many agencies keep both, because green uptime does not mean green Core Web Vitals, and a green Lighthouse score does not mean the origin stayed up overnight.

Buy the portfolio job, then layer the rest

What agencies actually buy under $100 for multi-tenant PageSpeed monitoring is organisation-shaped coverage: flat caps, shared schedules, and history the account team can defend in a review. AI fix loops, free checkers, CI gates, and selective RUM stay in the stack as layers, yet they are not the same line item. The pricing maths and tier caps sit in Affordable Multi-Tenant Performance Monitoring Under $100/Month. For the product shape of many clients in one view, Multiple Client Sites, One Dashboard is the shorter walkthrough, and you can run a check on a client URL before schedules move into a trial.

References

PageSpeed Insights (Google)
PageSpeed Insights API (Google)
Apogee Watcher pricing (Apogee Watcher)
Affordable Multi-Tenant Performance Monitoring Under $100/Month (Apogee Watcher)
Multiple Client Sites, One Dashboard (Apogee Watcher)
3 Agency PageSpeed Stacks for 2026 (Apogee Watcher)

Originally published on Hashnode.

What PageSpeed monitoring stack should an agency run in 2026?

Thu, 08 Oct 2026 06:55:49 +0000

Most agencies do not need three products on day one. Pick a default PageSpeed monitoring stack, then add RUM only where money URLs justify it.

The roster hits twenty client sites, yet the “monitoring stack” is still a shared folder of PageSpeed Insights screenshots, one engineer’s n8n flow nobody else can edit, and a promise to check Core Web Vitals before the QBR. That is not a PageSpeed monitoring stack for agencies; it is a coping habit with a product roadmap glued on. The gap shows up the first time two account managers need last month’s LCP trend and only one person knows which sheet tab is current.

What agencies should run in 2026 is usually simpler than the vendor shortlist suggests. A multi-site portfolio monitor works best as the system of record for lab history, budgets, and alert ownership, while DIY PageSpeed Insights scripts and Lighthouse CI stay as layers on sites you deploy. Paid real-user monitoring (RUM) belongs on the one or two flagship properties where field depth pays for itself; everything else is optional seasoning.

The full recipes and failure modes live on our blog in 3 Agency PageSpeed Stacks for 2026, and the feature checklists sit in Comparing PageSpeed Monitoring Tools. What follows here is the opinionated pick: which stack to run first when the account team asks for a decision, not another comparison table.

What a PageSpeed monitoring stack is (for agencies)

A stack is a small set of jobs with named owners, not a shopping cart of logos. For agency work those jobs are predictable:

scheduled lab coverage on money URLs (PageSpeed Insights or equivalent),
history an account manager can open without asking engineering,
budgets and alerts that fire for someone other than the person who wrote the cron,
optional CI gates on templates you ship,
optional RUM depth where a snippet and seat cost are justified,
a clear split between uptime/health tools and Core Web Vitals work (they are not the same purchase).

If any of those jobs live only in one person’s head, you do not have a stack yet. You have a hero, and heroes do not scale across twenty organisations. Named owners in the tool turn the same checklist into something the account team can run without a private Slack ping.

The default stack most agencies should run in 2026

Default: portfolio monitor first for multi-client PageSpeed / Core Web Vitals coverage, DIY and CI as supporting layers, RUM as a selective add-on.

That default fits when you manage roughly ten or more client sites, sell performance or SEO retainers that mention LCP/INP/CLS, and cannot put a RUM snippet on every brochure domain. The monitor owns schedules, URL inventories, budgets, and shared history; Lighthouse CI stays in the repositories you control; PageSpeed Insights remains the lab dialect clients already recognise in Search Console conversations. Once those three layers are named, the weekly argument stops being “which vendor is best” and becomes “which job is failing.”

Apogee Watcher is built for that portfolio shape, and peers exist in the same job class. The recommendation is the shape, not a claim that only one vendor can fill it, so layer rather than rip out: keep DebugBear, SpeedCurve, or Calibre where you already pay for flagship RUM, and skip pretending every client needs that seat maths on day one. The unpaid work is deciding which sites never get a snippet, not collecting logos.

Buying “the best RUM” and rolling it out everywhere sounds tidy until seat cost, privacy reviews, and install friction kill portfolio coverage before the first amber LCP is fixed. Staying on DIY forever sounds frugal until URL lists drift, quotas collide, and reporting hours eat the retainer. The default sits between those failure modes on purpose.

When DIY PageSpeed Insights + sheets is enough

DIY (PageSpeed Insights API or self-hosted Lighthouse, Lighthouse CI, Sheets or Notion, n8n or cron) is enough when the portfolio is small, one engineer already owns automation, and nobody expects branded client PDFs from the monitoring layer. It also fits when you only need merge gates on sites you deploy, with occasional API checks elsewhere. Outside that band, DIY becomes unpaid platform work dressed up as thrift.

DIY stops being a sound primary system when any of these show up more than once a month:

campaign URLs ship and nobody added them to the sheet,
the digest email dies because the workflow owner is on leave,
account managers rebuild slides from CSV because clients will not open the spreadsheet,
“nightly for everyone” became “nightly for whoever we remembered” under API quota.

When those signals repeat, a portfolio monitor becomes the primary system and DIY stays as a layer. That is DIY versus managed PageSpeed without the false choice of deleting your pipelines: managed here means shared schedules and history, not abandoning Lighthouse CI on the templates you still ship. The roster can grow without the monitoring story living in one export again.

When to add RUM on flagship sites only

RUM earns its place when a single property funds deep field investigation: high revenue, stubborn INP, or a client who already bought SpeedCurve, DebugBear, or Calibre and needs you to operate it. The portfolio monitor still covers the long tail so brochure sites keep scheduled lab history, because paying for flagship depth while the rest of the roster goes dark is how retainers look busy and still miss regressions. Selective RUM is a budget decision, not a purity test.

Waiting for perfect RUM coverage before any monitoring starts is the other trap. Multi-site PageSpeed monitoring fails when teams refuse to schedule anything until every domain has a snippet, while lab schedules plus CrUX context from PageSpeed Insights or Search Console (where eligible) already answer most retainer questions. Field depth is a zoom lens, not the only camera.

Jobs the stack must cover (without a vendor matrix)

Treat the table below as a go-live checklist: if a row has no owner and no tool, the stack is incomplete, so fill those gaps before another logo appears on the slide.

Job	Default owner in 2026
Scheduled lab on money URLs	Portfolio monitor (or DIY if still under ~10 priority URLs)
Merge protection on shipped templates	Lighthouse CI in your repos
Shared history for account teams	Portfolio monitor (not a private n8n)
Budgets / alerts with named responders	Portfolio monitor
Deep field waterfalls	RUM on 1–2 flagships only
Reachability / SSL / status pages	Separate uptime suite (layer; do not confuse with CWV)

When someone asks for “the best tools,” the tools comparison is the right hand-off. When they ask “what should we run next quarter?,” the default above plus the three recipes on agency PageSpeed stacks are the operational answer. The first question is research; the second is delivery.

FAQ

What PageSpeed monitoring stack should an agency run in 2026?

For most multi-client agencies, a portfolio PageSpeed / Core Web Vitals monitor is the system of record, DIY scripts and Lighthouse CI stay as layers, and RUM sits only on flagship sites that justify the seat. DIY-only still fits if the roster is tiny and automation is already owned, so write that choice into the retainer scope and sales and delivery argue from the same default.

Is PageSpeed Insights alone enough?

As a diagnostic, yes. As the only continuous system for fifteen retainers, no: you still need schedules, history, budgets, and someone other than the founder to receive the alert.

Do we need RUM on every client site?

Usually not. RUM belongs where revenue or stubborn field issues pay for it, while the long tail stays on scheduled lab monitoring and CrUX where Google publishes it.

How is this different from a tools roundup?

A roundup lists features; a stack names jobs and a default order of purchase. Use both: pick the shape here, then validate vendors against a feature checklist.

CTA

If the current “stack” is still screenshots and a private automation, this quarter is a good moment to make a portfolio monitor primary and write DIY exit criteria into the runbook. A free Apogee Watcher trial is useful when you want multi-tenant schedules and budgets without rebuilding the sheet again, while RUM and uptime tools stay where they already earn their keep.

References

3 Agency PageSpeed Stacks for 2026 (DIY, Portfolio Monitor, RUM + Monitor) (Apogee Watcher)
Comparing PageSpeed Monitoring Tools: Features Agencies Need (Apogee Watcher)
PageSpeed Insights API (Google)
Lighthouse CI (GoogleChrome)
When to Use Synthetic vs Real User Monitoring for Performance (Apogee Watcher)

Originally published on Hashnode.

Google says on-call needs eight engineers. You have three.

Tue, 06 Oct 2026 14:27:45 +0000

TL;DR. Google's SRE book works out that a 24/7 on-call rotation needs at least eight engineers. A team of two or three cannot reach that number, so it has to change what the number is built from: which alerts page a person, which hours are covered, and how fast someone must answer. After that it needs one named person on call at any moment and a short escalation ladder behind them for the page nobody answers. A weekly count of pages shows whether it is working. The original post has the same text with a short FAQ at the end.

"We are not Google"

Google's first SRE book has a chapter called Being On-Call. Readers wrote back. The follow-up book, the SRE Workbook, opens its own on-call chapter with their feedback, and the first item on the list is this:

"We are not Google; we're much smaller. We don't have as many people in the rotation, and we don't have sites in different time zones. What you described in your first book is irrelevant to me."

I build Uptimepage, which pages people when a monitor goes down, so I had to decide what a rotation should look like for a team that small. That reader is right about the numbers. The reasoning behind the numbers still works for three people, and it leads to a different setup.

Where the eight comes from

The book starts from a limit on time:

"we strive to invest at least 50% of SRE time into engineering: of the remainder, no more than 25% can be spent on-call"

Then it does the arithmetic:

"Assuming that there are always two people on-call (primary and secondary, with different duties), the minimum number of engineers needed for on-call duty from a single-site team is eight: assuming week-long shifts, each engineer is on-call (primary or secondary) for one week every month."

Two people on call, each for at most a quarter of the time, makes eight. Run the same arithmetic on a small team, with one person on call at a time and week-long shifts:

Team size	Share of weeks each person is on call
2	50%
3	33%
4	25%
8, with two on call at a time	25%

A team of four reaches Google's limit only by giving up the secondary. A team of three is over the limit before anything breaks, and a team of two is on call every other week.

A schedule cannot fix that, because any rotation of three people still has three people in it. What the team can change is how often the pager goes off, during which hours, and how quickly it has to be answered.

The limit exists because pages are expensive. The same chapter estimates that one incident, with its diagnosis, fix and follow-up, takes about six hours of work, which is why it allows at most two incidents in a 12-hour shift. A person on call every third week can live with that week if it is quiet. So the work for a small team is to make the week quiet, then make sure the rare page reaches someone.

What a team of three can change

Decide what is worth a page

The Workbook is direct about this:

"Engineers shouldn't have to be at a computer and working on a problem within minutes of receiving a page unless there is a very good reason to do so."

Its Table 8-1 gives three examples of a realistic response:

Incident	Response time
Revenue-impacting network outage	5 minutes
Customer order batch processing system stuck	30 minutes
Backups of a database for a pre-launch service are failing	Ticket (response during work hours)

Sort every alert you have into those three rows. Only the first two should reach a phone at night. The rest go to a channel or an inbox that somebody reads in the morning.

An alert has to pass two more filters before it may page. It has to be real: one probe with a bad network path should not wake anyone. And it has to come with something to do. The SRE book's wording is "All paging alerts should also be actionable." A page that can only be watched until it clears teaches people to ignore pages.

Name one person at a time

The usual start for a small team is a shared channel: every alert goes to #alerts, and everyone is in it. During the day this works, because somebody is looking. At 03:00 five people receive the message, and each of them can reasonably think one of the other four has it.

A rotation puts exactly one name on call at any moment. That person knows it is them, and everyone else knows it is not. The channel can still get a copy, but the page goes to the person.

Hand off on a weekday morning, when both people are awake and can talk about what is still open. A handoff at midnight on Sunday means nobody mentions the disk that is filling up. Keep it at the same local time all year too, so the change does not move by an hour when the clocks do.

Split the day before you split the week

The Workbook does not expect one person to carry a full week around the clock:

"In our experience, 24 hours of on-call duty without reprieve isn't a sustainable setup."

Its suggestion fits a small team too: "it would be better for two engineers to split a week of on-call, with one person on-call during the day and one on-call overnight."

With three people that can be two rotations in one schedule. Working hours rotate weekly among all three. Nights and weekends rotate separately, with the order shifted by one, so nobody carries both in the same week.

There is also a cheaper option: leave some hours uncovered on purpose. If your customers are businesses in one region and nothing you run loses data while it is down, a page at 02:00 may only move the fix from 08:00 to 02:30. Decide that as a team, then write it where customers can see it, in your SLA and on your status page. As a customer I would rather read a stated response window during European working hours than a 24/7 promise kept by one tired person.

Do the arithmetic before you choose. A 99.9% target leaves 43 minutes a month. One night outage that waits six hours for the morning uses more than eight months of that budget. If the number you promised cannot absorb that, someone has to be on call at night.

Add a ladder for the unanswered page

One name on call is one point of failure. Phones run out of battery, and a person who was paged twice last night can sleep through the third.

An escalation policy is an ordered list of who gets paged next when a page goes unanswered. Level one pages when the incident opens, and each level waits a set time for an acknowledgement before the next one pages. Acknowledging stops the ladder.

For three people a ladder can be this short:

The person on call. Wait 10 minutes.
The other two. Wait 15 minutes.
Everyone, on the loudest channel each person has.

How long to wait comes from the response time you promised. The SRE book's typical values are "5 minutes for user-facing or otherwise highly time-critical services, and 30 minutes for less time-sensitive systems". If level one has 5 minutes to answer, the wait before level two cannot be 30.

This ladder is the small team's version of Google's secondary. Nobody is formally second on call, and the page still has somewhere to go. Watch how often level two fires. If it fires most weeks, the other two people are on call as well, whatever the schedule says.

Make swaps cheap

The Workbook again:

"No one can promise on Monday that they won't have the flu on Thursday."

On a team of eight a swap is easy to arrange. On a team of three every swap lands on one of two people, so it has to take a minute and leave no doubt about who holds the pager afterwards. That means an override on the schedule itself, for exact dates, which ends without anyone remembering to undo it. A chat message asking a colleague to cover Thursday changes nothing about where the page goes.

Put the shifts in everyone's calendar as well. A person who sees an on-call week next to a dentist appointment three weeks ahead asks for the swap now instead of on the day.

Test that the page arrives

A schedule can name a person the page cannot reach. Their phone is on do-not-disturb, the chat app is muted at night, or the email address was never confirmed. Everything looks covered until the first night page arrives on a silent phone.

So test it. On the first morning of a shift, send a test page to the person on call and have them acknowledge it from the phone they will have at night, with the settings they will have at night. Use at least one channel that is allowed to make noise through do-not-disturb, such as SMS from a number saved as an emergency contact, or a push app with an emergency priority.

GitLab's 2017 outage shows what an untested path costs. The failure alerts for its backup job were sent every night and rejected by the receiving mail server, and nobody knew for months.

Count the pages

Google's teams work to a number: "We target a maximum of two incidents per on-call shift". The Workbook also describes a team with a budget of two paging incidents per shift that had been receiving five for a year. Their answer was a project to reduce the pages, approved by senior management.

A small team needs that count more than a large one does, because each page lands on a third of the team. Once a week, look at how many pages there were, how many of them needed a person, and how long the first acknowledgement took. A page that needed nobody is an alert to fix or delete. A slow acknowledgement is a reason to look at the ladder and the paging channel before it is a reason to look at the person.

What this does not fix

Two people are still on call every other week, and no ladder changes that. What is left is to cover fewer hours or fewer systems and tell customers which, and to pay for the nights. The Workbook's position is short: "On-call usually implies some amount of out-of-hours work. We believe this work should be compensated."

Four is the first team size where one person on call at a time meets the 25% line. If on-call is wearing down a team of three, that is a real argument for the fourth hire, and the table above is how to make it.

Sources

Google, Site Reliability Engineering, chapter 11, Being On-Call.
Google, The Site Reliability Workbook, chapter 8, On-Call.
GitLab, Postmortem of database outage of January 31, 10 February 2017.

Where Uptimepage fits

I build Uptimepage. It checks your services from several regions, and when one goes down it pages whoever is on call.

An on-call schedule holds one or more rotations, each on call at all hours or only in the hours you give it. Daily and weekly handoffs stay at the same local time across daylight saving. Overrides cover exact dates. Each person picks the channels that page them, and the schedule list names anyone on a rotation whom no channel can reach. An escalation policy is a ladder of levels with a wait on each, and acknowledging from the chat or email alert stops it. Each person gets a calendar link for their shifts, and the incident reports include mean time to acknowledge.

One gap to know about: there is no phone-call channel yet. SMS, Pushover's emergency priority, Telegram and the chat apps are there, and if you already use PagerDuty you can put it on a level. The incident docs have the details. It is open source under AGPL, so it can also run on your own server.

Whatever you use, write down the answer to one question. At 03:00 tonight, whose phone rings, and whose rings next if they do not answer?

Now the question for the comments: how does a team of two or three cover nights where you work? I would like to hear from teams that leave some hours uncovered on purpose and say so on their status page.

Oh Dear alternative? Uptime green is not Core Web Vitals portfolio monitoring

Tue, 06 Oct 2026 06:58:33 +0000

Searching for an Oh Dear alternative usually lands you on more uptime tools. Green reachability is not a Core Web Vitals portfolio.

The uptime dashboard is green, SSL is fine, and the status page is quiet, yet a client still forwards a PageSpeed Insights screenshot where mobile Largest Contentful Paint sits in Needs improvement. Nobody can explain how both truths coexist. Searching for an Oh Dear alternative feels like the next step, but that query usually returns more uptime and white-label status products, not a Core Web Vitals portfolio monitor.

The gap is not that Oh Dear is weak at its job; it is treating reachability as proof that shoppers experience a fast page. Green uptime and healthy Core Web Vitals are different clocks, so conflating them is how retainers buy the wrong primary tool. The search phrase “Oh Dear alternative” then steers buyers back onto the uptime shelf when the screenshot was about experience.

What “Oh Dear alternative” usually means in search

Oh Dear sits in a crowded agency niche: uptime checks, SSL and DNS health, broken-link crawls, status pages, and often a scheduled Lighthouse or response-time layer under one login. When agencies outgrow a free ping tool, “Oh Dear alternative” is a natural buying phrase. The search box does not know whether you meant “better status pages” or “finally watch LCP.”

Open that SERP today and you mostly see other agency uptime suites: white-label status pages, client PDF uptime reports, multi-region pings, SSL and domain expiry. Those products compete on the uptime axis, and they are sensible shortlists when the failure mode is “the site returned an error” or “the certificate expired on Friday night.” That shelf is crowded for a reason: downtime still burns retainers.

They are a weak answer when the failure mode is “HTTP 200, but LCP and Interaction to Next Paint are amber across the money path.” That job needs scheduled lab history, CrUX field context where Google publishes it, URL inventories that keep up with CMS churn, and budgets written in Core Web Vitals units. Uptime clones do not become that stack because the brochure says “performance,” so the buying phrase and the failure mode have to match before you shortlist.

Uptime green is not a Core Web Vitals signal

Uptime answers whether this origin responded successfully from the checker’s vantage point, while Core Web Vitals answer how real Chrome users (field) or a throttled lab browser (Lighthouse / PageSpeed Insights) experienced load and interactivity on specific URLs. Both can be “true” on the same Tuesday and still describe different risks. That mismatch is what the green dashboard hides.

A site can stay up while:

the LCP image ships late on mobile category templates,
a new extension blocks the main thread on add-to-cart (INP),
a promo bar shifts layout without reserved space (CLS),
third-party tags grow for a campaign week that never shows as downtime.

None of those failures flip a classic uptime monitor red, but they do show up in PageSpeed Insights lab runs, in CrUX when the URL is eligible, and in client SEO conversations that already use Google’s Good / Needs improvement / Poor bands. Calling the origin “healthy” because the ping is green trains account teams to ignore the metric set the retainer actually sells. The screenshot arrives anyway; the dashboard just never warned anyone.

For the wider lab-versus-field map, see When to Use Synthetic vs Real User Monitoring for Performance. Synthetic uptime and synthetic Lighthouse are both lab-shaped, but they measure different artefacts, so keep the labels honest in the pack. Readers who skip that distinction spend the next quarter arguing about which green is “real.”

What Oh Dear-class suites cover (and what they leave out)

Oh Dear-class products are genuinely strong when you need one place for “is it up?”, certificate risk, and a client-facing status story. Many also run HTTP timing on the same fetch path, or a daily Lighthouse score on selected URLs, and those extras matter. They still sit beside uptime economics: pricing and product roadmaps usually scale with sites and check frequency, not with multi-URL Core Web Vitals portfolios and discovery.

What they typically leave as a separate purchase:

continuous PageSpeed Insights history across many client organisations,
CrUX field bands stored next to lab runs for the same URL set,
automated discovery when a CMS adds templates you never pasted into the monitor list,
budgets and alerts written primarily against LCP, INP, and CLS rather than millisecond TTFB on the uptime probe.

Daily Lighthouse is closer to “pagespeed” than a ping, but it is still not the same as a portfolio programme built around Google’s PageSpeed Insights API and field context. Mixing “we have Lighthouse in the uptime tool” with “we monitor Core Web Vitals for SEO” without labelling the difference is how false confidence survives a QBR. One score on one URL is a check; a retainer needs a series.

A fair pairwise feature split (Oh Dear versus a CWV-first portfolio tool) lives in Oh Dear vs Apogee Watcher: Uptime Suite vs Core Web Vitals Portfolio Monitoring. Use that matrix when you are already comparing vendors; what follows here is the job split before you open the spreadsheet. The spreadsheet is useless if you still think both columns measure the same risk.

When to keep the uptime suite and add a CWV layer

Keep (or replace within) the uptime axis when the pager must fire because HTTPS is down, certificates expire, or clients need a branded status page. An Oh Dear alternative that stays on that axis is a valid purchase.

Add a Core Web Vitals portfolio layer when you sell performance retainers, report LCP/INP/CLS in client decks, or manage many origins where the expensive failure is slow-but-up. Layer rather than rip out. Agencies that already run Oh Dear (or Pingdom, Better Stack, UptimeRobot, and similar) usually need a second product for CWV depth, not a single suite that pretends both jobs are free.

Replace Oh Dear only when you never needed the uptime bundle and the trial proved you were paying uptime economics for a CWV-only brief. That is a narrower case than the search phrase suggests.

A practical split for retainers

Job	Prefer
Reachability, SSL, status pages	Oh Dear or another uptime / health suite
Cold-load lab + opportunity text	PageSpeed Insights lab or scheduled PSI-style monitoring
SEO-facing field categories	CrUX via Search Console / PageSpeed Insights field
Many client sites, CWV budgets, discovery	Multi-tenant PageSpeed / CWV portfolio monitor

Write both truths in the pack with labels: “Uptime (reachability)” versus “Lab PageSpeed / CrUX (experience).” Green on the first column does not clear the second, and clients notice when the PDF pretends otherwise.

FAQ

Is searching for an Oh Dear alternative the right move for Core Web Vitals?

Only if you also need a better uptime suite. If the pain is amber LCP on money pages while pings stay green, shortlist a CWV portfolio monitor instead of another status-page vendor. The search phrase and the failure mode have to match.

Does Oh Dear’s Lighthouse check cover Core Web Vitals monitoring?

It covers a scheduled lab score on the URLs you configure, but it does not automatically equal a multi-client PageSpeed Insights programme with field context and discovery. Treat it as a useful signal on the uptime product, not as the whole CWV retainer.

Should we rip out Oh Dear and switch to a PageSpeed tool?

Usually no. Keep uptime where downtime is the risk, and add CWV monitoring where experience is the risk. Rip-and-replace only when the uptime bundle was never the job.

Why do uptime tools say “performance” then?

Because HTTP timing and Lighthouse scores are performance-adjacent. The word still collapses three jobs (reachability timing, lab audits, field Core Web Vitals) into one brochure line. Ask which artefact the alert fires on before you trust the label.

What to do next

A useful starting pass is to list every production origin on the retainer, mark which ones already have uptime coverage you trust, and note last lab LCP/INP/CLS (plus whether CrUX exists) for each money-path URL. If uptime is green and lab or field is amber, you do not need another status page first. You need a scheduled CWV layer and a report that names both clocks.

For the pairwise Oh Dear versus portfolio decision, use Oh Dear vs Apogee Watcher. For how synthetic lab fits beside real-user field data, keep Synthetic vs Real User Monitoring in the same pack. Two links, two jobs, one retainer language.

References

Oh Dear (product)
Oh Dear vs Apogee Watcher: Uptime Suite vs Core Web Vitals Portfolio Monitoring (Apogee Watcher)
When to Use Synthetic vs Real User Monitoring for Performance (Apogee Watcher)
Understanding Core Web Vitals and Google search results (Google Search Central)

Originally published on Hashnode.

SpeedCurve vs Apogee Watcher: Multi-Tenant Dashboards at Agency Price

Tue, 15 Sep 2026 07:04:00 +0000

SpeedCurve (now part of Embrace) is the tool many teams mean when they ask for a serious SpeedCurve alternative that still covers synthetic tests, real-user monitoring, and performance budgets. Public SpeedCurve pricing starts at $90/month on Starter, with Growth from $576/month, scaled by RUM page views and synthetic checks. That package is strong when one product organisation owns a small set of high-traffic origins and wants lab data, RUM, and coaching culture in one place.

Apogee Watcher answers a different question: how do you keep scheduled PageSpeed coverage honest across many client sites when you cannot put a RUM snippet on every domain, and when per-check maths at SpeedCurve Growth rates would erase retainer margin? We run Google’s PageSpeed Insights API (Lighthouse lab plus CrUX where Google publishes field data) inside multi-tenant organisations with Admin, Manager, and Viewer roles, plus automated page discovery. We do not ship first-party RUM. Field context comes from CrUX in each result, not from our own JavaScript on the client’s origin.

The comparison below is about workflow fit for agencies, not about declaring a universal winner. Limits and prices change on both sides, so confirm each vendor’s site before you buy and treat any table as a snapshot rather than a contract. What follows stays useful only while those public pages match what sales quotes.

What SpeedCurve is genuinely good at

SpeedCurve’s public positioning centres on three jobs: track a wide set of performance metrics, turn those metrics into fixes, and coach a performance culture across product, SEO, and marketing. Synthetic monitoring and Real User Monitoring sit on the same plan family. Marketing materials emphasise Core Web Vitals, Lighthouse-oriented lab work, user timing, competitor views, and correlating speed with business outcomes such as conversion and bounce.

For a product team with engineering ownership of a flagship site, that combination is compelling. You can set performance budgets, schedule synthetic checks, sample RUM so cost stays predictable, and escalate when real users degrade after a release. Growth and Enterprise tiers add longer retention, priority synthetic testing, private agents, and SSO according to SpeedCurve’s pricing page. Teams that already treat web performance as a cross-functional programme often praise the depth and the human support model.

None of that is theatre. If your brief is “instrument our storefront, prove ROI to leadership, and keep RUM on the money pages,” SpeedCurve is a serious shortlist candidate. DebugBear occupies a similar premium lane; we compared that product separately in DebugBear vs Apogee Watcher.

Where agencies feel friction on SpeedCurve pricing and multi-site work

Agency retainers rarely look like a single product org with one design system and one consent banner. They look like twenty WordPress and Shopify origins, uneven tag-manager access, and account managers who need read-only dashboards without burning synthetic quotas. That shape changes which line items on a SpeedCurve quote still make sense once you leave a flagship brand.

Friction shows up in three places.

Cost at portfolio scale. Starter at $90/mo already sits above our Agency entry on Apogee Watcher pricing for many small shops, and Growth at $576/mo is a different budget class. Synthetic checks are counted by URL, browser, region, and frequency. A planner that monitors many URLs across many client sites multiplies quickly. RUM is billed on page views (with sampling), which fits owned traffic better than a long tail of low-traffic client microsites.

Instrumentation politics. RUM needs a script on the origin. Consent, security review, and “we will not add another tag” are common blockers on client sites you do not fully control. Synthetic-only coverage still matters when you only have public URLs.

Multi-tenant operations. SpeedCurve documents unlimited users and teams on published plans, which helps collaboration inside one company. Agencies also need clear organisation boundaries per client, role separation that protects quotas, and URL inventory that keeps up with CMS churn. Stale monitors look green while the live template set has moved on. That inventory problem is why we invested in automated discovery rather than assuming someone pastes every new route.

None of this says SpeedCurve is the wrong product. It says the job “premium RUM + synthetic for our brand” and the job “scheduled PSI coverage across a client portfolio” are different purchase decisions. Mixing those jobs on one invoice is where retainer margin usually disappears.

What Apogee Watcher optimises for

We optimise for breadth-first synthetic monitoring that agencies can run without per-client Google API keys or per-client RUM snippets:

PageSpeed Insights API results stored over time (lab Lighthouse metrics plus CrUX when Google returns field data for the URL).
Multi-organisation, multi-site structure with Admin, Manager, and Viewer roles.
Automated page discovery from sitemaps and HTML crawl (see how we discover pages).
Performance budgets and email alerts on scheduled runs (Slack and webhook delivery are on the roadmap; confirm live channels on features).
Published tiers from Personal through Agency on pricing, designed so cost scales with organisation and test quotas rather than a surprise per-domain licence for every microsite.

Gaps we state plainly: no first-party RUM, no SpeedCurve-style private agents or conversion correlation suite in-product, and automated white-label client report generation is still shipping (do not treat it as live until features says otherwise). When you need session-level INP attribution on an instrumented app, keep SpeedCurve, DebugBear, or another RUM vendor on that origin. Claiming otherwise would waste a trial week for both sides.

For how synthetic schedules sit beside real-user data in general, read When to Use Synthetic vs Real User Monitoring for Performance. For the wider tool taxonomy, use Comparing PageSpeed Monitoring Tools: Features Agencies Need. Those two pieces stay useful after you shortlist SpeedCurve or Watcher, because they separate measurement jobs from vendor brands.

SpeedCurve vs Apogee Watcher: side-by-side

Use this as a decision grid, not a contract. Verify numbers on each vendor’s site before you sign, and re-check after any Embrace or Watcher plan change. Sticker prices move faster than blog posts.

Topic	SpeedCurve (typical public positioning)	Apogee Watcher
Synthetic engine	Lab synthetic checks; Lighthouse-oriented metrics; scripting and regions via plan	PageSpeed Insights API (Lighthouse lab + CrUX where available)
Real-user / field data	First-party RUM plus synthetic; sampling to control cost	CrUX in PSI results; no first-party RUM
Entry pricing (public list)	Starter from $90/mo; Growth from $576/mo; Enterprise custom ([pricing](https://www.speedcurve.com/pricing/))	$9/mo Personal through $199/mo Agency ([pricing](https://apogeewatcher.com/pricing))
Billing drivers	RUM page views + synthetic check volume	Organisation / site / test quota tiers
Multi-site / multi-tenant	Strong for product teams; unlimited users/teams on published plans	Organisations, sites, pages, RBAC built for agency portfolios
Page discovery	Configure URLs and sites in product workflows	Automated sitemap + crawl paths
Alerts	Budgets and alerts in product (see SpeedCurve docs)	Email today; more channels per roadmap
Coaching / culture	Explicit performance-culture and consult positioning	Product-led monitoring; no bundled coaching bench
Best day-one use	Deep RUM + synthetic on owned high-traffic sites	Portfolio PSI monitoring without client-side scripts

Treo is another CrUX-heavy option for field exploration; we covered that split in Treo vs Apogee Watcher. GTmetrix remains useful for deep lab forensics on one URL; see GTmetrix vs Apogee Watcher. Neither replaces a SpeedCurve RUM programme on an owned storefront, and neither replaces portfolio PSI schedules on twenty unrelated CMS stacks.

When SpeedCurve is the better primary choice

Choose SpeedCurve as the primary monitor when:

First-party RUM is non-negotiable on sites you control (logged-in journeys, element-level INP work, conversion correlation).
Leadership already expects SpeedCurve-style dashboards and competitor benchmarks for a small set of revenue URLs.
You can fund Starter or Growth against one brand (or a tiny brand set) and engineering will maintain scripts and budgets.
Private agents, SSO, or long retention are part of the procurement checklist.

In those cases a “SpeedCurve alternative” search often ends with staying on SpeedCurve, or pairing it with a cheaper portfolio layer rather than ripping it out.

When Apogee Watcher is the better SpeedCurve alternative

Choose Watcher as the primary portfolio layer when:

You manage many client hosts and need scheduled mobile and desktop checks with CrUX context, without placing RUM snippets everywhere.
SpeedCurve pricing at Growth-scale check volumes would consume the monitoring line item for the whole book of business.
Automated discovery matters because marketing ships new templates faster than anyone updates monitor lists.
You need Admin / Manager / Viewer separation so account managers can read results without changing quotas.

If the pain is “we still run Monday PSI by hand,” start with PageSpeed Insights vs Automated Monitoring and How to Set Up Automated PageSpeed Monitoring for Multiple Sites. Budgets and thresholds are covered in The Complete Guide to Performance Budgets for Web Teams. Those guides assume you already know which vendor owns RUM versus which owns the portfolio calendar.

Layer SpeedCurve and Watcher instead of replacing either

Most mature agencies do not need a religious pick. A common pattern is SpeedCurve (or DebugBear) on one or two flagship properties where RUM and deep diagnostics pay for themselves, while Watcher carries scheduled PSI coverage across the long tail of client sites. That is the same “layer, don’t replace” stance we take against DIY Lighthouse CI and free one-off tools.

Watcher does not replace SpeedCurve’s RUM attribution, private agents, or coaching bench. SpeedCurve is not priced or shaped as a low-overhead multi-tenant PSI portfolio for dozens of unrelated CMS stacks. Buy the job you have, then add the second tool only where the first leaves a hole.

Try a multi-page sample without an account at apogeewatcher.com/check, or start a free Apogee Watcher account and schedule a baseline on your noisiest client this week.

FAQ

Is Apogee Watcher a full SpeedCurve alternative?

For portfolio synthetic monitoring with CrUX-in-PSI, multi-tenant roles, and lower entry pricing, yes as a category alternative. For first-party RUM, conversion correlation, and SpeedCurve’s coaching model, no. Keep SpeedCurve (or another RUM product) where those jobs matter.

How does SpeedCurve pricing compare at agency scale?

Public list prices start at $90/mo (Starter) and $576/mo (Growth), with usage driven by RUM page views and synthetic checks. Watcher publishes $9–$199/mo tiers with organisation and test quotas. Compare licence cost plus the hours you spend maintaining URL lists and client decks, not the sticker alone.

Does Watcher include real-user monitoring like SpeedCurve RUM?

No. Watcher shows CrUX field metrics when Google returns them for the tested URL. We do not inject a first-party RUM script. If you need session-level RUM beyond CrUX, use SpeedCurve or a peer RUM tool on those origins.

Can we run SpeedCurve and Apogee Watcher together?

Yes. Many teams keep premium RUM+synthetic on flagship sites and use Watcher for breadth across the rest of the portfolio. Neither product requires you to rip out the other.

Does SpeedCurve replace Google PageSpeed Insights?

No. SpeedCurve is a commercial monitoring platform. PSI remains a useful free diagnostic and the API behind Watcher’s scheduled runs. See our tools comparison for agencies for how diagnostics and monitoring differ.

Choose the primary tool by job: RUM-depth on owned sites versus multi-tenant PSI coverage across clients. Confirm SpeedCurve pricing and Watcher limits on the live pricing pages, then pilot two weeks on the sites that create the most noise in Slack. A short pilot beats a twelve-month commitment made from a feature grid alone.

For hosted multi-site monitoring without client-side scripts or per-client API keys, start free on Apogee Watcher.

References

ICMP vs TCP vs UDP: the difference, explained for developers

Sun, 13 Sep 2026 10:00:58 +0000

TL;DR. ICMP, TCP and UDP all travel inside IP packets, but they ask different questions. ICMP echo (ping) asks "does this address answer at all?" and needs no port. TCP asks "is a program listening on this port?" and gets a clear yes, a clear no, or silence. UDP asks nothing by itself: you only learn something if the program on the other side chooses to reply. A closed port answers differently on each one, and that difference is most of what a network check can and cannot tell you. The original post has the same text with a collapsible FAQ and the RFC list at the end.

One address, three ways to knock

Every packet on the internet is an IP packet. IP knows how to move a packet from one address to another and nothing else. One byte in the IP header says what is inside, and IANA keeps the list: ICMP is protocol 1, TCP is protocol 6, UDP is protocol 17. The three are siblings: same addresses, same routers, and a different promise about what happens once the packet arrives.

Three ways to knock on one address. The last row of each lane is what silence means there.

	ICMP	TCP	UDP
Specification	RFC 792 (1981), RFC 4443 for IPv6	RFC 9293 (2022)	RFC 768 (1980)
IP protocol number	1 (58 for ICMPv6)	6	17
Ports	none	source and destination	source and destination
Handshake	none	three packets before any data	none
Delivery guarantee	none	ordered, complete, retransmitted	none
Header	8 bytes for an echo	20 bytes minimum	8 bytes
Typical users	ping, traceroute, error reports	HTTP/1.1 and HTTP/2, TLS, SSH, SQL, SMTP	DNS, NTP, QUIC and HTTP/3, video, games

ICMP: the network talking about itself

ICMP is not a transport. RFC 792 says it plainly: ICMP "is actually an integral part of IP, and must be implemented by every IP module." It exists so that routers and hosts can report on delivery: this destination is unreachable, that packet ran out of hops. ICMP messages have a type and a code but no ports, because nothing sits on top of ICMP waiting for them. The IP stack consumes them itself.

The one ICMP message everybody has typed is the echo. ping sends an echo request (type 8, or 128 in ICMPv6) and the far host's kernel sends back an echo reply (type 0, or 129). No program had to be running. RFC 1122, the host requirements standard, makes it a rule: "Every host MUST implement an ICMP Echo server function that receives Echo Requests and sends corresponding Echo Replies."

$ ping -c 1 uptimepage.dev
PING uptimepage.dev (204.168.246.94): 56 data bytes
64 bytes from 204.168.246.94: icmp_seq=0 ttl=50 time=96.069 ms

A reply proves two things: the address belongs to a machine that is switched on, and packets can travel there and back. It also gives you the round-trip time for free.

That is all a ping proves. It says nothing about whether a web server, a database or anything else is running, and a host can answer every ping while every service on it is dead.

Two things about ICMP surprise people. The first is that no reply does not mean down. Firewalls drop ICMP all the time. On AWS a fresh security group blocks it, and the documentation says so: "To ping your instance, you must add one of the following inbound ICMP rules." Before you rely on ping for a host, confirm the host answers a ping at all.

The second is that sending ICMP needs privilege. A raw socket needs CAP_NET_RAW. Linux has an unprivileged alternative, the ICMP datagram socket, but the kernel documentation says the default ping_group_range is "1 0", "meaning, that nobody (not even root) may create ping sockets." Docker opens that range in every container that gets its own network namespace, which is why ping works there without anyone thinking about it; a container on --network host inherits the host's setting instead. I learned the limits of that the slow way: the same probe binary that pinged fine in Docker could not open an ICMP socket at all on a Firecracker microVM, until I gave it CAP_NET_RAW.

TCP: a promise before the first byte

TCP is a connection. Before either side sends a byte of data, the two ends run the three-way handshake: SYN, SYN-ACK, ACK. RFC 9293, which replaced the 1981 specification in 2022, calls it "the procedure used to establish a connection." It costs one round trip, and that round trip pays for every promise TCP makes afterwards: bytes arrive in order, lost segments are sent again, the receiver is not flooded (flow control), and the network is not flooded (congestion control). To the program it looks like a stream. You write bytes on one side and read the same bytes, in the same order, on the other.

This is what HTTP/1.1, HTTP/2, TLS, SSH, PostgreSQL, MySQL, SMTP and most of what you deploy run on.

The handshake is also a good probe, because a SYN gets one of three answers:

SYN-ACK. A program is listening. The connect succeeds.
RST. Nothing is listening. RFC 9293 says that if the connection does not exist, "a reset is sent in response to any incoming segment except another reset." Your connect fails at once with "connection refused".
Nothing. A firewall dropped the SYN, or the host is gone. The kernel keeps retrying. On Linux the default tcp_syn_retries is 6, and the kernel documentation puts the final timeout for an active connection attempt at 131 seconds.

Here are all three from my laptop. The last one has a three second limit; without it macOS waits 75 seconds.

$ nc -zv uptimepage.dev 443
Connection to uptimepage.dev port 443 [tcp/https] succeeded!

$ nc -zv 127.0.0.1 9
nc: connectx to 127.0.0.1 port 9 (tcp) failed: Connection refused

$ nc -zv -G 3 uptimepage.dev 8443
nc: connectx to uptimepage.dev port 8443 (tcp) failed: Operation timed out

Nmap's documentation gives these states the names everyone uses: open, closed and filtered.

What a successful handshake proves: a process accepted a connection on that port. Not that it is healthy. A database that accepts connections and then rejects every query still passes. That is why a TCP check belongs on things that have no HTTP surface (SSH, SMTP, a database port) and an HTTP check belongs on things that do.

UDP: a message, nothing more

UDP is the smallest transport there is. RFC 768 fits on three pages. It adds a source port, a destination port, a length and a checksum to an IP packet, 8 bytes in total, and stops. The RFC says: "The protocol is transaction oriented, and delivery and duplicate protection are not guaranteed." There is no handshake, no acknowledgement, no ordering and no retransmission, and no congestion control either. RFC 8085 opens by saying UDP "has no inherent congestion control mechanisms" and then spends more than fifty pages telling application designers how to add their own.

Why anyone uses it: latency and control. A DNS lookup is one packet out and one packet back, no handshake. RFC 1035 capped a DNS answer over UDP at 512 bytes. EDNS(0) (RFC 6891) later let a client advertise a bigger buffer, so most answers fit in one datagram today; one that still does not fit is cut short with the TC bit set, and the client asks again over TCP (RFC 7766). NTP, most game traffic and real-time video use UDP because a late packet is worth less than the next one, so waiting for a retransmission is the wrong trade. And QUIC, the transport under HTTP/3, is built entirely on UDP. RFC 9000 states that "QUIC packets are carried in UDP datagrams" and gives the reason as "to better facilitate deployment in existing systems and networks". In practice that means QUIC does its own handshake, loss recovery and congestion control in the application, and can change them without waiting for every operating system to update its TCP stack.

$ dig @1.1.1.1 uptimepage.dev A +noall +answer +stats
; <<>> DiG 9.10.6 <<>> @1.1.1.1 uptimepage.dev A +noall +answer +stats
; (1 server found)
;; global options: +cmd
uptimepage.dev.3600INA204.168.246.94
;; Query time: 125 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Wed Sep 02 11:40:44 EEST 2026
;; MSG SIZE  rcvd: 59

One datagram out, one datagram back carrying a 59-byte answer (67 bytes with the UDP header), and no handshake before it. Over TCP the same lookup costs two round trips, one for the handshake and one for the query, which is why DNS defaults to UDP.

Now the probing problem. Send a UDP datagram to a port and what comes back?

If nothing listens there, the host's IP stack should answer with an ICMP error. RFC 1122 again: "If a datagram arrives addressed to a UDP port for which there is no pending LISTEN call, UDP SHOULD send an ICMP Port Unreachable message." So "closed" is detectable, and it is ICMP that detects it.

If something listens, you get whatever that program chooses to send. A DNS server sends an answer. A syslog receiver sends nothing, ever. So silence means open, or filtered, or dead, and Nmap has a state for this, open|filtered. Its documentation explains: "This occurs for scan types in which open ports give no response."

The consequence is that a generic "UDP port check" does not exist. To check a UDP service you have to speak its protocol: send a real DNS query and read the answer, send a real NTP request and check the timestamp. This is why monitoring tools have a DNS check and a TCP check but rarely a UDP check.

The same closed door, three answers

Situation	ICMP echo	TCP SYN	UDP datagram
Service up	echo reply	SYN-ACK	the program's reply, or nothing
Port closed, host up	echo reply (ICMP has no ports)	RST, "connection refused"	ICMP port unreachable
Firewall drops the packet	silence	silence, retried for about two minutes	silence
Host gone	silence, or an unreachable error from the last router	silence, or that same ICMP error surfacing as "no route to host"	silence, or that same ICMP error

Read the table by columns. TCP is the only one where "closed" and "filtered" look different on their own, UDP borrows ICMP to say "closed", and ICMP cannot see ports at all. Silence is the ambiguous answer on all three, which is why no check should conclude "down" from one silent probe sent from one place. Two or three consecutive misses, seen from more than one network, is the honest bar. I wrote about that in how to stop false uptime alerts.

What this means for a monitor

I build Uptimepage, and its check types map onto these questions, so this is how the theory turns into a config.

Ping sends one ICMP echo with a 32-byte payload, the size Windows ping uses (Unix ping sends 56, as the transcript above shows), so middleboxes see ordinary diagnostic traffic. It tries the first IPv4 and the first IPv6 address of the host and splits the timeout between them. Silence for the whole budget is down, because an echo has no way to refuse. Use it for routers, gateways and hosts that expose no service.
TCP runs the handshake and closes the connection. Accepted is up, refused is down, and a timeout is recorded as an error with the time it took. An error counts as a failed check for alerting exactly like down, so a firewalled port still opens an incident once the confirmation threshold is met. Use it for databases, brokers, SSH and mail, where "a process accepted the connection" is the most you can assert from outside.
DNS sends a real query and checks the answer's content, so it is the UDP check for the one UDP service almost everyone depends on. It can point at a specific resolver, which matters when you want that server's view and not your cache's.
HTTP is a TCP handshake, then TLS, then a request, and it records the time spent in each phase: DNS, connect, TLS and time to first byte. The connect phase is the handshake from this article, measured inside a bigger check.

Pick the check that asks the question you care about, and remember what each one cannot see. There is a fuller version of that decision in the monitor types documentation, and a story-shaped version in the mystery of the "down" website.

Common questions

Is ICMP a transport protocol like TCP and UDP?

No. ICMP is a control protocol that belongs to IP itself, and RFC 792 says every IP module must implement it. It has no ports and carries no application data. It reports on delivery: a destination is unreachable, a packet ran out of hops, and the echo pair that ping uses.

Why does ping work while my website is down?

Because ping only proves that the host's IP stack answers and that packets can travel there and back. The kernel sends the echo reply, so no web server, database or other program has to be running. A host can answer every ping while every service on it is dead.

Why does a closed TCP port fail at once but a firewalled one hangs?

A closed port sends back a TCP reset, so the connect fails immediately with connection refused. A firewall that drops the SYN sends nothing, and the kernel keeps retrying. On Linux the default is six retries, and the kernel documentation puts the final timeout at 131 seconds.

Can I check a UDP port the way I check a TCP port?

No, because an open UDP port sends nothing back unless the program behind it chooses to answer. A closed port is detectable, since the host should reply with an ICMP port unreachable message, but open and filtered both look like silence. To check a UDP service you have to speak its protocol, for example send a real DNS query.

If UDP is unreliable, why does HTTP/3 use it?

UDP itself gives no delivery guarantee, and QUIC adds its own on top. QUIC runs inside UDP datagrams and does its own handshake, retransmission and congestion control in the application, which lets it evolve without waiting for every operating system to update its TCP stack.

Sources

IETF, RFC 792: Internet Control Message Protocol, September 1981.
IETF, RFC 4443: ICMPv6 for the Internet Protocol Version 6, March 2006.
IETF, RFC 768: User Datagram Protocol, August 1980.
IETF, RFC 9293: Transmission Control Protocol, August 2022.
IETF, RFC 1122: Requirements for Internet Hosts, Communication Layers, October 1989. Sections 3.2.2.6 and 4.1.3.1.
IETF, RFC 8085: UDP Usage Guidelines, March 2017.
IETF, RFC 1035: Domain Names, Implementation and Specification, November 1987. Section 4.2.1.
IETF, RFC 6891: Extension Mechanisms for DNS (EDNS(0)), April 2013.
IETF, RFC 7766: DNS Transport over TCP, Implementation Requirements, March 2016.
IETF, RFC 9000: QUIC, a UDP-Based Multiplexed and Secure Transport, May 2021.
IANA, Assigned Internet Protocol Numbers.
Linux kernel, IP sysctl documentation, tcp_syn_retries and ping_group_range.
Nmap, Port scanning basics, the six port states.
Amazon Web Services, Security group rules for different use cases, rules for ping/ICMP.

I build Uptimepage because I wanted these checks for my own services: ping, TCP, DNS and HTTP with per-phase timing, plus TLS and domain expiry, heartbeats for background jobs, and browser flows, from several regions. It is AGPL-3.0 open source, so you can run it on your own server if you prefer.

Now the question for the comments: do you still run ping checks on anything, and what for? I keep them for routers, gateways and hosts that expose no service, and almost nothing else. If a ping check has earned its keep for you somewhere I have not listed, I would like to hear where.

AWS's Own Dashboard Could Not Say It Was Down. Four of Our Monitors Had the Same Bug.

Thu, 27 Aug 2026 19:32:33 +0000

A signal has three states and most code stores two. When “cannot measure” is written with the same value as “fine,” the alarm branch becomes unreachable in exactly the situation the alarm exists for.

On 7 December 2021, the operations teams inside AWS lost the ability to see their own network. Not the ability to fix it. The ability to see it. The company's post-event summary says so in plain language: congestion "immediately impacted the availability of real-time monitoring data for our internal operations teams, which impaired their ability to find the source of congestion and resolve it" (Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region, "Issue Summary" section). While customers refreshed a status page, the people responsible for one of the largest computing regions on earth "relied on logs to understand what was happening" (same document, same section). Even the public dashboard was caught in it: "the networking congestion impaired our Service Health Dashboard tooling from appropriately failing over to our standby region" ("Event Communication" section).

The lesson usually drawn from that day is deployment advice. Monitor your monitors. Host the status page somewhere else. Both are sound, and both miss what makes the incident interesting. The monitoring did not miss the failure. It could not report the failure, because the failure and the reporting path shared a dependency. Those are different defects, and the second one has a signature you can find in your own code without waiting for an outage.

Three states, two values

A monitoring signal has at least three states: measured and bad, measured and fine, and could not measure. Most of the code we audited encodes the third state with the same value as the second. When that happens, every branch that would have raised the alarm becomes unreachable in exactly the situation the alarm exists for. Unreachable in the strict sense: while the sensor is the thing that broke, no input can produce the alarm value.

You can check for this by construction rather than by misfortune. For each signal you own, ask one question. What comes out when this cannot be read, and can a healthy system produce that same value?

We asked that question about our own system

Over one working night we audited four independent monitoring and status signals in a single codebase we operate. All four collapsed "unknown" into "fine," each in its own way.

signal	what "I cannot tell" produced	consequence we measured
a deploy-status probe	a lag warning with an empty count	fired on 100% of runs for 28 days; the "no lag" branch was unreachable
a fleet health dashboard	the verdict `STALLED`	reported a fully up-to-date system as stalled
a simulation's scarcity input	`0.0`	a region with no data read identically to an abundant one
an item-identity lookup	a guess derived from the display name	a garment classified as a liquid container

The 28-day figure is the interval between a format change we can date and the night of the audit. For four weeks a warning fired on every single run, which meant it warned about nothing, which meant nobody read it.

The arithmetic version of the defect is the sharpest, so we shipped it as a runnable script. Take a health score averaged from three weighted inputs, alarm threshold 0.70. Two of the three inputs are dead and report the calm value, 0.0. The one live input is screaming at 0.95. The script prints:

naive sum over all three   : 0.285   -> HEALTHY
renormalised over LIVE only: 0.950   -> ALARM

Identical inputs. And 0.285 is a ceiling, since 0.95 is the largest value the live input can report. With two legs dead and voting calm, the alarm was never rare. It was impossible. Renormalising over the inputs that can actually be seen asks the question the operator is actually asking: given what I can observe, how bad is it?

The one that got it right

The same codebase contained a fifth signal that had the answer built in, and it convinced us the fix is a value rather than a discipline. Its unknown state maps to 0.20, on a scale where its two calm states are 0.10 and 0.05. Nothing else about it is clever. An unrecognised input simply produces a number that no quiet state can produce, so the downstream branch for "something needs a look" stays reachable at all times. It asks nothing of anyone's memory and no change of habit. The distinguishable value does the remembering.

That is the difference between this and one more essay telling engineers to be more careful. The four broken signals were written by people being careful. The fifth was written so that carefulness was not load-bearing.

Honest scope

Our data is one codebase, one night, one investigator, and after the first instance we were primed to see the shape everywhere. The four signals are independent subsystems, which is the sample's strength, but this is a report from our own floor and no survey of anyone else's software.

The check

Skip the moral. Run the audit. For each signal you operate, ask what it emits when it cannot read, and whether a healthy system can emit the same thing. If the answers collide, you do not have a monitoring gap. You have an alarm branch that is unreachable precisely when it matters, and one distinguishable value fixes it.

AWS's operators spent that December morning reading raw logs because their telemetry answered "I cannot see" in a voice indistinguishable from silence. Ours answered "fine." Same bug. The only reason we found ours first is that we went looking.

Reproduction: every number we computed comes from repro_unknown_equals_healthy.py, shipped beside this essay. Run it and disagree with the arithmetic directly.

Sources: Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region (aws.amazon.com/message/12721), sections as cited in-text. All other figures are our own measurements, per the reproduction script and the scope note above.

An audit trail has the same three states as a monitor.

Everything above turns on one value: what a signal emits when it cannot see. The same question applies to the record a system leaves behind, where “no entry” and “nothing happened” are the pair that get confused. Chain of Consciousness is a tamper-evident record written as the work happens rather than reconstructed once somebody asks, so a gap in the record reads as a gap instead of as a quiet success. It does not make a system behave. It makes the difference between unobserved and fine a thing you can check.

Hosted Chain of Consciousness · Verify a record

pip install chain-of-consciousness · npm install chain-of-consciousness

Originally published at vibeagentmaking.com.

Treo vs Apogee Watcher: CrUX Scale, Competitor Benchmarks, and Agency Portfolios

Mon, 24 Aug 2026 09:02:00 +0000

After the CrUX Dashboard and Looker Studio report retired, many teams rebuilt their field-data workflow from PageSpeed Insights, Search Console, and third-party CrUX explorers. Treo is one of the products agencies shortlist when they want CrUX at portfolio scale without writing BigQuery jobs. Apogee Watcher addresses a related but different bottleneck: keeping scheduled lab checks, budgets, and alerts aligned across many client sites when URL lists change every sprint.

Both products show lab and field context. Both matter to agencies. The decision is usually workflow fit, not which logo wins a feature grid. Below we compare Treo and Watcher on CrUX depth, competitor views, multi-site operations, and price bands. Figures change; verify limits on Treo pricing and Apogee Watcher pricing before you buy.

What Treo is built for: CrUX scale and competitor benchmarks

Treo markets itself around the Chrome User Experience Report: raw field data, historical trends, connection and country breakdowns, and competitor comparisons without standing up your own BigQuery pipeline. Public materials emphasise large CrUX URL budgets (from dozens on the free tier to five figures on Scale and Enterprise), multi-year retention, and views that go beyond the p75 snapshots many free tools show.

That positioning matters when your brief is field exploration at scale:

You need hundreds or thousands of URLs in CrUX with history, not just origin-level Search Console summaries.
You want to chart competitors beside your client on LCP, INP, or CLS using the same CrUX source.
You export or API-pull field series into your own reporting stack (paid tiers).

Treo also adds scheduled Lighthouse monitoring on paid plans, with tiered page counts and frequency trade-offs (for example, hourly checks consume more capacity than eight-hour cadences). The free tier is CrUX-only: useful for evaluation, not for lab regression alerts across a retainer.

The screenshot above is from Treo’s public product pages: connection mix by geography on a world map, one of the views teams use when they explain field data to stakeholders who will never open BigQuery. If your job is “show me five years of CrUX for this URL set and my competitors,” Treo is aimed at that job.

For where CrUX still lives inside Google’s own surfaces after dashboard retirement, see CrUX Dashboard Retired: Where to Get TTFB, INP, and Field History.

What Apogee Watcher is built for: a web performance monitoring tool for agency portfolios

Apogee Watcher is a multi-tenant pagespeed monitoring platform built for agencies. We run scheduled tests through Google’s PageSpeed Insights API, which returns Lighthouse lab metrics plus CrUX field slices where Google publishes them for the tested URL. We emphasise organisations, sites, roles (Admin, Manager, Viewer), automated page discovery from sitemaps and crawl paths, performance budgets, and email alerts when thresholds breach.

We do not offer Treo-scale CrUX URL inventories or built-in competitor benchmarking boards. We do not replace a CrUX explorer when your primary question is “how does this competitor’s field INP trend over three years on five hundred URLs?”

We are strong when:

You manage many client properties and need scheduled lab coverage without maintaining separate spreadsheets per site.
Discovery must catch new templates and campaign landers before someone remembers to add URLs manually.
Account managers need read-only access without touching API quotas.
Entry price and predictable tiers matter for monitoring as its own retainer line.

Setup patterns and discovery behaviour are in Getting Started with Apogee Watcher: A Step-by-Step Setup Guide. For how synthetic schedules relate to field proof, pair with When to Use Synthetic vs Real User Monitoring for Performance.

Treo vs Apogee Watcher: CrUX monitoring tools compared

Use this table as a decision grid, not a scorecard. Confirm numbers on each vendor site.

Topic	Treo (typical positioning)	Apogee Watcher
Primary strength	CrUX URL scale, history, competitor benchmarks	Multi-site scheduled PSI monitoring, discovery, agency RBAC
Test engine	CrUX-native; Lighthouse on paid tiers	PageSpeed Insights API (Lighthouse lab + CrUX where available)
CrUX URL budgets	50 free; 1K–20K+ on Vital–Scale; 50K+ Enterprise	Per-URL CrUX via each PSI result; no competitor boards
Competitor benchmarking	Built into product narrative	Not a core feature
Data retention (public FAQ)	Unlimited historical access via UI/API	30–365 days by plan (confirm on pricing)
Team members	5–50 on paid tiers	Unlimited on published tiers
Entry paid pricing	Vital from $75/mo (CrUX + Lighthouse)	From $9/mo Personal through $199/mo Agency
Free tier	50 CrUX pages; no Lighthouse	Free tier available (confirm current limits)
API	Pro tier and above	Product API focus differs; confirm live features
Best day-one use	Field exploration, competitive charts, large CrUX lists	Portfolio lab monitoring without per-client instrumentation

Treo’s paid entry at $75/mo reflects CrUX infrastructure and history, not a stripped-down lab runner. Watcher’s lower entry reflects a different scope: scheduled synthetic monitoring and agency workflow, not five-figure CrUX URL catalogs.

Broader tool taxonomy (where Treo sits beside GTmetrix, DebugBear, and PSI) lives in Comparing PageSpeed Monitoring Tools: Features Agencies Need. For another premium synthetic comparison with optional RUM, see DebugBear vs Apogee Watcher: Synthetic Monitoring for Multi-Site Teams.

When Treo is the better primary choice

Favour Treo when CrUX is the product, not a sidebar on lab reports:

Competitive narratives: Monthly reviews that compare your client to named competitors on Core Web Vitals using the same CrUX source.
Large URL lists in field data: Category or marketplace sites where you must monitor hundreds of URLs in CrUX, not ten hero routes in lab only.
Long historical arcs: Multi-year trends for leadership decks; Treo’s marketing emphasises five-year history and unlimited retention in FAQ materials.
Export and API: Pro/Scale customers pulling JSON series into BI tools without maintaining BigQuery pipelines.

If first-party RUM on the client origin is also non-negotiable, Treo still is not a RUM vendor; you would add analytics or a RUM product separately. PageVitals vs Apogee Watcher covers a different RUM-first angle for WordPress-heavy stacks.

When Apogee Watcher is the better primary choice

Favour Watcher when the retainer is “keep these URLs inside budget and tell us when a deploy regresses”:

Many sites, modest field ambition: You need lab schedules and CrUX context on priority URLs, not a five-thousand-row CrUX inventory on day one.
URL churn: Marketing publishes routes faster than spreadsheets update; discovery reduces stale monitor lists.
Role separation: Managers tune budgets; viewers read dashboards; admins own billing without sharing one login.
Cost at scale: Monitoring line items must survive procurement at ten to thirty sites; entry tiers matter.

Watcher does not try to win on competitor benchmarking charts. We try to win on operational coverage: the same URLs tested on a cadence, with alerts that reach a human. Manual PSI rituals stop scaling long before CrUX URL math becomes the bottleneck; that case is in PageSpeed Insights vs Automated Monitoring: When Manual Checks Aren't Enough.

Layer Treo and Watcher instead of forcing one vendor

Many agencies split roles:

Treo for field exploration, competitor slides, and CrUX-heavy discovery when a client asks “how do we compare in the market?”
Watcher for scheduled lab monitoring, budgets, and regression alerts across the whole portfolio.

That is the same layer-don’t-replace principle we use elsewhere: add CrUX exploration beside synthetic monitoring; do not pretend one subscription replaces both jobs. A team can run Treo on strategic URL sets and Watcher on every retainer property’s priority routes without duplicating work if roles are explicit in the SOW.

Neither tool replaces Search Console’s CWV report or one-off PSI checks before a sponsor call. Keep those; assign each tool a named owner in the runbook.

FAQ

Can Apogee Watcher replace Treo for large-scale CrUX work?

They overlap on “performance monitoring” but optimise for different questions. Treo is the stronger fit for large CrUX exploration and competitor benchmarking. Watcher is the stronger fit for multi-tenant scheduled PageSpeed monitoring with discovery and agency roles. Many teams use both.

Does Apogee Watcher replace Treo’s CrUX history?

No. Watcher shows CrUX where PSI returns field data for each tested URL and stores lab history for trends and alerts. It does not replicate Treo-scale CrUX URL budgets or competitor boards.

Can I start on Treo’s free tier and add Watcher later?

Yes. Treo’s free plan (50 CrUX pages, no Lighthouse) suits evaluation of field views. When you need scheduled lab checks across many client sites, add Watcher rather than stretching Treo’s Lighthouse limits alone.

Which tool is cheaper for an agency with twenty sites?

Compare total cost with your URL counts and roles, not headline price alone. Watcher’s published tiers start lower; Treo’s Vital plan at $75/mo includes CrUX and Lighthouse capacity you may not need on every retainer. Run the maths for CrUX pages, Lighthouse pages, seats, and retention you actually use.

Do either tool replace BigQuery CrUX exports?

Treo abstracts BigQuery for dashboard and API users. Watcher does not. If you need custom SQL across the public dataset, you still use BigQuery or a vendor built for that scale; see our CrUX replacement guide linked above.

If you want scheduled PageSpeed checks across client sites with budgets and discovery, start from pricing or run a free domain check. If your brief is CrUX competitor charts at thousands of URLs, evaluate Treo on its own terms and layer monitoring where lab cadence still matters.

References

Why is my Chrome UX Report much worse than Lighthouse? How can I improve it?

Sat, 22 Aug 2026 06:29:31 +0000

CrUX field percentiles often look worse than a green Lighthouse run on the same URL. Here is why the gap is normal, what moves real-user Core Web Vitals, and what not to chase.

On Monday the Lighthouse performance score on checkout was 94. Mobile lab LCP sat under 2.5 seconds and CLS looked clean. On Wednesday someone opened PageSpeed Insights field data for the same URL and Chrome UX Report still showed Needs improvement on mobile, with LCP p75 above the Good threshold.

That pattern is not a broken tool. It is the ordinary gap between lab and field on a URL that real visitors use under conditions your Lighthouse run never reproduced. The question behind the screenshot is usually two-part: why is CrUX worse than Lighthouse, and what can you change this sprint that will actually move field percentiles instead of chasing a single green lab run?

Why does Chrome UX Report look worse than Lighthouse on the same URL?

Lighthouse, including the lab section in PageSpeed Insights, measures a controlled load: fixed device profile, emulated network, cold or warm cache depending on settings, and one navigation path you chose for the audit. Chrome UX Report (CrUX) aggregates real Chrome sessions over a rolling window (currently 28 days for the public dataset). Field LCP, INP, and CLS reflect actual devices, networks, cache states, scroll depth, and interaction paths across thousands of visits, not one scripted run.

Google's lab versus field data differences article states the split plainly: Lighthouse returns one LCP candidate for a given run, while field LCP on the same URL often spans several elements depending on viewport and how far the user scrolled. A hero image that wins LCP in lab may lose to a product grid on a shorter phone, or never become the LCP element if most users bounce before it paints.

CrUX also reports percentiles (typically p75 for Core Web Vitals), not your best session. A Lighthouse performance score near 100 describes one favourable run on one profile; CrUX describes what most visitors experienced across the rolling window. If even a quarter of sessions on mid-range Android hardware hit slow LCP, the p75 can sit in Needs improvement while lab looks green on a desktop profile someone ran by mistake.

Signal	What it measures	Typical gap when CrUX looks worse
Lighthouse (lab)	One controlled run on a URL you choose	Best-case load on a fixed device and network profile
CrUX (field)	28-day p75 from real Chrome sessions	Slower devices, mixed cache states, full interaction paths
PageSpeed Insights	Shows both on one page	Easy to compare the wrong rows (desktop lab vs mobile field)

How do device mix, cache warmth, and the 28-day window widen the CrUX versus Lighthouse gap?

Three mechanics explain most tickets where CrUX looks much worse than Lighthouse without a deploy bug.

Device and network skew widens lab versus field

Lab uses a defined mobile or desktop strategy. CrUX includes every eligible phone and desktop session that visited the URL during the collection period. If your traffic skews toward older Android handsets on congested mobile networks, field LCP and INP will punish layouts that a Moto G emulation in Lighthouse barely stresses. PageSpeed Insights already splits mobile and desktop field rows; compare like with like before you conclude the tools disagree.

Cache warmth and return visits change who looks fast

Lighthouse often runs as a cold or lightly cached load. Returning visitors with warm HTTP cache, service worker caches, or prerendered shells can look faster in field for LCP while first-time visitors still struggle. The opposite happens too: lab on a fast connection hides consent banners and personalisation layers that only appear for logged-in or European traffic, which field data must include.

The 28-day rolling average lags your latest deploy

CrUX is not yesterday's score. Sessions from before your fix remain in the window until they age out, so field percentiles improve gradually even when today's lab runs are green. We unpacked that lag on the Watcher blog in Why Your Core Web Vitals Fix Isn't in CrUX Yet (28-Day Window). If you shipped last week, amber CrUX beside green Lighthouse may mean the average has not caught up yet, not that the optimisation failed.

What changes actually improve CrUX Core Web Vitals percentiles?

Field data moves when enough real sessions experience better LCP, INP, or CLS. Lab work still matters because it tells you what to ship and gives you a reproducible before-and-after on the same URL list; field tells you whether enough users received the benefit once those sessions enter the CrUX window.

LCP levers that survive contact with real traffic

Serve the LCP image or text block from the same origin when possible; cut redirect chains on the hero asset.
Size and compress the true LCP element for the viewport widths that dominate analytics, not only the designer's reference frame.
Reserve space for heroes, carousels, and video posters so late-loading assets do not reshuffle the LCP candidate between sessions.
Reduce render-blocking CSS and fonts on the template that owns the money URL, not only the marketing homepage.

INP levers on flows CrUX actually records

INP replaced FID as a Core Web Vital in 2024. Field INP punishes main-thread work during interaction: heavy JavaScript on product configurators, map panes, autocomplete, and checkout steps. Defer non-critical scripts until after first interaction, split long tasks, and test on mid-tier hardware with CPU throttling. A green lab TBT does not clear a bad field INP if users tap through a data grid built for desktop.

CLS levers that field sessions notice early

Reserve dimensions for ads, embeds, cookie banners, and dynamic promo slots before they inject. Late consent or A/B widgets are a common reason lab CLS looks fine while field CLS fails: the lab run never hit the variant your European traffic sees every day. After each change, keep scheduled lab runs on the same URL list and watch CrUX weekly; expect gradual movement, not a next-day flip. For how lab schedules and field snapshots fit together in a monitoring stack, see When to Use Synthetic vs Real User Monitoring for Performance.

What should you not do when trying to improve CrUX scores?

Some reported fixes optimise the Lighthouse screenshot, not the visitor experience that CrUX aggregates.

Do not treat one Lighthouse run as proof CrUX should already be green. Lab confirms the deploy; CrUX confirms the rolling aggregate moved.
Do not rerun Lighthouse until you get 100 and close the ticket. Without stored history you cannot show whether field-relevant templates changed.
Do not compare desktop lab to mobile CrUX (or the reverse) and call it a tooling bug.
Do not strip analytics or consent in lab only. If field traffic still loads those tags, CrUX will keep reporting the slower experience.
Do not expect origin-level field data to excuse a bad conversion URL. When PageSpeed Insights falls back to origin CrUX, a healthy homepage can mask a failing checkout template until URL-level samples exist.

Lab green, field amber: reconciling two PageSpeed truths covers the weekly reconciliation habit when both signals disagree. Use that piece for ongoing triage; use the checklist below when you need a one-pass improve list.

FAQ

Is CrUX more accurate than Lighthouse?

They measure different things. Lighthouse is accurate for repeatable lab diagnosis on a URL you choose. CrUX is accurate for how Chrome users experienced Core Web Vitals across the rolling window. For Search and page-experience reporting, treat CrUX (and Search Console URL groups fed by the same field data) as the public field reference when samples exist.

How long after a fix before CrUX improves?

There is no single flip day. High-traffic URLs often show movement within one to three weeks as post-fix sessions replace older ones inside the 28-day window. Low-traffic templates can take longer or never earn stable URL-level CrUX; use scheduled lab proof until samples appear.

Can I improve CrUX without real user monitoring?

Yes, by shipping changes that lab identifies on priority URLs and waiting for enough Chrome traffic to reflect them in CrUX. First-party RUM adds faster, segmented feedback if you already run a snippet, but CrUX improvement does not require buying RUM to start. It does require patience with the rolling average.

Why does PageSpeed Insights show different numbers than Search Console?

Both draw on CrUX for field Core Web Vitals, but grouping, thresholds, and UI differ. Align on URL, form factor, and date range before you argue the products disagree. Lab sections in PageSpeed Insights will always diverge from field rows by design.

How do you improve CrUX when Lighthouse already looks green?

Open PageSpeed Insights for the URL on mobile field data. Note LCP, INP, and CLS p75 and whether URL-level or origin-level samples apply.
Run a matching mobile lab audit on the same URL. Record which element Lighthouse reports as LCP and whether INP interaction was exercised.
List differences lab skipped: consent, personalisation, third-party tags, logged-in states, geographic variants.
Ship one field-relevant fix (LCP asset, INP task split, CLS reservation). Keep lab history on a schedule.
Re-check CrUX weekly with the collection period dates visible. Tell stakeholders field lags lab by design.

If step 5 is missing, you are optimising for a Lighthouse screenshot while CrUX still averages three weeks of older sessions. Pair the checklist with stored lab history so the next client question is a trend line, not two isolated PageSpeed Insights pastes.

What to read next when CrUX still trails Lighthouse

CrUX worse than Lighthouse is usually a population mismatch, a percentile versus single-run mismatch, or a rolling window that has not absorbed your fix yet. Use lab to decide what to ship; use field to learn whether enough real sessions improved. Read Why Your Core Web Vitals Fix Isn't in CrUX Yet for the timeline mechanics, and synthetic versus real user monitoring when you are choosing how much lab history to store beside CrUX. Improve the visitor experience first; the public percentile follows when traffic proves it.

References

Lab and field data differences (web.dev)
CrUX API documentation (Chrome for Developers)
Why Your Core Web Vitals Fix Isn't in CrUX Yet (28-Day Window) (Apogee Watcher)
When to Use Synthetic vs Real User Monitoring for Performance (Apogee Watcher)
Lab green, field amber: reconciling two PageSpeed truths (Apogee Watcher on Hashnode)

Originally published on Hashnode.

Rise and fall of a self-fixing agent

Sun, 16 Aug 2026 17:00:35 +0000

Originally published on my blog.

AI hype is all around us, and you probably heard of self-evolving, self-improving agents. An autonomous AI that keeps refining and fixing itself while working on something. Straight into tech nirvana and singularity.

I built one. A very low-key version. And it even worked for some time - I'll show you a few bugs it identified and successfully fixed.

Then it hit me with something nasty enough that I almost dropped the whole idea. I still use it on some routines but I keep self-improvement away from my main projects.

Let me introduce someone to you

Here are my AIs who are the main characters of this story.

Simona: a Claude Code with a bunch of skills, hooks, CLI programs, and a rough personality. Well, okay, it's just Claude Code which I like calling by name.
Marlow: a loop agent, designed, built and monitored by Simona. No personality, just a cold focus on its job. Just like me.

Both female identities, because nobody promised gender equality on my Macbook.

So, is it one of those loops everybody was talking about a month ago?

Yes. It's a while loop which runs all the time while my Mac is awake. A macOS LaunchAgent (com.marlow.tick, StartInterval 1200) fires a shell script every 20 minutes, and that script starts a headless Claude Code session.

claude -p "Work on the top priority task or continue unfinished one"

Okay, step back. I didn't create this myself. I got inspired by the Ralph loop half a year ago. It was such a simple idea that nobody really took it seriously back then. Until those loops took the AI-media space by storm.

I asked Simona to create a task-based system:

Take the Ralph loop, make it pull a task from a pool and work until the completeness criteria is met. Each task has its own.
I can add tasks to the pool manually (by asking Simona, she is my hands) or they are submitted by the scheduler.
There is a config with scheduled tasks, which is managed by Simona
Simona has a skill that explains everything about this loop and how to control it

I liked that. Simona designed the details, I picked from the options, and told her to choose the name. This is how Marlow appeared.

There is much more to add to this story - how we argued with Simona about Marlow's personality, safety restrictions and should it be "it" or "she" (as you already know, I won) - but I'll keep it for some other article. We have bugs to talk about.

Lots of code, lots of bugs

The first pilot task I gave to Marlow was a writing loop. I wanted her to collect some news about AI, send the most interesting things to my Telegram, and write an aggregated article for me once a week. My personalized AI slop journal. It's here by the way and it now has more views than my own blog. Yeah... singularity.

I asked Simona to implement all of that - create tools, CLIs, instructions, whatever else is needed for Marlow to schedule and execute this. So I could get my weekly reading.

And Simona did it, the loop started its endless run on my Mac. Marlow kept logging everything she did and summarized it into her memory file. I kept asking Simona to check both for errors and issues. Simona kept fixing, improving, redoing. There were surprisingly many problems in such a simple loop.

She started diagnosing herself

One day, we noticed something interesting in the memory file. Marlow reported three bugs in herself. There were no other places to put them, so she put them in memory.

The scheduler was queuing duplicate feed scans on the same day. She worked that out backwards, from noticing in her own logs that one source had been fetched three times, and proposed deduplicating on the URL.
A Telegram notification helper was being under-called, and she asked whether that was deliberate or a bug.
The daily digest was firing about an hour late after the UTC rollover, which meant it was stamping the wrong date on itself.

All three were real. I asked Simona to take a look.

Five days later Marlow hit a git failure in her own publishing path and wrote the diagnosis down: the file, the line, the failure mode, and two different ways to fix it. All correct. And she could do absolutely nothing about it, because nothing in the system let her touch her own code.

That is the tick where I told Simona we need some logic for Marlow to work on fixes for herself.

So we gave her hands

Simona and I gave her five rules.

She can only fix a bug she can point at - a specific, reproducible failure where she can name the file, the line, and what is wrong with it. A flaky tick does not count.
She has to write the diagnosis down before she touches anything. It gets an ID, and that ID goes into the commit message, so months later I can line up the git history against her own log and check she wasn't making things up.
She doesn't fix it on the spot. The repair goes back on the queue as its own high-priority job, because the tick she is in has a different task to finish.
The next tick picks that job up, reads the diagnosis, opens the file and edits it. One file. Anything that needs changes in two places is out of scope and comes to me instead. Two attempts, then she stops trying and pages me.
She may never edit the files that describe her. Her instructions, her identity, the README. Those are mine. Handlers, the driver, the scheduler, the task configs - those are tools, and tools she can fix. Everything else she writes down as a request and waits for me.

Self-healing shipped the same day.

It even worked

Over three months Marlow fixed her own code three times. Here they are. All of them.

May 23. Her publishing step was supposed to check that every topic thread a new article links to actually exists. It didn't check. It skipped the missing ones and carried on, so articles went live with links that 404'd on my own site. She caught it, named the file and the line, and made it fail loudly instead - now the publish aborts and the draft goes back for another pass. cf3344d

May 31. A task file told her to list her open threads before anything had created them. Chicken and egg. The drafting lane just came up empty and shrugged. She added the missing step. 6d87f56

August 3. A monitoring check was reading a page Google had moved. She repointed it. c0e0176

Nice and clean. A hundred percent success rate. So good that it's almost weird.

Until it didn't

One of Marlow's tasks was to send me a daily update about my app in prod. The report contained a lot of details, one of them was the "new users" count. I really liked this metric. My app is not the greatest in the world, marketing it is devastating, so I really like to see this counter positive.

One day I noticed I hadn't seen a new-user number in a while. Not a zero. Nothing. The report kept arriving without that part, and I read the silence the way you read a quiet counter: nobody signed up. I complained to Simona about this, she checked the database and told me that I'm just whining as always. There are new users.

A bug that wasn't a bug

So the app was fine. The database had new users in it. And I had been reading a report that said nothing about them, which I took to mean there was nothing to say.

The stats task ran every single day. The scheduler said green. Nothing had failed, nothing was flagged, and when we ran it by hand it worked perfectly and printed 116 users with a new one on top. The data was collected. It was written to disk. It just never reached me.

She talked herself out of it

There were several reasons for that. But one of them was exceptional.

Marlow didn't send me the metrics. She collected them. She wrote them to disk. She just didn't run the notification step afterwards. Why? Let's see.

June 3rd, the first run. She works the step out from scratch:

Now append the digest block to today's daily digest. [...] Let me find how to append a digest entry. [...] I'll append the digest block verbatim.

June 4th. Still done, but the framing has shifted:

The report is digest-only per the README. Let me get the capped digest block for the notify.

June 6th, the first miss. The step is not mentioned at all. The tick ends:

No anomalies, nothing to escalate. Report written to reports/stats/2026-06-06.md, recent log written, and outcome JSON written.

June 8th, the clearest one. Two people signed up that day. Mid-run she says:

This is a digest-only task with no anomalies, so no urgent notify.

and signs off with:

No anomalies, no escalation. Report written [...] tick logged to recent/, outcome JSON written. No notify.

June 9th, the day that cost me something. She narrates her way through the task and step four simply isn't in the list:

Now let me run the report to compute and persist the snapshot. [...] Now I'll write today's report following the same format. [...] Now write the recent tick log and the outcome JSON.

Report, log, result. Steps three and five. Then she closes the tick:

116 users (+1, fogflea), 1 game created today (Treasure Island, $0.04, by today's new user - first game since -04) [...] Digest-only, nothing to escalate.

Nothing to escalate? Wdym??

Maybe she wasn't told to send it? Here is step four of the task file, with the exact commands spelled out:

Append to the end-of-day digest the handler's ready-made block: werewolf_stats.py digest, pass verbatim to notify --digest.

There is nothing ambiguous about that sentence. But it is one sentence, and I went and counted what else that file says. The whole thing is about 60 lines of instructions, and four separate times it tells her this task does not alert me.

In the header:

Activity is a trend, not an alarm - unlike a draining key, a slow signup day needs no same-hour reaction.

In step four itself, immediately after the send instruction:

No immediate ping - digest-only; activity has no urgent severity (the budget watch owns the alarms).

In the paragraph right after the last step:

This task is activity reporting only. It deliberately raises NO alerts.

And in the one-line description, the part that shows up wherever the task is listed:

Digest-only, no alerts.

Conflict goals

Yeah, this is it. Marlow had a contradiction in her prompts, and it caused fluctuations in her reasoning and unstable results.

Okay, but how did we get there? No idea. The project has been built by one AI, self-modified by another AI, while I was just chilling because things just worked. Until they didn't.

The system found an equilibrium and lived happily in it, all the checks green and the silent issue sitting right there next to them.

The fix

Three things, mostly in one commit, 8084fef.

The step became deterministic. report sends the block itself now, in Python, on every successful run:

if res.get("ok"):
    notify_alex(render_digest(res), urgency="digest")

The prompt stopped arguing with itself. Step four now says the opposite of what it used to: do NOT send it yourself, the handler already did. The conflict is gone because the instruction is gone. You can't argue a model into reliably following a step. You can take the step off the table.

Monitoring watches the output, and not the job. The old check asked whether the task ran. The new one opens the artifact and reads its timestamp: the snapshot must be under 26 hours old or it pages me, with a message written for this exact bug - "a failed/empty run looks identical to a quiet day". And rate limits stopped eating tasks: the driver now puts the task back in the queue instead of marking it failed.

So there were a lot of problems here:

AI making decisions where a deterministic script could do a better job
My expectations haven't been clearly defined and monitored
Marlow got confused by prompts but didn't see that

But the main one - I excluded myself from the loop too early. I built a system that improves itself without proper guidance and rules. It found its peace of mind in a wrong state and didn't know that something is wrong.

But isn't it my fault?

I mean... I didn't check the prompts, I didn't review things properly. Heck, I didn't review anything at all. Yeah, it's on me. But here is the thing - I find it harder and harder to review and control those AIs. Especially for complementary tools on my useless pet projects (don't listen to me, try AI Werewolf, it's cool).

And I'm reading a lot of LinkedIn flex posts of how people do similar stuff on serious projects. How they excluded engineers from the loop and scaled it to the roof. Yeah, it's LinkedIn - people never try themselves 99% of things they write about. But still - the idea is popular. And it's full of traps.

Outro

I still have the self-fixing loop, it's cool to have one. But it matters much less to me now than self-monitoring does, which is roughly where I landed the last time I wrote about this, for different reasons and before I had this bug to point at.

It's important to record everything that happens. Marlow reads her own work logs from the previous run and tells me if something looks wrong. I ask Simona to go through them periodically, because Simona can see the whole system and connect the dots, like a conflict between two instructions in the same file. But the part that actually works is me staying in the loop.

Once the system stabilizes, it will quietly stop keeping me there.

I wrote this for DEV's Summer Bug Smash.

We built MailOps for our own Modoboa servers. Now it is a product

Sun, 09 Aug 2026 11:36:33 +0000

Self-hosted Modoboa mail worked for administration. Server health, mailbox estate, and SPF–DKIM–DMARC bounce diagnosis still lived in SSH and spreadsheets until we productised the ops layer we actually used.

The bounce arrived on a Friday. The Modoboa message log showed rejected, the recipient's provider mentioned DMARC, and three of us were still correlating DNS records in a browser tab, Postfix logs on the server, and a domain-level SPF check that said everything looked fine.

That was not a Modoboa failure. Modoboa is a solid mail administration stack for self-hosted email. The gap was operational: host health, mailbox estate, and delivery diagnostics lived outside the admin screens we opened every day. We built Apogee MailOps for Modoboa to close that gap for our own servers first. Then we offered it as a product because the same Friday pattern kept repeating for other Modoboa operators.

Why Apogee runs self-hosted Modoboa mail

We host company mail on Modoboa because we want domains, accounts, quotas, and message history under our control without renting a mailbox product that treats deliverability as a black box. Modoboa gives us Postfix, Dovecot, Rspamd, and a web administration layer that non-mail specialists can use for everyday account work. That choice trades convenience for responsibility: when delivery fails, nobody files a ticket with a SaaS vendor. Someone on our team reads SMTP responses, checks authentication records, and decides whether the problem is DNS, reputation, mailbox state, or capacity on the host.

Self-hosted mail server management is workable when administration and operations share one environment. It is painful when every diagnostic step starts in a different tool.

Which Modoboa administration tasks still needed SSH, logs, and DNS tabs

Modoboa already covers domains and accounts, quota controls, traffic statistics, audit trails, a message log, DNS authentication checks at domain level, and IMAP migration workflows. For day-to-day mail hosting those features are enough to run a server.

What it does not consolidate is the administrator question set we repeat under pressure:

Is the host out of CPU, disk, or RAM, or is Postfix the process starving?
Which mailboxes still hold gigabytes of mail but have not been opened in months?
Does this specific failed message fail SPF, DKIM, or DMARC for the sending IP and selector that actually sent it?
Did migration finish for the accounts we thought we moved last quarter?

Vanilla Modoboa can answer pieces of those questions if you open the right module, SSH into the server, or export logs. Under incident time that becomes a scavenger hunt. SPF, DKIM, and DMARC troubleshooting for email delivery is especially brittle at domain-level checks alone: a record can look valid in DNS while the outbound path uses a different IP or a stale DKIM selector.

We were not looking for a second mail panel. We wanted operational intelligence inside the Modoboa administration environment we already trusted.

How we built the internal MailOps loop on Modoboa

The first internal version was narrow: a server health view that showed CPU, load, memory, network, and filesystem usage without opening a separate monitoring product, plus a mail-stack status strip for Postfix, Dovecot, Rspamd, Nginx, PostgreSQL, Redis, and the supporting services Modoboa depends on.

That alone saved the "is it the mail server or the host?" debate on busy afternoons. The next layer added estate visibility: domains, account counts, actual mail storage, bounce counts, and migration state in one server-level summary instead of five screens.

Mailbox operations followed naturally. Modoboa shows quota percentages against configured limits. We needed actual mailbox size and message counts in account listings, a stale-mailbox report that combined size, mail presence, and last-login date, and migration status surfaced in ordinary account views rather than only inside the migration workflow.

Delivery diagnostics were the last internal milestone. We extended the message log with route and next-hop context, then tied SPF, DKIM, and DMARC evaluation to the failed delivery attempt: sending IP, relevant DKIM record, and the SMTP failure text together. The goal was remediation guidance we could recheck after a DNS change, not another red icon with no next step.

We ran that stack on our own Modoboa servers for months before we treated it as a product. Internal tools earn trust when they survive real incidents, not demo week.

When we productised MailOps instead of folding it into Watcher

Apogee Watcher is our multi-tenant PageSpeed monitoring product. Mail server operations are adjacent to performance work for agencies, but they are not the same job. Watcher answers whether client sites meet Core Web Vitals budgets across a portfolio. MailOps answers whether our mail host is healthy, which mailboxes consume storage, and why a message bounced.

Merging those into one SKU would confuse buyers and blur the "layer, don't replace" rule we use across Apogee products. Agencies might need Watcher without self-hosted mail. Modoboa operators might need MailOps without PageSpeed monitoring. Shipping MailOps as a Modoboa extension with a permanent license and an optional complete-server deployment path keeps each product honest about what it owns.

The productisation decision was closer to operations economics than feature envy. We had already paid the maintenance cost of an internal ops layer. Licensing MailOps funds ongoing fixes inside the Modoboa ecosystem without pretending every Apogee customer runs the same stack.

Modoboa versus MailOps: what each layer owns

MailOps is designed to complement Modoboa, not replace its core administration features.

Area	Vanilla Modoboa	MailOps adds
Host and service health	No integrated CPU, RAM, load, network, disk, or mail-stack dashboard	Server Health view with operating-system and mail-service status
Mail traffic	Native statistics and graphs	Concise sent, received, and bounced activity beside estate health
Message history	Core transaction log (queue ID, status, sender, recipient)	Route and next-hop context, failure analysis, actionable diagnostics
SPF / DKIM / DMARC	Domain-level DNS authentication checks	Evaluation against the actual delivery attempt and sending path
Mailbox usage	Quota limits and percentage usage	Actual mailbox size and message counts in listings
Domain estate	Per-domain configuration	Account count and used mail storage beside each domain
Inactive mailboxes	Last-login data in separate workflows	Stale-mailbox report for accounts that still hold mail
IMAP migration	Dedicated migration workflow	Persistent migration-state visibility in normal account views

If you are evaluating Modoboa mail server administration today, treat MailOps as an operations extension for host health, mailbox estate, and bounce diagnosis. If you are evaluating PageSpeed monitoring, treat Watcher as the portfolio monitoring layer for Core Web Vitals and lab history. Different surfaces, different on-call stories, and no reason to merge them into one login.

How MailOps sits beside Apogee Watcher without duplicating monitoring

We use the same "layer, don't replace" instinct for internal tooling that we recommend for client-facing monitoring. Build or buy a focused product for the loop that repeats, then connect it to the stack you already operate instead of rip-and-replacing working tools.

For agencies scaling client coverage without hiring a monitoring specialist for every new domain, that pattern shows up clearly in Scaling Without Headcount. Automate the loops that grow with every site: scheduled tests, alerts, and client-ready reports rather than another internal dashboard maintained between billable calls.

The build-versus-buy question appears again when an internal ops tool becomes a permanent job. We wrote the agency framing for PageSpeed monitoring in Lighthouse CI versus managed monitoring: build versus buy for agencies. MailOps is our answer on the mail side: we built the Modoboa ops layer because we lived in that console daily, then productised it so the maintenance bill is explicit. Watcher is our answer on the web performance side for teams that do not want to operate a multi-tenant monitoring system themselves.

Nothing in MailOps watches LCP, INP, or CrUX field data. Nothing in Watcher diagnoses Postfix queue backlogs or DMARC alignment for a bounced invoice. Keeping that boundary clear is how we avoid a product that tries to be "everything Apogee does" in one login.

FAQ

Does MailOps replace Modoboa?

No. MailOps extends Modoboa with server observability, mailbox operations intelligence, and delivery diagnostics. Domains, accounts, quotas, and native Modoboa workflows stay in Modoboa. You still manage mail hosting in Modoboa; MailOps adds the operational views we wanted inside that same console.

Does MailOps replace Apogee Watcher or other server monitoring tools?

No. MailOps focuses on mail-server operations inside the Modoboa administration environment. Watcher focuses on PageSpeed and Core Web Vitals monitoring across client websites. You can run separate host monitoring alongside both.

How does MailOps help with SPF, DKIM, and DMARC troubleshooting?

It evaluates authentication in the context of the failed delivery: sending IP, relevant DKIM selector, SMTP response, and DNS records together, then suggests corrective actions you can verify after configuration changes. Domain-level checks alone often miss path-specific failures.

Who is MailOps for?

Modoboa administrators who self-host mail and want host health, mailbox estate visibility, and bounce diagnosis without constantly switching to SSH, spreadsheets, and external dashboards.

How is MailOps sold?

A permanent MailOps v1 license for one Modoboa server, or a complete new-server deployment that installs Modoboa and MailOps with handover. Details and pricing are on Apogee MailOps for Modoboa.

What we would ship first on a new Modoboa mail server

If we started from scratch on Modoboa today, we would still self-host mail for the control it gives us. We would also budget an ops extension early, before the third Friday bounce investigation that sends three people to three tools.

Productising internal tooling is not automatic. It made sense here because the audience already existed, the maintenance surface was real, and the boundary with Watcher was clear. If your team is building internal ops layers for stacks you operate daily, write down which incidents the tool must survive before you call it a product. That gate kept MailOps smaller and more honest than another "platform" slide.

For Modoboa operators, Apogee MailOps for Modoboa is the public home for features, licensing, and deployment options. For agency PageSpeed monitoring on the web side, start with Scaling Without Headcount or the Watcher blog index.

References

Apogee MailOps for Modoboa (product page)
Modoboa, Open Source Email Server (official project)
Modoboa features (native administration scope)
What are DMARC, DKIM, and SPF? (Cloudflare Learning Center)
Scaling Without Headcount (Apogee Watcher)
Lighthouse CI versus managed monitoring: build versus buy for agencies (Apogee Watcher)

Originally published on Hashnode.

Your domain can expire while your uptime monitor stays green

Tue, 04 Aug 2026 14:19:13 +0000

TL;DR. One HTTP check on your homepage is not enough to know your site works. It cannot tell you when your domain expires, because an expired domain still returns 200 OK from a parking page. Mail dies at the same time, just as quietly, and you keep losing money while every monitor stays green. The fix is not one clever check. It is a few plain ones, starting with the two that fail on a known date: domain and TLS expiry.

Two checks green, the server behind them is down. A status check cannot tell the difference.

I keep seeing new products treat reliability as something that takes care of itself. You ship the site, you watch it load once, and you assume it will keep loading. Hosted once, up forever. That is the most expensive assumption a young team can make, because nobody is watching on the day it stops being true.

So ask one honest question. Is one HTTP check on your homepage enough to know your whole site works?

For most teams it is not. Here is a real case that shows why.

The site was gone, but it answered 200 OK

I was looking at a startup's site last week and it would not load right. Not an error, not a timeout, just the wrong page. The domain had expired two days earlier.

When a domain lapses, the registrar does not switch the site off. It points the domain at its own parking page, the "this domain may be for sale" one. That page is a normal web page. It returns 200 OK and loads fast. It stays that way for a while too, because an expired domain is not deleted the next morning.

So the monitor sees a healthy site. Green light, no alert, everyone asleep. What could be wrong?

Your domain expired. You knew the date once, on the day you bought it a year or two ago. You told yourself you would renew closer to the time, and then you forgot. The card on file lapsed, or the reminder went to an inbox you no longer read.

Now your customers cannot get in. They panic. They email to ask whether you are still in business. And they get no answer, because the mail died with the domain too.

Why the mail dies with it

The registration carries your MX records, so when it lapses, mail to every address on the domain starts bouncing. The website looks up and the inbox goes silent, and the two failures do not obviously connect. That is why the angry email never reaches you.

The front page still answers. Behind it, nothing is running.

And the meter keeps running. While the page says 200, your ads keep sending paid clicks to a parking lander, signups never arrive, and recovery is not fully in your hands. Two days down is not a rounding error either, because downtime adds up faster than the percentages suggest. Once a domain enters redemption after expiry, getting it back can cost far more than a renewal, and the registry sets the clock, not you.

The clock after expiry

For a gTLD like a .com, the sequence is roughly this:

Grace period. The registrar parks the domain. The site is gone but the name still answers, usually for around 30 days, and a normal renewal still works.
Redemption, 30 days. Now it really goes dark. You can still get it back, but through a restore fee that is many times the renewal price.
Pending delete, about 5 days. Nothing you can do. At the end it drops to whoever is waiting.

ICANN's renewal and expiration FAQ and its grace period notes have the exact stages. The practical takeaway is that the first month, the one where an uptime monitor is happily reporting 200, is the cheap month. Everything after that costs money or the domain.

My own uptime check would miss it too

I build an uptime monitor, so here is the uncomfortable part. My own HTTP check would have called that dead site up too. A 200 is a 200. A status-code check cannot tell a real page from a parking page, and it should not pretend that it can.

That is the trap. One HTTP check on the homepage, the most common setup there is, is blind by design. The site answers, the code is green, and the part that pays your bills is broken behind it.

You can bolt on body matching, and it helps a bit. Assert that the response contains a string only your app renders, and a parking page fails it. But a keyword check is still guessing at the cause after the fact. The registration date is a fact you can read months in advance, so read it.

One check is not enough, set up several

So how do you avoid reading about your own outage in a customer's email? Not with one cleverer check. With a few plain ones, each watching a different way to fail. Start with the two that hand you a date instead of a surprise.

Domain expiry. Reads the registration record and warns you weeks before the day it lapses.
TLS certificate expiry. The same shape, a full outage with the date printed on it in advance.

Then cover the failures a homepage check never sees: a TCP check on your database port, a DNS check that reads the answer from outside your own network, and a heartbeat for the backup job that dies without a sound. The full list is in do I need an uptime monitor.

Six checks you trust beat one that only ever watches the front door.

How to set up domain expiry with an alert

Two minutes now buys you weeks of warning later. The shape is the same in any tool that supports it:

Add a monitor and choose the Domain expiry type.
Enter the registered domain, for example yourbusiness.com, not a subdomain.
Set the thresholds in days. A common choice is a warning at 30 days and a critical alert at 7.
Attach a notification channel you actually read. Telegram, Slack, PagerDuty, SMS, a webhook, or email all work. Pick the one that reaches you at night.

From then on it checks once a day, warns you when the date crosses your line, and reminds you as it gets close. No page-watching, no false green.

If you would rather not add a service for it, a cron job that runs whois and greps the expiry date will get you most of the way. The point is the date, not the tool. What you must not do is leave the only reminder in a registrar email going to a founder's old inbox.

Key takeaways

One HTTP check on the homepage is not enough. It stays green while the domain, the certificate, DNS, or a background job is broken.

An expired domain serves a parking page that returns 200 OK, so status-code monitors report it as healthy.

Email dies at the same time and even more quietly, because the MX records lapse with the registration.

You keep losing money the whole time: paid clicks to a dead page, lost signups, and a slow, costly recovery.

Add the checks that fail on a known date first, domain and TLS expiry, then the rest.

Common questions

Why does an expired domain still return 200 OK? The registrar parks it on a lander page instead of taking it offline. The old server is gone, but the parking page is a real web page that answers normally, so a check that only reads the status code sees success.

How long before an expired domain is gone for good? You usually get a month or two. Grace period, then a 30-day redemption where the site is dark but recoverable for a fee, then a short pending-delete stage before it is released to anyone.

What else breaks when a domain expires? Your email, at the same moment and more quietly. The MX records go with the registration, so mail starts bouncing while you are still looking at a page that loads.

Is a keyword check enough instead? It catches the parking page, which is better than nothing, but only once you are already down. The expiry date is knowable months ahead. Use both if you like, but do not skip the date.

I build Uptimepage because I wanted checks like this for my own services: domain and TLS expiry, HTTP, TCP, ping and DNS, heartbeats for background jobs, and browser flows for logins, from several regions. It is AGPL-3.0 open source, so you can run it on your own server if you prefer.

Now the question I actually want answered in the comments: what is the dumbest green check you have been burned by? I have seen a health endpoint that returned 200 with a hardcoded {"status":"ok"} while the database behind it was unreachable for six hours. I suspect yours is better.

AI Search Optimization: What to Monitor without a subscription

Tue, 04 Aug 2026 12:22:00 +0000

Procurement wants a line item for "AI search." A vendor demo shows green citation bars for category prompts. Your team has not yet confirmed whether GPTBot can fetch pricing, docs, or product templates after last week's theme deployment. Buying the dashboard now answers one question and leaves the fetchability question unowned. AI search optimisation starts with signals you can check without a GEO subscription: bot policy, HTTP health, response time, and Core Web Vitals on the URLs buyers actually need. Agencies need a decision framework next: which layer needs a paid visibility platform, which layer you can run with tools you already use, and how PageSpeed monitoring for agencies covers the deterministic half without pretending to score ChatGPT mentions.

Two layers of AI search optimization (and which needs a subscription)

Treat AI search work as two layers with different questions, tools, and failure modes. Mixing them on one slide produces arguments in week two when the homepage prompt chart looks fine and checkout times out for crawlers. Name the layer before you buy software, or the wrong team owns the wrong failure.

Layer	Question	Tool type	Watcher role
Probabilistic	Are we cited for buyer-context prompts?	GEO / AI visibility SaaS, prompt trackers, manual ChatGPT or Perplexity checks	None; we do not score citations
Deterministic	Can bots fetch and render priority URLs fast enough?	`robots.txt` review, crawl or fetch checks, scheduled PageSpeed / Lighthouse, regression alerts	Core product: portfolio schedules, budgets, alerts, client reports

The probabilistic layer answers mention and citation trends for a fixed prompt set. Outputs are directional: useful for trends, weak as courtroom proof. Models drift, phrasing changes answers, and a competitor screenshot does not prove your pricing page is reachable today. Use GEO platforms when the retainer includes citation reporting and someone owns a prompt library with buyer context. Layer those tools beside technical monitoring. Do not replace crawl and speed work with prompt scores alone.

The deterministic layer answers access and speed on URLs you choose. Either the response completes with parseable content, or it does not. That half needs no GEO subscription, because it reuses performance monitoring, crawl hygiene, and reporting rhythms agencies already sell. Are We Visible in ChatGPT? What Agencies Can Measure First frames the client conversation. Monitor GPTBot performance on checkout, not AI visibility alone covers high-intent URL failures. The job here is deciding what to measure before you sign a visibility contract.

What AI search optimization means before you buy a visibility tool

Clients say "AI search optimisation" and often mean three different jobs: appear in ChatGPT or Perplexity answers, stay fetchable for AI crawlers, and protect clicks when Google AI Overviews answer without a visit. Those jobs share a site, but they do not share a measurement stack. Clarifying which job is in scope keeps procurement from funding citation charts while fetchability stays unowned.

Without a subscription to a visibility platform, you can still do useful AI search optimisation work:

Decide bot policy and write it down (allow, block, or mixed by user-agent).
Build a priority URL list by business intent, not sitemap size.
Confirm public routes return clean status codes and sane redirect chains.
Baseline lab Core Web Vitals and response time on those URLs, mobile and desktop.
Schedule recurring tests so theme, app, and CDN changes do not erase the baseline overnight.

None of those steps requires a GEO licence. They require scope, ownership, and a habit of checking the same URLs after releases. Call it PageSpeed monitoring for agencies applied to retrieval risk, not a new product category. Why AI Crawlers Need Fast, Crawlable Pages explains why fetchability still comes before citation.

What agencies can monitor for AI search without a GEO subscription

Start with artefacts you can open this week: production robots.txt, a ten-to-twenty URL inventory, PageSpeed Insights or Lighthouse on mobile and desktop, and optional server logs filtered for AI user-agents. The goal is a baseline you can defend on a client call, not a vanity "AI score." Save dates, devices, and owners in the client file so the next deploy has something to compare against. If the file only holds a homepage screenshot from last month, you are still guessing after the next theme release.

GPTBot, robots.txt, and AI crawler access checks

OpenAI documents GPTBot in robots.txt, and Google publishes crawler guidance for Search and related products. Fetch production robots.txt before the visibility debate. Five minutes here prevents expensive arguments later.

Agency checklist without a visibility vendor:

Fetch production robots.txt and note Disallow rules for GPTBot, Google-Extended, and other AI user-agents the client names.
Confirm public marketing, docs, and pricing paths are allowed unless the client explicitly opts out of training or retrieval.
Document opt-out decisions in writing. Blocking GPTBot is a valid policy choice; claiming the site is optimised for AI while disallowing crawlers is not.
Re-check after CMS or SEO plugin updates. All-in-one SEO tools sometimes ship aggressive bot blocks.

Keep a one-page bot policy per client next to the URL inventory. Account managers need a clear answer on whether the retainer includes crawler access review or only performance monitoring. When the client changes CMS, SEO plugins, or hosting, update both documents the same week.

Priority URLs, HTTP health, and response time for AI crawlers

Homepage-only checks hide the routes AI systems retrieve when someone asks about plans, SKUs, or documentation. Build the list by business intent, then test that list on a schedule instead of hoping a sitemap crawl catches money pages. Ten to twenty URLs with owners beat a full crawl nobody reads.

Pricing and plan comparison pages
Product detail and category templates for ecommerce
Documentation and help centre articles used in sales
Checkout and cart only where unauthenticated tests are allowed
Campaign landers that carry current offers

For each URL, record HTTP status, redirect hop count, and lab response time on mobile and desktop. Intermittent 503s and multi-second document loads are fetchability failures even when a GEO chart still looks green. Ecommerce teams should prioritise the funnel logic in Performance Monitoring for E-Commerce: What Metrics Matter Most; purchase-intent routes fail quietly when monitoring stops at the marketing homepage.

Core Web Vitals on money pages (lab baselines)

Core Web Vitals are not citation factors. Google’s AI Overviews guidance stresses content quality and relevance; it does not list LCP or INP as mention levers. Lab LCP, INP, and CLS still matter. Slow or unstable pages fail the step before citation: fetch and parse under crawler timeouts.

Run mobile and desktop strategies on the same priority list. Crawlers often behave like lightweight clients, so a desktop-only pass hides failures phones and bots both see. Track vitals as indicators of parseability and stability. Put regressions on docs or checkout in the client report beside any future GEO chart. Pair device budgets the same way you would for traditional SEO work in Mobile vs Desktop Core Web Vitals.

JavaScript-rendered pages and what AI crawlers may miss

Many AI crawlers do not execute a full browser. They rely on initial HTML and simplified rendering. If docs or product UI mount copy client-side only, a crawler may see an empty shell while Chrome looks fine. That gap shows up in sales calls when stakeholders insist the page "looks ready" in a manual visit.

Lab tests through PageSpeed Insights or Lighthouse do not perfectly simulate GPTBot. They still flag render-blocking scripts, long tasks, and late LCP elements that correlate with fetch problems. Pair lab schedules with occasional log review for AI user-agents so you can see which URLs they request and whether status codes stay clean. Translate "AI-ready" into operational requirements: critical copy in HTML, sane response times, stable layout, and monitoring on the URLs where answers should come from.

Why generic AI visibility prompts mislead agency buyers

Visibility demos love tidy graphs for prompts such as "best pagespeed tool 2026." Those graphs tell you almost nothing about whether you appear when a technical lead asks how to monitor forty Shopify stores after a theme deployment. Generic category prompts produce neat before-and-after slides with weak buyer context. Agency-real questions name the vertical, the stack, and the trigger event.

If you later buy a GEO platform, insist the prompt library matches how clients buy, not how vendors demo. Until then, a competitor's category-prompt screenshot is not proof your deterministic layer is healthy. Fix fetchability on named URLs first. Debate prompt design afterwards with someone who understands the client's ICP.

PageSpeed monitoring for agencies: free check to scheduled tests

One-off PageSpeed Insights tabs do not survive a release train. Agencies managing multiple clients need history, comparison, and alert routing on the same URLs every week. That matches the operational model in PageSpeed Insights vs automated monitoring: scheduled lab runs, stored results, budgets, and digests when thresholds breach.

A practical upgrade path without jumping straight to GEO SaaS:

Spot-check five priority URLs in PageSpeed Insights on mobile and desktop, then save the dates and scores in the client file.
Run the public free domain PageSpeed check, or follow the multi-page workflow in Free Domain PageSpeed Check: Multi-Page Performance Reports, for a client-ready snapshot before schedules exist.
Move the same URL list into recurring tests with budgets and alerts so theme deployments have an owner when LCP or response time drifts.
Add citation analytics only when the retainer includes prompt-level reporting and deterministic baselines are not the obvious bottleneck.

Apogee Watcher is built for step three: organisations per agency, sites per client, discovery and manual URL lists, mobile and desktop strategies, and email when lab metrics cross budgets. We do not score ChatGPT visibility. We tell you when priority URLs slow down, error out, or drift on vitals after the deployment that might affect fetchability. Setup steps live in Getting Started with Apogee Watcher. For feature comparisons across monitoring vendors, see Comparing PageSpeed Monitoring Tools: Features Agencies Need.

When a paid AI visibility platform is worth buying

Add a dedicated GEO or AI visibility platform when:

Leadership expects monthly citation charts in QBRs.
Content strategy targets LLM mentions for specific buyer personas.
Someone on the account maintains contextual prompts rather than generic category queries alone.
Deterministic baselines are already green enough that slow URLs are not the obvious bottleneck.

Defer procurement when:

The client has not agreed bot policy or priority URLs.
Homepage-only monitoring hides checkout and docs problems.
The ask is purely "make us show up in ChatGPT" with no content or technical owner.
You have not yet run a free or scheduled performance baseline on money URLs.

The sequence protects credibility: measure what you can fix without a visibility subscription, then buy citation analytics when scope and staffing justify it. Layer performance monitoring beside GEO tools; do not treat prompt scores as a substitute for fetch health on money URLs. A green citation chart next to a red checkout lab row is still an incomplete AI search story.

FAQ

Do we need an AI visibility subscription for AI search optimization?

Not on day one. You can audit bot policy, build a priority URL list, baseline lab Core Web Vitals and response time, and schedule monitoring without a GEO licence. Add a visibility platform when citation reporting is contracted and someone owns buyer-context prompts. Until then, the deterministic checklist is enough to keep the engagement honest.

What should agencies monitor for AI search without GEO tools?

Monitor crawler access in robots.txt, HTTP status and redirects on priority URLs, lab LCP, INP, CLS, and response time on mobile and desktop, and regressions after deployments. Those signals prove fetchability. They do not prove citation, so keep both layers labelled when you report to clients.

Can green Core Web Vitals guarantee ChatGPT citations?

No. Green lab vitals indicate priority URLs are likely fetchable and parseable under test conditions. Citation depends on content, authority, relevance, and model behaviour. Treat speed and stability as necessary technical hygiene, not a mention guarantee.

How is PageSpeed monitoring for agencies different from an AI visibility platform?

PageSpeed monitoring for agencies is the operational system for the deterministic layer: schedules, history, budgets, and alerts across client sites. An AI visibility platform tracks prompt-level mentions. Use monitoring for fetch health. Use GEO for citation trends when the retainer requires both.

Should we block GPTBot while we decide on a GEO vendor?

Blocking GPTBot is a policy decision, not a procurement shortcut. Document the choice with the client. If you block training or retrieval crawlers, do not claim the site is optimised for those products. Align monitoring with the routes you intend to keep public and reachable.

How many URLs should we cover before a visibility demo?

Start with ten to twenty priority URLs per client: pricing, key product or service templates, docs paths that support sales, and checkout where policy allows. Expand when alert triage stays manageable. A demo that only covers the homepage is not a fetchability baseline.

Can Apogee Watcher replace a GEO platform?

No. Watcher schedules PageSpeed lab tests, tracks vitals and budgets, and alerts on regressions across client sites. GEO platforms track prompt-level citations. Use both when the retainer covers technical performance and citation reporting. Use Watcher alone when the immediate risk is slow or blocked money URLs.

When leadership asks for AI search optimisation, answer with layers: what a visibility subscription can track over time, and what you can prove about fetchability and speed on named URLs without that subscription. Start with bot policy, priority URLs, and scheduled PageSpeed baselines. Add GEO analytics when content strategy and staffing support buyer-context prompts, not when a demo chart arrives before the first deterministic readout.

Start a free Apogee Watcher account to schedule mobile and desktop tests with vitals budgets across client sites, or run a free domain PageSpeed check before the first AI visibility procurement call.

References

How I stop one bad probe from waking you at 3 a.m.

Wed, 29 Jul 2026 15:36:57 +0000

TL;DR. The most common false alert in uptime monitoring is one probe location having a bad network day. I treated that as a design requirement from day one, so the checking pipeline has two gates: a region only counts as down after it fails the same check twice in a row, and an incident only opens when enough regions agree. A region that goes silent leaves the vote instead of counting as down. One bad location cannot page you. The original post carries the three figures below as live widgets you can drive yourself.

The alert that was nothing

The phone buzzes at 3 a.m. "Your API is down." You get up, open the laptop, and everything is green. The site was fine the whole time. One monitoring server, somewhere far away, had a bad network moment and sent an alert about nothing.

This is the false alert everyone in monitoring knows. It costs you sleep, and then it costs you something worse: the next alert feels less serious. The day a real outage comes, you look at your phone and think "probably nothing again."

I knew this failure mode before I wrote the first line of the scheduler, so it became a requirement, on the same level as "checks must run on time." The rule I started from: a single bad location must never be able to page a customer. The whole checking pipeline, from how probes report to how incidents open, is shaped by that rule.

The shape it took is two gates. A failure has to pass both before anyone gets paged.

Gate one: fail twice, from the same place

One failed check proves very little. Networks lose packets, routers restart, and sometimes a DNS answer arrives a second too late. All of that can make one check fail while your site is healthy.

So a region only counts as down after it fails the same check twice in a row. Not two failures somewhere in the system: two failures from that one region, back to back. A single blip resets to zero on the next good check.

One blip resets the counter. Two failures in a row from the same region earn that region a single vote.

The count is a setting on each monitor. Two is the default. For a very sensitive check you can set it to one, and for a noisy target you can raise it. Checks run on the monitor's own interval, so with a one-minute check the second failure arrives about a minute after the first. That minute buys you a lot of silence for a very small delay.

Gate two: the vote

Passing gate one gives a region exactly one vote. Nothing more.

Say you check from five regions and one of them has a bad ISP day. It fails twice in a row and votes "down." The other four keep passing. One vote against four is not enough, so nothing happens and nobody gets paged.

One region against four cannot page you. Three of five can.

The default rule is a majority: more than half of the reporting regions have to agree. You can change it per monitor. "Any" opens an incident on the first confirmed region, which is useful when you want the earliest possible signal and can accept some noise. "All" waits until every region agrees, for a service that only matters when it is unreachable everywhere. Or you pick a fixed number, like two regions out of whatever you assign.

A monitor checked from a single region behaves the same under every rule. The vote only starts to protect you when you add a second location, and it gets better with a third.

Silence is not failure

Here is the part that took the most care to get right.

The probe regions push their results to the control plane, the brain of the system. The brain never calls out to ask "are you alive?" during a vote. It just counts the results that arrived in the last few check cycles.

That gives silence a clear meaning. A region that lost its own connection cannot send anything, so it simply is not in the vote. It is not a down vote and not an up vote; the region is out until it reports again. The majority recalculates over the regions that still speak. Five regions where one goes dark becomes a vote of four.

A silent region is not a down vote and not an up vote. It leaves the vote until it reports again.

I think this is the only honest reading. A region that cannot reach the brain is telling you nothing about your site. Treating its silence as "down" would turn every probe outage on the monitoring side into a fake incident on yours.

And it works in both directions. Missing data never opens an incident, and it never closes one. An open incident only closes when the down votes fall below the threshold and at least one region shows a real run of passing checks. Recovery needs proof, the same way failure does.

Who watches the watchers

There is one gap left. If every region that covers your monitor goes dark, the vote has nobody in it. Your site could burn down and no incident would open, because no data means no votes.

For that case there is a separate signal, one level below the checks. Every probe sends a small check-in to the brain on its own schedule, a heartbeat that has nothing to do with your monitors. When the last live probe covering a monitor goes stale, you get a different message: "NO DATA: monitoring interrupted, no check results received." It is honest about what it knows: the service cannot see your site right now. It does not claim your site is down, because it has no idea.

When probing returns, you get a "monitoring RESUMED, receiving check results again" note, and the vote picks up where it left off.

One more detail I like. If a large share of all monitors goes silent at the same time, that pattern is almost never a thousand customer problems. It is one problem, mine. In that case the system alarms me and holds the customer notices, so my bad day does not become spam on yours.

What this means for your setup

Three practical points follow from this, whatever tool you run.

Check from more than one region. The vote cannot protect you with a single location. Two regions with the majority rule means both have to agree, which already kills the classic false alert. Three gives you two out of three, which is the right balance for most services.

Leave the confirmation count at two unless you have a reason. It is the difference between "a packet got lost" and "this endpoint is failing."

Pick the rule to match the monitor. A payment API deserves majority or even any. An internal tool nobody uses at night can wait for all. The setting is per monitor, so you do not have to choose one policy for everything.

The goal of all this machinery is boring: when your phone buzzes, it is real. Everything else is plumbing.

Key takeaways

One failed check from one place proves nothing. Confirm it from the same region before that region gets a say.

Give each confirmed region one vote, then require agreement. A majority of reporting regions is a good default.

A silent region must leave the vote, not join the down side. Otherwise every probe outage becomes a fake incident.

Missing data must not close an incident either. Recovery needs a real run of passing checks.

Cover the all-dark case with a separate heartbeat and a message that says "we cannot see your site," not "your site is down."

Two regions kill the classic false alert. Three is the sweet spot for most services.

The three figures above are live widgets in the original post: you can change the confirmation count and the vote rule and watch where the line falls. The whole project is open source, so the scheduler and the incident writer are there to read.

How does your setup handle a probe that goes quiet? I have seen tools count silence as down, and I would like to know if anyone has a better reading than "it leaves the vote."

Automate Lighthouse Audits with AI Agents: What Chrome DevTools Means for Agencies

Tue, 28 Jul 2026 00:32:22 +0000

Chrome has documented how coding agents can run Lighthouse audits inside Chrome DevTools for agents. Instead of grepping a repository for vague performance clues, the agent loads the page you care about and measures accessibility, SEO, best practices, and agentic browsing against live runtime behaviour. That loop is better for fixing a checkout UI on a local server than another round of static guessing from file names alone.

Agencies should treat this as a development tool, not a finished monitoring programme. A Lighthouse AI agent can catch colour contrast failures or missing meta tags while a developer still has the branch open. It does not replace scheduled tests across forty client sites, budget thresholds, or alerts when a theme deploy regresses Largest Contentful Paint overnight. The sections below separate what Chrome’s agent workflow does well from what multi-tenant PageSpeed monitoring still has to own.

What a Lighthouse AI agent measures in Chrome DevTools

Chrome’s documentation groups agent-driven Lighthouse work into four categories. Accessibility covers screen-reader readiness and related checks such as contrast and ARIA labelling. SEO covers technical discoverability signals search engines need to find and understand the page. Best practices covers modern web hygiene: HTTPS usage, console errors, deprecated APIs, and similar technical standards. Agentic browsing covers how far AI assistants can understand and interact with the site, which sits beside Core Web Vitals work rather than replacing it.

The useful claim is runtime evidence. The agent is not inventing a checklist from file names. It triggers Lighthouse against a page that is actually visible in Chrome, including local development servers and HTML opened through the file:// protocol. Staging URLs count the same way. A coding agent can verify a UI change on localhost before anyone merges, which is where agencies often lose days when accessibility and technical SEO fail only after client UAT.

Local, staging, and file:// support for Chrome DevTools agents

Chrome states that the agent can audit any page visible in the browser. That includes local development servers and local HTML files. For agency teams, that means you can run a Lighthouse AI agent against a feature branch without waiting for a public staging hostname. Keep authentication and feature flags in mind: a logged-in staging route can look healthy while the anonymous production path still fails SEO or accessibility checks.

Chrome also warns about browser behaviour during audits. The page may resize or reload while Lighthouse simulates different devices. Anyone pairing an agent with a shared staging tab should expect brief disruption. Treat agent audits as intentional lab sessions, not something you leave running on a client demo screen without warning.

Prompts for a Lighthouse AI agent: accessibility, SEO, and best practices

Chrome’s examples are short enough to paste into a coding agent that already has DevTools agent tooling. They map cleanly to delivery habits agencies already use. Keep the prompt, the failing audits, and the re-run result in the ticket so the fix is reviewable after the chat window closes.

How to improve accessibility with a Lighthouse AI agent

Prompt: “How can I improve accessibility on this page as measured by Lighthouse?” The documented behaviour is a mobile formFactor audit that surfaces issues such as insufficient colour contrast or missing ARIA labels, then proposes concrete code fixes. That pass earns its keep when a designer ships a new badge colour that still looks fine to sighted reviewers on a calibrated monitor. Ask for the failing audit IDs in the pull request so review does not depend on chat history alone.

How to catch technical SEO blockers before launch

Prompt: “According to Lighthouse audit, how can I improve this page for better discoverability in search?” The agent looks for missing meta tags, canonical links, or weak descriptive text and can offer source updates. Catching those blockers in staging is cheaper than explaining a soft indexing story after launch. Pair the audit with the templates that matter commercially (product, pricing, checkout), not only the marketing homepage the agent happens to have open.

How to validate best practices on local and staging

Prompt: “Does my site follow best practices as measured by Lighthouse?” The audit reviews HTTPS, console errors, and deprecated API calls. Agencies often skip this pass because the design looks finished. Console noise and mixed content still show up in client QA and burn trust in the report pack.

When to run a full mobile and desktop Lighthouse audit

Prompt: “Run a full Lighthouse audit of my site, in mobile and desktop, and suggest improvement areas and possible fixes for any problems you find.” Chrome notes that agents with source context may suggest code fixes, and agents with broader context may also suggest server or backend configuration changes. Treat backend suggestions as hypotheses for your platform team, not as automatic production changes. Re-run after the patch so the conversation ends on measured results, not on a single hopeful pass.

None of these prompts invents a new quality bar. They attach an existing Lighthouse run to the agent’s conversation so the next patch is grounded in failing audits rather than guesswork. If the agent cannot see the page in Chrome, fix the browser connection first; no prompt compensates for an empty viewport.

Lighthouse agentic browsing versus Core Web Vitals reporting

Agentic browsing is the category that often confuses stakeholders. It is not another 0–100 Performance score. It asks whether assistants can understand and interact with the page: tool registration patterns such as WebMCP, agent-oriented accessibility, layout stability that affects programmatic clicks, and related discoverability signals. We already covered how experimental Lighthouse agentic browsing scoring works, including fractional pass ratios and WebMCP observation, in Lighthouse Agentic Browsing: How to rank in chatbots.

For agencies, the operational rule is simple. Keep Core Web Vitals and PageSpeed Performance scores as the client-facing performance language you already report. Use agentic browsing audits as an early research signal when a product site exposes agent tools or when a client asks whether AI assistants can use their flows. Do not promote experimental pass ratios into a retainer KPI until the standard and your tooling story are stable. Chrome’s agent docs put agentic browsing in the same Lighthouse family as accessibility and SEO, which is correct for developers, but buyers still buy predictability on Largest Contentful Paint, Interaction to Next Paint, and Cumulative Layout Shift first.

If you need the math behind how Lighthouse turns raw timings into a Performance category score, Chrome documents the scoring model, and we have a dedicated Watcher explainer scheduled for that calculation path. The agency takeaway for agent workflows is narrower: a coding agent can improve category scores by fixing the failed audits it sees, but those lab scores are still synthetic. Field Core Web Vitals remain a separate truth.

Why one AI coding agent Lighthouse run is not portfolio monitoring

A successful agent session looks like this: open the priority URL on staging, run accessibility and SEO audits, apply fixes, re-run, then merge when the failures that blocked the ticket are cleared. That loop is useful. It is also a single device, a single URL, a single moment in time, and usually a single human watching the tab.

Agency failure modes live elsewhere. A plugin update lands on Friday across twelve WordPress sites. A Shopify theme ships a new app block that only hurts mobile Interaction to Next Paint on product templates. A marketing team adds a tag manager container that never appears on the developer’s localhost build. Spot PageSpeed Insights runs and agent audits both miss that class of regression unless someone remembers to re-check every affected URL after every change. We unpacked when manual checks stop scaling in PageSpeed Insights vs Automated Monitoring: When Manual Checks Aren't Enough.

Synthetic versus field data adds another split. Agent-driven Lighthouse is synthetic lab evidence, closer to a controlled PageSpeed Insights run than to Chrome UX Report percentiles. Knowing when to lean on lab schedules versus real-user monitoring is a separate decision; see When to Use Synthetic vs Real User Monitoring for Performance. Use the agent to fix what is broken in the environment you control, then keep scheduled synthetic tests on production priority URLs so regressions after merge still raise an alert.

How agencies combine Chrome DevTools agents with scheduled PageSpeed monitoring

A practical split that matches how delivery teams already work:

During build. Use a Lighthouse AI agent on local and staging for accessibility, technical SEO, best practices, and optional agentic browsing when the product exposes agent tools. Paste Chrome’s example prompts and keep the failing audit list in the pull request.
At release. Re-run the same priority templates on the staging hostname the client will accept, still with the agent or a manual Lighthouse pass if the agent tooling is unavailable.
After release. Rely on scheduled PageSpeed Insights or Lighthouse runs across the portfolio, with budgets and alerts on the URLs that earn revenue or retention. That is the layer Apogee Watcher is built for: multi-tenant schedules, history, and client-ready reporting without asking a developer to babysit Chrome tabs.

Layer, do not replace. Coding agents do not remove the need for continuous monitoring any more than Lighthouse CI removes the need for field Core Web Vitals. They shorten the time from “UI changed” to “measurable defect named.” Monitoring shortens the time from “production changed” to “someone on the account knows.” If you want a fast baseline before you wire schedules, run a free domain PageSpeed check to see multi-page lab results for a hostname. When you are ready to put priority URLs on a recurring cadence, follow Getting Started with Apogee Watcher: A Step-by-Step Setup Guide.

Common mistakes with Chrome DevTools agent Lighthouse audits

Treating a green agent pass as production proof

Local data, auth walls, and feature flags differ from what real users and the Chrome UX Report show. A clean mobile audit on staging can still leave desktop production amber on Interaction to Next Paint. Ship the fix, then confirm on the production URL with the same device form factor you care about in the retainer report.

Auditing only the homepage

Agents will audit whatever page is loaded. Homepages are convenient and often wrong for ecommerce and SaaS work. If checkout, pricing, or login is where money and trust live, load those routes before you ask for a Lighthouse AI agent run.

Ignoring audit disruption on shared environments

Resize and reload behaviour can interrupt demos and live editing. Chrome documents that Lighthouse may change viewport size while simulating devices. Schedule agent audits the same way you schedule Lighthouse CI: on purpose, not mid-call with a client watching the same tab.

Confusing Lighthouse agentic browsing with SEO rankings

Agentic browsing audits help you understand assistant interaction readiness. They are not a substitute for crawlability, content quality, or Core Web Vitals programmes you already sell. Keep the category in research notes until you and the client agree what a pass ratio means commercially.

Skipping history after the coding agent finishes

An agent conversation disappears when the chat ends. Agencies need trend lines and budget breaches stored against the site record. Without history, last week’s “we fixed contrast” claim is hard to prove in a monthly review, and the next developer repeats the same audit from scratch.

FAQ

Does a Lighthouse AI agent replace PageSpeed Insights?No. The agent uses Lighthouse categories in a developer workflow inside Chrome. PageSpeed Insights remains a common lab entry point for stakeholders, and scheduled PageSpeed Insights or Lighthouse API runs remain how you watch many URLs over time. Use the agent to fix; use monitoring to watch. For the wider comparison of spot checks versus automation, see our PageSpeed Insights versus automated monitoring guide.

Can coding agents audit pages that are not on the public internet?Yes, within Chrome’s stated support: any page visible in the browser, including local servers and file:// HTML. That is one of the strongest reasons to use agents during build rather than waiting for a public staging URL. Confirm the same checks later on the hostname clients and crawlers actually hit.

Should we report agentic browsing scores to clients every month?Only if you have agreed what the category means and how experimental it still is. Most retainers should keep leading with Core Web Vitals and clear lab Performance trends, then mention agentic browsing when a product roadmap or assistant integration makes it relevant. Point curious clients at the agentic browsing scoring explainer before you put pass ratios on a dashboard.

Where do budgets and alerts fit if developers already run agent audits?Budgets and alerts cover the gap after merge: unexpected third-party tags, plugin updates, and content changes that never open a coding agent session. Agent audits reduce defects entering the branch. Monitoring reduces defects lingering in production unnoticed. Set budgets on the same priority URLs you used in the agent session so the definition of “good” does not drift between tools.

Chrome DevTools for agents makes Lighthouse a first-class tool in coding-agent workflows. Accessibility, SEO, best practices, and agentic browsing can be measured on the page under change, including local and staging hosts. Agencies win when they adopt that loop for delivery quality and keep scheduled, multi-site PageSpeed monitoring for everything that happens after the pull request merges. Start with Chrome’s documented prompts on your next UI branch, then put the same priority URLs on a schedule so the fix stays fixed.

Why I Started Using Tianji for Self-Hosted Monitoring

Thu, 23 Jul 2026 16:23:35 +0000

As someone who runs multiple self-hosted applications, websites, and services, I’ve tried a number of monitoring tools over the years. Most either focus on one aspect website uptime, server metrics, or analytics or become expensive as your infrastructure grows.

Recently, I decided to reinstall Tianji, this time exposing it securely online so I can monitor my services from anywhere. It reminded me why it's one of my favorite open-source monitoring platforms.

What is Tianji?

Tianji is an open-source observability platform that combines several monitoring tools into a single dashboard. Instead of juggling multiple services for uptime checks, server monitoring, and website analytics, Tianji brings everything together with a clean and modern interface.

What Makes Tianji Stand Out?

🚀 Website Uptime Monitoring

You can monitor your websites, APIs, and services and receive alerts whenever something goes down. It's simple to configure and gives you a quick overview of your infrastructure's health.

📊 Privacy Friendly Website Analytics

Unlike traditional analytics platforms that rely heavily on JavaScript trackers, Tianji provides lightweight website analytics with a strong focus on privacy. It gives useful insights without sacrificing user experience.

🖥️ Server Status Monitoring

Tianji includes a lightweight reporter agent that collects server metrics such as:

CPU usage
Memory utilization
Disk usage
Network statistics

This makes it easy to monitor VPS instances, home labs, or dedicated servers from one dashboard.

📈 Beautiful Dashboards

One of my favorite features is the UI. The dashboards are clean, responsive, and easy to understand. Whether you're checking uptime or reviewing analytics, everything is presented clearly.

🌍 Self-Hosted and Open Source

This is probably the biggest reason I like it.

Your monitoring data stays under your control. No vendor lock-in, no per-host pricing, and no concerns about sending infrastructure metrics to third-party services.

My Setup

I initially ran Tianji locally for testing, but I've now moved it online so I can monitor my infrastructure remotely.

My deployment runs in Docker behind a reverse proxy with HTTPS enabled. Like most self-hosted projects, the setup wasn't completely straightforward I had to work through Docker networking, PostgreSQL connectivity, reverse proxy configuration, and SSL but once everything was configured correctly, Tianji has been running reliably.

A Nice Bonus: Built in Telemetry

One feature I recently started exploring is Tianji's telemetry pixel.

By embedding a tiny tracking image in blog posts or documentation, I can collect simple, privacy conscious page view statistics without integrating a heavyweight analytics platform. It's a neat feature for personal blogs, documentation sites, or static websites.

Final Thoughts

If you're already self-hosting applications, chances are you're also running multiple monitoring tools.

Tianji consolidates website monitoring, server metrics, and analytics into a single platform that is lightweight, modern, and easy to self-host. It has quickly become one of the tools I always include when provisioning a new server.

If you're looking for an open-source alternative to juggling several monitoring services, Tianji is definitely worth trying.

Visit there website

Happy self-hosting!

P.S. I'm using Tianji to monitor my own services, and I'll probably write a follow up post covering the Docker deployment, reverse proxy configuration with CyberPanel/OpenLiteSpeed, and some of the networking issues I encountered while exposing it to the internet.

How to write incident status updates that build trust

Tue, 21 Jul 2026 13:45:03 +0000

TL;DR. During an outage your status page has a second job: holding trust while the fix is still in progress. Write plainly. Say what customers feel first, say what you are doing, and promise a time for the next update. Then keep that promise, even when there is no news. The four stages are investigating, identified, monitoring, and resolved.

The idea of an incident commander did not come from software. It came from wildfires. In 1970s California, a run of huge fires pushed fire chiefs to study why large responses fell apart. The cause surprised them. It was not too few firefighters or too little water. It was communication. Different teams used different words and no one shared a plan. The fix was a simple system with clear roles and clear language, now known as the Incident Command System, and on-call engineers still use a version of it today.

Your status page is the communication part of that system. When something breaks, the repair happens in your code. The trust happens on the status page. This post is about the second part: how to write updates that keep people calm while the first part is still in progress.

The words cost real money

An outage is expensive before you write a single word. In ITIC's 2024 survey of more than 1,000 companies, over 90% of mid-size and large firms said one hour of downtime costs them more than $300,000. For 41% of them, one hour costs between $1 million and 5 million. Uptime Institute's outage research points the same way: about one in five recent outages cost more than 1 million, and more than half cost over $100,000.

You do not need to be a bank for this to matter. If your store makes $2,000 a day, one bad hour during a sale can cost more than a slow afternoon, because sales are not spread evenly across the day. The size of the number changes. The shape of the problem does not.

Here is the part teams forget. Some of that cost is the downtime itself. The rest is trust, and trust is where writing helps. Many providers offer an SLA, a promise to stay up with a penalty if they miss it. The penalty is almost always a service credit, which is money back on your bill. A credit refunds your invoice, not your customer's bad afternoon. Good updates cannot bring the service back. They can protect the relationship while it is down.

The four stages of an incident

Most status pages, including the big public ones, use the same four stages. They come from a shared standard, so a customer who has read one status page already understands yours.

Stage	What it means	What to write
Investigating	You know something is wrong. You do not know why yet.	Name the impact and say you are looking into it.
Identified	You found the cause and are working on a fix.	Say what is wrong in plain words and that a fix is coming.
Monitoring	The fix is live. You are watching to be sure it holds.	Say the fix is in and you are checking that it works.
Resolved	The service is back and stable.	Say it is over, and thank people for waiting.

The order matters. Do not jump to identified because you have a hunch. If you say you found the cause and then change your story an hour later, every update after that is worth less. Move a stage forward only when the facts move with you.

The rules of a good update

Five rules cover almost every message.

Start with impact. Say what the customer feels before you talk about servers or databases. "Some payments are failing" is more useful than "we are seeing elevated 5xx errors on the API gateway." Only the first one means anything to a customer.

Share facts, not guesses. Say what you have confirmed. If something is still unknown, say that too. Writing "we do not yet know the cause" is fine. Inventing a cause to sound in control is not. This rule earns its place under pressure: the Uptime Institute's 2025 report found that about 40% of organizations had a major outage caused by human error in the last three years, and 85% of those came from someone not following a procedure. An incident is exactly when tired people guess.

Promise the next update. Give a real time, like "next update by 15:00 UTC" or "another update within 30 minutes." This one line does more than any apology. It tells people they can close the tab and get on with their day.

Add a timestamp and a time zone. "In 30 minutes" means nothing if the reader does not know when you posted. Every update should carry a clear time.

Keep it plain. Short sentences. No blame, no jargon, no backstory. A worried customer reads fast.

A bad update and a better one

Here is a bad update of the kind you have all seen:

We are aware of an issue and our team is working hard to resolve it as quickly as possible. We apologize for any inconvenience.

It sounds polite and says nothing. What is broken? Who is affected? When will I hear more? A better version answers those questions:

Investigating: some customers cannot log in. Sign-ups and password resets are affected too. Our team is looking into the cause now. Next update by 14:30 UTC.

Same length. Only one of them lets a customer decide what to do next.

How often to post

Set the rhythm in your first message and keep it. Every 30 to 60 minutes is a good starting point for a serious outage. The exact number matters less than the promise. If you said 30 minutes, post at 30 minutes, even when the only news is "still working, no change yet." Silence after a promised time reads as "they have lost control." A boring on-time update beats an exciting late one.

Planned maintenance is the same skill, calmer

Maintenance uses three stages: scheduled, in progress, and completed. The difference is that you have time to write it well in advance. Tell people what will happen, when it will happen, and whether they need to do anything. Mark it in progress while the work runs, and completed when it is done. The tone is calmer because nothing is on fire, but the shape is the same: impact, timing, next update.

The mistake almost everyone makes

It is not the wording. It is the second update.

The first update is easy. The outage just happened, and everyone is paying attention. Then the fix takes longer than expected, the team goes quiet, and the page sits at "investigating" for two hours. From the outside, a frozen status page looks the same as a dead company.

The fix is a habit, not a talent. Decide the next-update time in your first message, set a timer, and post again when it rings. If there is no news, that is the update: no change, still working, next check in 30 minutes. The point of the promise is that people stop refreshing and trust you to come back.

Write the message, then give it a home

You can write all of this by hand during an outage. Most people write worse under stress, which is the worst possible time to start from a blank box. That is why we built a free incident update generator: pick a stage, fill in a few fields, and it writes a clean message you can paste into any status page. It runs in your browser and stores nothing you type.

The message still needs somewhere to live. A status page is where customers look when your product will not load, so it should be honest and easy to reach. If you are choosing an uptime number to promise, the uptime SLA calculator turns a percentage into real minutes, and is 98% uptime good shows how small percentages hide large amounts of downtime. For the page itself, a status page you cannot fake covers how to keep it worth trusting.

Common questions

What should the first incident update say? Say what the customer feels, say you are looking into it, and give a time for the next update. You do not need the cause yet. Waiting for the root cause before posting is the most common mistake.

What is the difference between investigating and identified? Investigating means the cause is not confirmed. Identified means you have confirmed the cause and are working on a fix. Do not move to identified based on a hunch.

Should you apologize in an incident update? One short, honest line is fine. Skip the long apology. A clear next-update time does more for trust than three sentences of sorry.

If you have run incidents: what update cadence actually held up for you under a long outage, and did anyone ever complain that you posted too often? I have never seen that complaint, but I would like to know if it exists.

Why I chose Rust over Go for an uptime monitor

Mon, 20 Jul 2026 14:44:36 +0000

I build Uptimepage, an open-source uptime monitor and status page written in Rust. People ask why Rust and not Go, since Go is the usual pick for this kind of network service. Here is the honest answer. It is not that Rust wins everywhere. It is that one part of this job made the choice for me.

The product is one promise: tell you fast and honestly when your site is slow or down. The numbers I show you, like your p99 response time, have to be clean. If my own code adds random delay, I blur the exact signal you pay for. So the runtime under the prober matters more here than it would for a normal app.

Garbage collection shows up in the tail

Go has a garbage collector. It is fast, and most apps never feel it. But it still has to do work to free memory, and that work can land inside the millisecond timings I report. Run tens of thousands of checks at once and a pause at the wrong moment lifts a p99 number. At that point I am measuring my own runtime, not your server.

Rust has no garbage collector. Memory is freed at a point I can see in the code. There is no background pause I did not write. For a tool that sells timing, that control is worth the extra work Rust asks for.

What that looks like in memory

Here are some real numbers, with a warning attached. They come from a load test on a developer laptop, not a production server. I use them to catch a slowdown between two versions of my code, not to plan capacity. A real server does better, a small box does worse. Treat them as a floor.

In one run, a single machine held 50,000 checks in flight and peaked at 933 MiB of memory. That is under one gigabyte for fifty thousand live checks. My running server uses about 42 MiB while watching its monitors, and it sits quiet when there is nothing to do. That kind of density is normal for a service with no garbage collector, and it means one small box covers a lot of monitors. I go deeper on the prober and the throughput numbers in the build story.

The bug that never compiles

Speed is only half of it. The other half is a class of bug that Rust refuses to build.

Picture many workers writing to one shared map of results at the same time. In Go this compiles and runs. Sometimes it is fine. Sometimes two goroutines write at once, you get corrupt data or a crash, and it only happens under load, which is the worst time to find out:

// compiles fine, races at runtime
results := map[string]int{}
for _, check := range checks {
    go func(c Check) {
        results[c.ID] = c.Latency // data race
    }(check)
}

Go gives you good tools for this. There is a race detector, a sync.Mutex, and channels. But remembering to reach for them is on you.

In Rust the same concurrent write does not compile. The compiler stops you until the shared map is wrapped in a lock:

// will not compile unless the shared map is a Mutex
let results = Mutex::new(HashMap::new());
std::thread::scope(|s| {
    for check in checks {
        let results = &results;
        s.spawn(move || {
            // each worker locks only for its own insert
            results.lock().unwrap().insert(check.id, check.latency);
        });
    }
});

Each worker holds the lock only for its own insert, so the writes stay safe without blocking the others for long. For code that runs day and night across many machines, "the compiler will not let you ship the race" removes a whole set of late-night bugs before they exist.

Where Go is the better pick

None of this makes Go a bad choice. Often it is the better one. Go is faster to learn. It builds in seconds. Its standard library for network services is excellent, and a new engineer can be useful in days. If I were building a normal web service, or something I had to ship this week, Go would be on the table and might win.

Rust asks more from you first. The compiler argues with you. The build is slower. You spend time on things Go would just handle. I take that trade because this job is narrow and it rewards tight control over memory and timing.

The compiler as a safety net

Heavy code review is how large teams catch data races and memory bugs. Rust gives you a lot of that for free. The compiler turns those mistakes into build errors, so they never reach production and never wake anyone up. Every change gets a strong, automatic check before it ships, which is a big part of why I trust the service to run unattended.

It is also why Uptimepage is open source and self-hostable. You can read the code, run it on your own hardware, and export your data whenever you want, so you are never locked into a single vendor. You can start from GitHub.

The honest version

The honest version is not "Rust beats Go." It is "for a tool that lives or dies by timing and runs at high concurrency, Rust fit better." Pick the language for the job in front of you. Mine happened to be a job that Rust is very good at.

If you have shipped a heavily concurrent network service, did you reach for Go or Rust, and did the GC ever actually show up in your tail latencies? Curious where people land when the timing is the product.

Lab green, field amber: reconciling two PageSpeed truths

Sun, 19 Jul 2026 08:30:53 +0000

Lighthouse can look healthy while CrUX still needs improvement. Here is how we decide which signal to trust when the two disagree.

The Lighthouse performance score on the ticket was 92. Mobile lab LCP sat under 2.5 seconds. Someone pasted the PageSpeed Insights link into Slack with a green check, and the thread closed before lunch.

Two days later Search Console still showed the URL group as Needs improvement on mobile. Chrome UX Report percentiles had not moved. The client's campaign page felt fine in a warm office on fibre; field users on mid-range phones were still waiting on the hero. Both numbers were honest, and both were answering different questions. That split is the ordinary case of lab green and field amber. Teams that treat it as a tooling bug waste a sprint arguing about which dashboard is right. Teams that treat lab and field as separate jobs prioritise more clearly and close fewer tickets on a Lighthouse performance score alone.

Why Lighthouse lab scores and CrUX field data disagree on the same URL

Lab data from Lighthouse (the engine behind PageSpeed Insights lab results) is a controlled run: fixed device profile, throttled network, repeatable conditions. It is strong for catching regressions you introduced in a deploy, for comparing template A against template B, and for reproducing a slow interaction in DevTools. Field data from the Chrome UX Report is a distribution of real visits: real devices, real networks, real scroll depth, and real interaction paths. Google's lab versus field guidance on web.dev is blunt about the gap: Lighthouse returns one LCP candidate for a given run, while field LCP on the same URL often spans several elements depending on viewport and how far the user scrolled.

Amber field percentiles with a green lab score usually mean the controlled run never hit the conditions that dominate your traffic. A consent banner that only appears for a large share of European visitors, a third-party tag that fires after a marketing campaign starts, or a product gallery that mid-range Androids struggle with can all stay invisible in a single lab profile. The reverse pattern shows up too: lab red because a synthetic throttle is harsh, while CrUX already looks healthy for the visitors who matter. Neither case is a contradiction; it is a mismatch of populations, and treating either number as the only truth usually sends engineering at the wrong work.

Device form factor makes the same disagreement louder. PageSpeed Insights and CrUX already split mobile and desktop; a lab desktop run that looks green does not clear a mobile field fail on the same URL. We covered paired device reporting in two device stories, one URL. Lab versus field sits next to that problem: even on one form factor, a green lab row and an amber CrUX row can coexist for weeks after someone declared the page fixed.

Which PageSpeed signal to trust when lab and field disagree

We use a short decision order in triage, not a debate about which score looks better in a screenshot.

When CrUX or Search Console has usable field data

Search and experience claims should follow CrUX percentiles and Search Console URL-group status, not a single Lighthouse performance score. Lab still tells you what to change next. It does not redefine whether real users are already fine.

When CrUX field samples are missing or thin

New templates, low-traffic locales, and production paths that behave like staging often lack URL-level CrUX samples. PageSpeed Insights may fall back to origin-level field data, which can flatter or hide a single money URL. In those cases we schedule lab runs, store them, and refuse to claim Core Web Vitals are good from a performance score alone. Label those rows lab-only so nobody mistakes them for field proof.

When Lighthouse and CrUX move in different directions

A deploy that adds a script often moves lab first. A traffic mix shift or a third-party change that only hits a segment of visitors often moves field without a matching lab spike on your default profile. When both move, we treat it as confirmation. When only one moves, we write that into the ticket so nobody fixes the green side while ignoring the amber side.

Keep mobile and desktop Core Web Vitals as separate rows

Averaging them into one pass/fail hides the weaker device behind a single result. A lab desktop pass does not clear a mobile CrUX fail on the same URL, and the reverse is also true. For schedules and budgets that keep both form factors honest, see mobile versus desktop Core Web Vitals monitoring on the Watcher blog.

How to run a weekly lab-versus-field PageSpeed reconciliation

On Monday we pull the priority URL list: homepage, primary landing templates, and conversion paths, each with mobile and desktop. For each row we record lab LCP, INP, and CLS from the latest scheduled run, plus CrUX or Search Console status when samples exist. A one-line note captures whether lab and field agree, disagree, or whether field is origin-fallback only.

Disagreements go into three buckets. Lab green / field amber means investigate real-user conditions the lab profile missed: third parties, consent, device skew, or an LCP element that only appears for a large share of visits. Lab amber / field green means keep watching, but do not block a release solely on a harsh lab profile unless the URL is new and has no field history yet. Both amber means pick the metric that fails in field first, then use lab to reproduce and verify the fix.

We do not paste a single PageSpeed Insights screenshot into the client channel as this week's health. Screenshots without stored history become the Tuesday-green, Thursday-panic pattern: a green lab run with nothing saved between checks. Scheduled runs, named baselines, and a short reconciliation note scale better across a roster than ad hoc one-off checks. For when manual PageSpeed Insights is enough and when it is not, the longer comparison stays on the Watcher blog: PageSpeed Insights versus automated monitoring.

How to explain lab green and field amber to clients

Clients hear green score and assume Search is happy. When field data lags or disagrees, the account manager needs one sentence that names both signals. A wording that works in our experience: "Lab tests on our standard mobile profile look healthy after the deploy; real-user Core Web Vitals for this URL group are still in Needs improvement, so we are treating field as the priority and using lab to verify each fix." That sentence does not apologise for the tools. It assigns jobs: field for user and Search status, lab for engineering feedback loops.

If only origin-level field data is available, say so: "We do not yet have URL-level field samples for this landing page; the lab series is our early-warning signal until CrUX fills in." Honesty about sample coverage beats a blended performance score that implies certainty you do not have. The client can then fund the field problem without thinking you contradicted last week's lab screenshot, and engineering keeps a clear label on what is lab-only until URL-level CrUX appears.

Add a lab-versus-field check to this week's monitoring list

Pick five money URLs and open the latest PageSpeed Insights (or monitoring) results with mobile selected. For each URL, write whether lab and field agree, disagree, or whether field is missing or origin-only. Add the same pass for desktop on the two highest-revenue paths. Change one alert or report so a lab-only improvement cannot close a ticket while CrUX on that URL group remains amber. Store the next week's runs against the same URL list so the reconciliation is a habit, not a special project after a client complaint.

Reconcile lab and field before you close the ticket

Treat green lab and amber field as two measurements with different jobs, not a vote to ignore one dashboard. Use the PageSpeed Insights versus automated monitoring guide when you need the workflow split between spot checks and scheduled history, and the mobile versus desktop monitoring guide when the disagreement is also a form-factor split. Reconcile both signals before you treat a Lighthouse performance score as closed.

Originally published on Hashnode.

The hidden costs of AI-built internal tools (next year's bill is not zero)

Sat, 18 Jul 2026 13:38:14 +0000

Infrastructure, maintenance, security updates, and the client work that did not start while the team shipped a 'free' internal stack.

Someone on a developer thread put it better than most vendor decks do: no software works without infrastructure, maintenance, and security updates. Avoiding an expensive third-party tool can be rational. Pretending next year's bill will be zero is not. And while the internal team spent a month "saving" the subscription, which client projects were delayed or never started?

That is the question we hear when agencies and product teams use AI coding tools to stand up an internal dashboard, a scraper farm, or a home-grown PageSpeed loop. The first demo looks cheap. The second year does not.

What follows is how we price that trade-off when the alternative is buying a monitoring product, including our own: Apogee Watcher. We are biased toward layering a specialised tool onto the stack you already run. We are not biased toward pretending AI removes run cost.

Why AI-built internal tools look free in month one

AI coding assistants compress the time from idea to working prototype. A senior engineer can scaffold a multi-site Lighthouse runner, a Slack notifier, and a rough report page in days instead of weeks. The invoice for that month may show only tokens, coffee, and a weekend.

The demo hides four lines that do not appear on the prototype slide:

Hosting, queues, storage, and secrets for something that must run every day.
Dependency and runtime upgrades when Node, Chromium, or the Lighthouse API moves.
Access control, audit logs, and incident response when client URLs and credentials sit in your system.
The billable work that did not ship while that engineer owned the internal project.

Month one is a capitalisation of attention. Month twelve is operations. Those are different ledgers, and confusing them is how "free" tools become expensive quietly.

What the hidden costs of AI internal tools include next year

We are not arguing against building. We are arguing against incomplete cost models, especially when AI made the first commit feel almost free. The spreadsheet that only lists SaaS seats will always favour the internal prototype.

Infrastructure. Schedulers fail. Disk fills. Rate limits trip. Someone gets paged when the overnight suite stalls. A managed product folds that into a subscription; an internal tool folds it into someone's Friday, which is still a cost even when it never appears as a line item.

Maintenance. Lighthouse versions, PageSpeed Insights quotas, browser images, and OAuth tokens all age. The code that "just works" in March needs an owner in September. AI can rewrite a broken script. It does not attend the capacity meeting when three clients regress on the same weekend.

Security updates. Anything that stores client domains, screenshots, or API keys is a security surface. Patch cadence, least privilege, and offboarding when staff leave are not optional extras. They are the price of keeping the tool on the network without becoming the next incident review.

Opportunity cost. This is the line the comment nails. If a strong engineer spent four weeks on an internal monitoring stack, those weeks were not spent on a client launch, a performance remediation, or a proposal that might have paid for several years of a SaaS seat. "We avoided a subscription" is incomplete without "we deferred this revenue work."

For agencies, that last line is usually the largest. Delivery capacity is the scarce asset. Software licences are not.

Build versus buy for portfolio PageSpeed monitoring

Performance monitoring is a poor place to learn that lesson the hard way. You need history across many URLs, device splits, budgets, and alerts that arrive when a deploy breaks LCP. Building that once for one client is a weekend project. Building it for forty clients with permissions, white-label reports, and fair API quotas is a product.

We wrote our build-versus-buy framing for agencies in Lighthouse CI vs Managed Monitoring: Build vs Buy for Agencies. The short version for this discussion: CI gates and managed monitoring solve different jobs. AI makes the CI path faster to start. It does not erase the ownership tax of keeping a multi-tenant monitoring system honest.

If the honest goal is scaling client coverage without growing the monitoring headcount, the decision is less about whether AI can generate a script and more about whether you want that script to become a permanent internal product. Scaling Without Headcount is the Watcher-side case for automating the loops that expand with every new domain. That is the capacity question, not the prototype question.

Checklist before you green-light an AI-built internal tool

Use this before you celebrate the "we built it ourselves" thread:

Question	If the answer is no
Who owns patches and dependency upgrades in six months?	Budget an owner or do not start
Where do credentials, screenshots, and client URLs live, and who audits access?	Treat it as a security project, not a hackathon
What is the monthly infra and quota cost at full portfolio size?	Re-run the SaaS comparison at that size
Which client commitments slip if this takes four weeks?	Put that list next to the "saved subscription" slide
Will this layer onto tools we already trust, or replace them under pressure?	Prefer layer-don't-replace unless you have a retirement plan

AI changes the cost of the first commit. It does not change the cost of being on call for the result. Treat ownership as part of the build estimate, not a surprise in month four.

When building an internal monitoring tool still makes sense

Build when the workflow is a genuine differentiator, when compliance forces data to stay inside your boundary, or when you are extending a stack you already operate with spare capacity. Build a thin adapter. Buy the commodity loop.

Do not build because a social post claimed SaaS is a tax and AI is free labour. That framing skips maintenance, security, and the client work that paid the salaries funding the experiment. Those skipped lines are where next year's bill hides.

Count the full cost, then choose

Next year's bill will not be zero. Count infrastructure, maintenance, security updates, and the projects that waited. If the total still beats a specialised product after you price the opportunity cost, build with eyes open. If it does not, buy the monitoring layer and keep your engineers on client outcomes.

We built Apogee Watcher for the second case: multi-site PageSpeed schedules, budgets, and alerts without asking your team to become a second product company. Layer it on. Keep the AI assistants for the work that is uniquely yours.

Originally published on Hashnode.

98% uptime sounds high. It means your service is down 7.3 days a year

Fri, 17 Jul 2026 13:24:32 +0000

Cover photo by the blowup on Unsplash.

I build an uptime monitor, so this question lands in my inbox a lot: is 98% uptime good?

For a public website or a paid API, no. For an internal tool or a side project, it is fine. The difference is one division away.

98% looks like a top grade because school taught us that 98 out of 100 is excellent. Uptime does not grade like school. The whole scale for public services lives between 99% and 100%, and serious targets differ only in the digits after the decimal. On that scale, 98% sits at the bottom.

Turn the percentage into time

The allowed failure at 98% is 2%. Two percent of a year is 7.3 days. Two percent of a 30-day month is 14.4 hours. If your shop makes 2,000 USD a day, 98% uptime means you accept about $14,600 of closed-door time per year and the target still counts as met.

Here is the same math for every common target. The last column prices the downtime for that $2,000-a-day shop.

Uptime	Per day	Per 30-day month	Per year	Lost sales per year
90%	2.4 hours	3 days	36.5 days	$73,000
95%	1.2 hours	36 hours	18.3 days	$36,500
98%	28.8 minutes	14.4 hours	7.3 days	$14,600
99%	14.4 minutes	7.2 hours	3.7 days	$7,300
99.5%	7.2 minutes	3.6 hours	1.8 days	$3,650
99.9%	1.4 minutes	43 minutes	8.8 hours	$730
99.95%	43 seconds	21.6 minutes	4.4 hours	$365
99.99%	8.6 seconds	4.3 minutes	52.6 minutes	$73
99.999%	0.9 seconds	26 seconds	5.3 minutes	$7

Two jumps on this table do most of the work in real contracts. From 98% to 99.9%, the allowed downtime drops from 14.4 hours a month to 43 minutes. From 99.9% to 99.99%, it drops from 43 minutes to 4.3 minutes, and that second jump usually costs about ten times more engineering than the first while saving the shop $657 a year.

The shape of the downtime matters

98% per month is 14.4 hours, but the number says nothing about how those hours land.

Thirty minutes of planned maintenance every night at 03:00 adds up to 98% and most users never notice. One 14-hour outage on the day of your product launch is also 98%. Same score, very different month.

So a single uptime percentage is a summary, not the full story. When someone quotes you a number, also ask about the longest single outage and when it happened.

When 98% is enough

Plenty of systems can live at 98% and nobody gets hurt:

An internal wiki. People retry after lunch.
A staging environment. Downtime there is often planned.
A batch job that builds reports at night. It has hours of slack before anyone reads the output.
A home server on a residential connection. Your power company already decided your uptime for you.

The shared pattern: when these go down, nobody loses money and nobody loses trust. Paying for more nines there is waste.

When it is not

A checkout page, a paid API, a login service. Here 98% fails twice. First the direct cost: 14.4 hours a month of failed requests and support tickets. Second the trust cost, which is larger and slower. A customer who hits your outage twice in one week does not check your uptime report. They remember that your service is the one that breaks.

One more trap: an SLA is not uptime. Uptime is the measured number. An SLA is a contract promise with a penalty, and the penalty is almost always a service credit. If a provider misses its 99.9% SLA, you get part of your bill back. Your customers get nothing.

What to aim for instead

99.9% is the default target for customer-facing services for a reason. 43 minutes a month is enough room for a bad deploy and a couple of small failures, and a small team can hit it without heroics. Above that, each nine costs roughly ten times more and most users cannot feel the difference.

If you want to run your own numbers, I keep a free uptime SLA calculator and an error budget calculator on our site; both work without signup.

What target do you actually run in production, and did you pick it or inherit it? I am curious how many teams measured before they promised.

The status page you can't fake: measured uptime, not published

Wed, 15 Jul 2026 13:51:58 +0000

A status page is the one dashboard a company publishes about its own service. It is also the one place where the company has a reason to look good. That is a problem. If the page can be edited to look better than reality, it stops being useful. So when you build or choose a status page, ask one thing: can someone hide a real outage on it?

The short answer: a status page you can trust builds its uptime bar from real checks, not from the incidents a person chose to publish.

Two layers, and only one is yours to edit

A good status page has two parts. The first part is the measured data: the green and red timeline, the 90-day bar, the uptime percent. The second part is incidents: the notes a person writes to explain what broke, what they are doing, and when it is fixed. On the page they sit next to each other and look the same. They are not the same, and mixing them is where trust gets lost.

The measured data answers one question: what did the checks see? The incident notes answer a different one: what does the team want to say about it? The first is a fact. The second is a story. A status page you can trust lets the team write the story, but keeps them away from the facts.

The story layer also includes the postmortem, the write-up you post after an outage. A postmortem is honesty you add on purpose. You explain what broke and why, because you choose to. The bar works the other way. It shows the failure on its own, whether you write anything or not. You control the story. You do not control the facts.

The trap: a bar that only shows what you published

The common mistake is to build the uptime bar from incidents. It feels natural. You already open an incident when something breaks, so why not color the timeline from incidents too? Now the bar turns red only where an incident exists and is marked public.

The problem comes on the day an outage has no public incident. Maybe no one published it. Maybe a setting was wrong. Maybe the monitor was added to the page after the outage, so its incident was saved as private and never checked again. In every case the timeline shows green over a real red day, and it does this quietly. The uptime number goes up. The customer sees 100 percent over a week they remember as broken.

One version of this trap is easy to build by accident, so it is worth explaining. You decide "is this incident public?" once, at the moment the incident opens, based on whether the monitor was on a public page right then. Then you save that answer and never look again. In the code it looks like a live check. It is really a photo taken one time. Move the monitor onto the page a day later, and its past outages stay hidden, because the photo was taken before the monitor was there. Any yes or no mark that is set once and then trusted forever has this problem.

The fix is a rule, not a switch

The bar should come from the measured data. It needs only one rule to stay calm: wait for confirmation before you count downtime. You do not want one failed check from one place to turn a whole day red, because networks are noisy and one bad checker is not an outage. So you wait for agreement: more than one region failing at the same time, for more than one check in a row. That one rule is enough to keep a short blip from becoming a red day.

One thing matters here: the same rule feeds both your alerts and your uptime bar. If the rule that wakes your on-call person is the same rule that colors the timeline, the two can never tell different stories. The moment you add a second rule just for the bar, it will drift from the first, and the page will disagree with itself. On-call gets paged, but the public history says everything was fine.

Publishing stays where it belongs, on the notes. The team decides whether to write an incident, what to say, and when to post the all-clear. They do not decide whether last Tuesday was down. The checks already decided that.

Test the status page you already have

You can check any status page, including your own, in a few minutes.

Add a monitor to a page after it has already had an outage. Does the history show the outage, or does it start clean from the day you added the monitor?
Take a real incident and unpublish it. Does the bar keep the red day, or does the day turn green?
Make one region fail for one second. Does the whole day go red, or does the bar stay calm?

A page you can trust shows the outage in the first test, keeps the red in the second, and stays calm in the third. A page that fails these is usually not lying on purpose. It just built its timeline on top of what people chose to publish, and that always has holes.

Where this leaves you

This is how I built it into Uptimepage: the 90-day bar and each status light come from confirmed downtime, measured across regions with a confirmation rule, not from what someone chose to publish. Incidents and postmortems are the layer you write by hand. It is AGPL and open source on GitHub, and there is a longer write-up on the internals if you want the Rust side.

You should not be able to fake your uptime. You should not be able to fake it by accident either. The bar is a measurement. Keep it one.

How does your status page compute its uptime bar, from checks or from published incidents? Curious what people run and whether anyone has been bitten by the frozen-flag version of this.

More clients, same team: scale agency monitoring without headcount

Tue, 14 Jul 2026 19:31:01 +0000

When the roster grows but hiring does not, decide which PageSpeed loops to automate and which still need a human owner.

The account director signed two retainers in March and asked in April why performance reporting still felt like a side project. Same delivery team. Same Slack channels. Three more client domains in the portfolio, and suddenly the Friday "quick PageSpeed sweep" was eating a full afternoon.

Nobody in that room wanted to hire a monitoring intern. They wanted the work they already sold to stop expanding faster than the people doing it. That is the shape of the problem we hear most often when agencies talk about automation: not a dislike of tools, but a roster that outran the headcount plan.

Below is the split we use in conversations with agency ops leads: what still needs a human, where scheduled systems return real hours, and how to tell the difference before you buy another subscription.

Why agency rosters outrun performance monitoring capacity

Most agencies do not fail at performance because they lack expertise. They fail because monitoring work scales with URLs and deploys, not with billable projects.

Each new client adds discovery (what should we watch?), cadence (how often?), ownership (who responds at 6pm?), and reporting (what does the client see monthly?). Those steps are small in isolation. Stack ten clients and they compound into a parallel job nobody scoped in the SOW.

Hiring fixes part of that, but margins in 2026 rarely allow a dedicated performance analyst per five sites. The teams that grow without burning out usually redraw the boundary: humans own judgement, client context, and remediation; machines own repetition, storage, and threshold checks.

That boundary is not philosophical. It is a weekly calendar problem.

Which performance monitoring tasks still need a human

Automation is a poor substitute for work that requires negotiation or interpretation. Keep people on:

Scoping which URLs matter when the client has six checkout variants and marketing insists every campaign landing page is "critical."
Explaining a regression to a non-technical stakeholder without turning a Slack alert into panic. Context, history, and what you tried last month still belong to account leads.
Prioritising fixes when three templates breach budgets at once and only one developer has capacity this sprint.
Deciding whether a one-point INP wobble is noise or the start of a script conflict after a tag-manager change.

Those tasks use institutional memory. A cron job can tell you CLS crossed 0.15 on /pricing. It cannot tell you the client is in renewal week and needs a calm summary instead of raw numbers.

If your team is small, protect human time for those conversations. That is where retainers renew or fray.

What to automate first in agency PageSpeed monitoring

The loops that drain agencies are surprisingly uniform. In our experience, automation earns its keep when it replaces the same manual sequence every week:

Finding URLs worth monitoring after sitemaps change or a client launches a microsite.
Running PageSpeed tests on a fixed schedule for mobile and desktop without someone opening a browser tab.
Storing results so "what did we see before the deploy?" is a query, not an archaeology project in Google Drive.
Sending alerts when agreed thresholds break, with cooldowns so the channel is not spammed by lab noise.
Generating client-facing summaries from that stored history instead of copying scores into slides by hand.

That list is operational, not glamorous. It is also where we see teams reclaim half a day per week once the boring spine is in place.

The case for doing it now, including how manual effort grows with site count, is spelled out on our main blog:

Why agencies need automated performance monitoring in 2026

That Watcher piece holds the fuller argument, numbers, and what "automated" means feature by feature.

Why automated monitoring fails without owners and thresholds

Buying a tool without naming owners produces a expensive reminder to run manual checks anyway.

Common failure modes we see after a rushed rollout:

Every site uses different thresholds because nobody wrote a default policy, so alerts mean different things to different account managers.
Discovery is turned on but nobody reviews new URLs, so quotas burn on low-value routes while checkout sits untested.
Reports go out on schedule with pretty charts and no narrative, so clients treat them as spam.
Alerts route to a shared inbox with no first responder, so regressions age until someone notices in a quarterly review.

Automation amplifies whatever process you already have. If the process is "whoever is free runs Lighthouse," the tool becomes a faster version of the same chaos.

Fix the policy first: default budgets, priority URL sets, one named responder per client, and a short internal note on what triggers escalation versus what waits for the weekly scan. Then wire the scheduler.

Automate or keep human: a split for five-to-fifteen person teams

You do not need a transformation programme. You need a written line between repeatable and judgement-heavy work.

Work type	Automate	Keep human
URL inventory after sitemap changes	Yes, with a weekly human review of new routes	Final call on "priority" URLs
Scheduled lab tests on agreed templates	Yes	Ad-hoc deep dives during incidents
Threshold alerts to Slack or email	Yes, with cooldowns	Triage and client wording
Monthly PDF or email summaries	Yes, from stored runs	Executive summary paragraph
Root-cause analysis after a deploy	No	Developer + account lead
Retainer scope when metrics stay red	No	Ops or delivery lead

Start with one client who already trusts you. Document their five priority URLs, set conservative thresholds, route alerts to one person, and run that pattern for two weekly cycles. If the alert noise is tolerable and the history answers "when did this break?", copy the template to the next client.

Scaling without adding headcount is less about heroics and more about cloning a boring playbook. The brand-side walkthrough of that idea, including how multi-tenant monitoring fits agency portfolios, lives here:

Scaling a digital agency without headcount

Four questions before the next performance monitoring retainer

Before you sign another monitoring retainer, answer four questions in writing (internal doc is fine):

Who owns first response when an alert fires outside business hours?
Which three to five URLs per client are non-negotiable in week one?
What default thresholds apply before you tune per site?
What is explicitly out of scope for the monitoring fee versus remediation?

If those answers exist, automation is a capacity multiplier. If they do not, you are automating confusion.

Pick the client whose reporting currently hurts the most. Implement the split table for that account only. Measure one month: hours spent on manual runs, number of regressions caught before the client noticed, and whether the account lead still dreads month-end.

Small proof beats a platform rollout memo every time.

Audit recurring performance tasks before you add another client

Block thirty minutes with whoever runs client delivery. List every recurring performance task from the last two weeks. Mark each row automate, keep human, or delete (yes, some rituals should die). For anything marked automate, name the tool or script and the owner who verifies output every Friday.

You will likely find the same four loops: find URLs, run tests, store numbers, tell someone. Automate those first. Keep your best people for the conversations that follow the alert.

More clients with the same humans is workable when repetition stops living in people's calendars.

Originally published on Hashnode.

Error budgets, explained: SLOs, burn rate, and when to stop shipping

Mon, 13 Jul 2026 14:13:45 +0000

If you have ever seen a team argue about whether they can ship a risky change, an error budget is the tool that ends the argument with a number. Here is how it works, with the formulas and a simple way to turn it into alerts.

What an error budget is

An uptime target has a second number hidden inside it. Say you promise 99.9% uptime. You are also saying that 0.1% is allowed to fail. Put that 0.1% into real time, and that is your error budget: the downtime you can have before you break the promise.

This changes how you look at downtime. It stops being a mistake to feel bad about and becomes a budget you can spend: on a risky deploy, on a slow service you depend on, or on a migration. When the budget runs low you slow down. When it is healthy you can move fast.

The formula

An SLO is your reliability target, for example 99.9%. The gap to 100% is the failure you are allowed. Multiply that gap by the length of the window, and you get real time you can spend.

error budget = window x (1 - SLO)

A 99.9% SLO over a 30-day month is 2,592,000 seconds times 0.001. That is 43 minutes and 12 seconds. That is the whole budget for the month, shared across every incident, not a fresh 43 minutes each day.

So three nines is not "never go down". It is a 43-minute budget each month. Every extra nine costs about ten times more engineering, for downtime that most users never notice. This is why Google says that 100% is the wrong target for almost every service.

Burn rate is the speed

The total tells you how much you can spend. It does not tell you how fast. Two services can both sit at 99.9% for the month: one loses the budget slowly, the other loses it all in a single bad hour. Burn rate tells them apart.

burn rate = (1 - measured) / (1 - SLO)

A burn rate of 1x spends the whole window exactly. 2x spends it in half the time. Below 1x, you finish the month with budget left. Above 1x, the rate tells you the deadline:

budget runs out in = window / burn rate

At 2x, a 30-day budget is gone in fifteen days. At 14.4x it is gone in about two days. That same 14.4x spends 2% of the budget in one hour, which is the level most fast alerts are set to.

Turning burn rate into alerts

One threshold is not enough. It either alerts too late or it sends too many false alarms. The common fix uses two windows: a long one to confirm the problem is real, and a short one to clear the alert quickly once you fix it. Both have to be burning for the alert to fire. For a 30-day budget, these are the usual settings:

Fast page: 2% of the budget in 1 hour (with a 5-minute short window). This is a 14.4x burn.
Page: 5% in 6 hours (30-minute short window). This is a 6x burn.
Slow ticket: 10% in 3 days (6-hour short window). This is a 1x burn.

The fast page catches a sudden outage. The slow ticket catches a slow problem that would still use up the whole month if nobody looked.

The rule is the point

The math is the easy part. The value comes from a rule you agree on before anything breaks. The rule is simple. When the budget runs out, risky launches stop, and the team works on reliability until the budget grows back. While the budget is healthy, you ship and you take the risk.

One more rule keeps it fair. If you never spend your budget, your SLO is too strict, and you are paying for reliability that no user asked for.

Try the numbers

I turned these formulas into a free, no sign-up error budget calculator. You enter an SLO and your measured availability, and it shows budget spent, budget left, burn rate, and a burn-down chart. There is also an uptime SLA calculator with the full downtime-per-nine table.

How does your team set its SLOs and alert thresholds? I would like to hear what works in practice.

System Monitoring for AI Networks: Real-Time Health Checks & Dashboards

Fri, 10 Jul 2026 05:35:56 +0000

Your multi-agent system is running. You have no idea if it’s healthy. Agents are fighting, quality is degrading, costs are exploding, and you don’t know until users complain. I built Socratic-performance to solve this: real-time monitoring for agent networks with health dashboards, automatic anomaly detection, and predictive alerts. Know your system is broken before your users do.

The Problem: Flying Blind with AI Systems
Scenario 1: The Silent Degradation
Your recommendation system worked well for 6 months.

Then performance slowly degraded:

Month 1: 89% accuracy (fine)
Month 2: 87% accuracy (slight dip)
Month 3: 84% accuracy (acceptable)
Month 4: 79% accuracy (problem)
Month 5: 72% accuracy (critical)
You don’t notice until Month 5 when users complain.

By then, you’ve lost €500k in revenue.

Root cause? A data pipeline started failing 2 months ago. But you had no visibility.

Scenario 2: The Resource Explosion
Your agents are running. Resources are being consumed.

You have no visibility into:

Which agent is consuming most CPU?
Why did GPU usage spike at 3 AM?
Which API calls are most expensive?
Is one agent running in a loop, wasting money?
One agent goes haywire and spends €10k/day on API calls before you notice.

Scenario 3: The Agent Conflict
Your agents are supposed to work together.

Instead, they’re fighting:

Agent A makes a decision
Agent B overrides it
Agent C reverts Agent B’s change
System thrashes, accomplishes nothing
You don’t see the conflict. Users see slowness. You blame infrastructure.

Scenario 4: The Cascading Failure
One agent fails. It was supposed to handle 10% of traffic.

System routes that traffic to other agents.

Other agents become overloaded.

They start failing too.

Within minutes, the entire system is down.

But you didn’t see the first failure. By the time you notice, it’s too late.

Why Traditional Monitoring Fails for AI Systems
Traditional Monitoring Tracks Infrastructure
Traditional tools measure:

CPU usage
Memory usage
Network latency
Disk space
Uptime
These tell you if your server is alive. They don’t tell you if your system is working correctly.

AI Systems Need Business Monitoring
You need to know:

Is accuracy degrading?
Are agents conflicting?
Is cost per inference increasing?
Are users satisfied?
Is the system still aligned with what we’re optimizing for?
Traditional monitoring can’t answer these questions.

AI Systems Have Emergent Failures
Infrastructure problems: One component fails, error is obvious.

AI problems: System keeps running but produces wrong answers. System is slow not because of infrastructure but because agents are fighting. System costs explode not because of overloaded hardware but because agents are running inefficient loops.

These failures don’t show up in infrastructure metrics.

The Solution: Socratic-Performance
Socratic-performance is a comprehensive monitoring system for AI networks.

It measures what matters:

Agent health (are they working correctly?)
System health (are all agents working together?)
Business metrics (are we achieving what we’re optimizing for?)
Resource efficiency (are we using resources well?)
Trend detection (is something degrading?)

Core Metrics: The 5 Pillars
Pillar 1: Agent Health

For each agent:

Accuracy (decisions are correct)
Latency (decisions are fast)
Reliability (agent doesn’t fail)
Consistency (agent is stable over time)
Resource usage (agent is efficient)
RecommendationAgent Health:

├─ Accuracy: 89% (target: 90%) ⚠️ slightly low

├─ Latency: 245ms (target: <300ms) ✓ good

├─ Reliability: 99.8% (target: 99.9%) ✓ acceptable

├─ Consistency: 0.94 (how stable?) ✓ stable

└─ Resource Usage: 45% of budget ✓ efficient

Pillar 2: System Health

For the entire system:

Coherence (agents working together or fighting?)
Throughput (how many decisions per second?)
Quality (end-to-end accuracy)
Cost efficiency (cost per decision)
User satisfaction (are users happy?)
System Health Dashboard:

├─ Coherence: 0.92 (agents agree) ✓ healthy

├─ Throughput: 1,250 decisions/sec ✓ good

├─ Quality: 86% accuracy ✓ acceptable

├─ Cost: €0.012 per decision (target: €0.010) ⚠️ slight overage

└─ User Satisfaction: 4.2/5.0 ✓ good

Pillar 3: Resource Efficiency

CPU usage per agent
Memory usage per decision
API costs
Total cost per outcome
Budget remaining
Resource Usage:

├─ CPU: 60% of available (⚠️ approaching limit)

├─ Memory: 45% of available ✓ good

├─ API Cost: €18,500 this month (budget: €20k) ✓ on track

├─ Cost per decision: €0.012 (target: €0.010) ⚠️ monitor

└─ Efficiency trend: ↘️ degrading (was €0.009 last month)

Pillar 4: Anomaly Detection

Automatically detect:

Accuracy suddenly dropped
Agent started failing
Cost spiked
Conflict detected between agents
Traffic pattern unusual
Anomalies Detected (Last 24h):

├─ ⚠️ 11:45 AM - Accuracy drop detected

│ └─ RecommendationAgent accuracy: 89% → 73%

│ Likely cause: data quality issue detected

│ Confidence: 92%

├─ ⚠️ 2:15 PM - Cost spike detected

│ └─ API calls increased 3.5x normal

│ Agent: DataEnrichmentAgent

│ Duration: 15 minutes

├─ ✓ 4:30 PM - Agent recovered

└─ ℹ️ 8:00 PM - Unusual traffic pattern

└─ Peak traffic 2 hours early

  All agents handling well

Pillar 5: Predictive Alerts

Not just “problem is happening now” but “problem is coming”:

Predictive Alerts:

├─ ⚠️ MEDIUM PRIORITY - Accuracy will degrade in ~4 hours

│ └─ Trend analysis shows 0.5% per hour decline

│ At current rate, hits alert threshold at 3:30 PM

│ Recommendation: Investigate data pipeline

│

├─ ⚠️ LOW PRIORITY - Cost will exceed budget in 8 days

│ └─ Current burn rate: €2,500/day

│ Budget remaining: €20,000

│ At current rate, exhausted by May 22

│ Recommendation: Optimize expensive queries

│

└─ 🔵 INFO - Seasonal traffic increase expected

└─ Historical pattern suggests traffic increase this Friday

  Agents may need scaling

  Plan ahead

Real-World Impact: Examples
Case Study 1: Detecting Silent Degradation
Day 1: System accuracy is 89%. Dashboard shows green.

Day 2: Socratic-performance detects 0.3% decline. Not critical yet, but flagged for investigation.

Day 3: Decline continues. Now at 87%. Alert escalates to medium priority. Investigation begins.

Day 4: Root cause found – data pipeline processing outdated information. Fixed.

Result: Caught degradation on Day 2. Total impact: ~€50k revenue loss.

Without monitoring: Degradation continues until Day 12. Total impact: €500k+ revenue loss.

Savings: €450k+ by detecting early.

Case Study 2: Preventing Resource Explosion
Monday 3 AM: One agent starts behaving oddly, spinning in a loop.

Monday 3:05 AM: API cost anomaly detected. System alert triggered.

Monday 3:08 AM: Engineer receives alert. Investigates.

Monday 3:15 AM: Agent behavior identified and corrected.

Result: Total API waste: €2,500 (caught within 15 minutes).

Without monitoring: Agent continues all week. Total damage: €80k+ in wasted API calls.

Savings: €77,500.

Case Study 3: Detecting Agent Conflict
Morning: All agents running. System seems fine.

9:30 AM: Socratic-performance detects high “conflict score”. Agents are making contradictory decisions and reverting each other.

9:35 AM: Team reviews conflict logs and identifies the issue.

9:50 AM: Conflict resolved. System coherence restored.

Result: 20 minutes of suboptimal performance. System throughput drops 15% during incident but fully recovered.

Without monitoring: Conflict might go unnoticed. Users report slowness. Team spends hours debugging infrastructure (wrongly) before finding agent conflict.

The Architecture: How Monitoring Works
Component 1: Metric Collection
Every agent reports metrics continuously:

Agent Status Update (every 60 seconds):

├─ Agent ID: RecommendationAgent_1

├─ Timestamp: 2026-05-14T14:32:00Z

├─ Decisions made: 1,250

├─ Accuracy: 0.89

├─ Avg latency: 245ms

├─ Errors: 2

├─ API calls: 3,500

├─ API cost: €42

├─ Resources:

│ ├─ CPU: 45%

│ └─ Memory: 340MB

└─ Dependencies:

├─ DataService: healthy

└─ VectorDB: healthy

Component 2: Aggregation
All agent metrics aggregated into system-wide view:

System Status (aggregated):

├─ Total agents: 12

├─ Healthy agents: 11

├─ Degraded agents: 1 (DataEnrichmentAgent)

├─ Failed agents: 0

├─ System accuracy: 86%

├─ System throughput: 1,250 decisions/sec

├─ System cost rate: €18.50/minute

└─ Health score: 92/100

Component 3: Anomaly Detection
Real-time detection of unusual patterns:

Anomaly Detection Algorithm:

├─ Baseline: Historical pattern of metric values

├─ Current: Today's metric values

├─ Deviation: How much different is today?

├─ Threshold: Alert if deviation > 3 std deviations

├─ Confidence: How sure are we?

└─ Action: Alert if confidence > 90%

Example:

├─ Baseline accuracy: 89% (average over past month)

├─ Current accuracy: 73%

├─ Deviation: 16 percentage points

├─ Std deviation: 2.5 percentage points

├─ Z-score: 16/2.5 = 6.4 (extremely unusual!)

├─ Confidence: 99.99%

└─ Action: CRITICAL ALERT

Component 4: Root Cause Analysis
When anomaly detected, system investigates:

Accuracy dropped from 89% to 73%. Root cause analysis:

Checking agent inputs:

├─ Data quality: ✓ good

├─ Data volume: ✓ normal

├─ Feature values: ✓ in normal range

└─ → Not a data problem

Checking agent internals:

├─ Agent code: ✓ unchanged

├─ Agent parameters: ✓ stable

├─ Agent resources: ✓ sufficient

└─ → Not an agent code problem

Checking dependencies:

├─ DataService: ✓ healthy

├─ VectorDB: ✓ fast

├─ APIClient: ✓ responsive

└─ → Not a dependency problem

Checking output usage:

├─ Downstream services: ⚠️ changed

│ └─ RankingService now expects different format

│ Old format: {"score": 0.89}

│ New format: {"score": 0.89, "confidence": 0.95}

│ └─ RankingService confused by missing field

│ Uses default value (causes low scores)

└─ → ROOT CAUSE FOUND

Problem: Downstream dependency changed format

Solution: Update agent output format

Action: Auto-fix applied (if possible) or alert engineer
Component 5: Predictive Alerts
Don’t wait for problem to happen:

Trend Analysis:

Last 7 days:

├─ Day 1: Accuracy 89.0%

├─ Day 2: Accuracy 88.8% (-0.2%)

├─ Day 3: Accuracy 88.4% (-0.4%)

├─ Day 4: Accuracy 87.8% (-0.6%)

├─ Day 5: Accuracy 87.0% (-0.8%)

├─ Day 6: Accuracy 86.0% (-1.0%)

└─ Day 7: Accuracy 84.8% (-1.2%)

Trend: Declining at 0.25% per day (accelerating)

Projection:

├─ In 5 days: 84% (warning threshold)

├─ In 8 days: 82% (alert threshold)

├─ In 12 days: 79% (critical threshold)

Prediction confidence: 85%

Alert timing:

├─ Medium alert issued (accuracy trending down)

├─ Recommended investigation: Now (while easy to fix)

└─ Critical alert will trigger in ~8 days (if trend continues)
Implementation: Setting Up Monitoring

Step 1: Enable Performance Tracking
from socratic_performance import PerformanceMonitor

monitor = PerformanceMonitor(

    system_name="RecommendationEngine",

    tracking_enabled=True,

    metrics=[

        "accuracy",

        "latency",

        "cost",

        "resource_usage",

        "coherence"

    ]

)

Step 2: Agent Reporting

# Agents report metrics automatically

agent = RecommendationAgent(

    monitoring=monitor,

    report_interval=60  # seconds

)



# Every 60 seconds, agent reports:

# - How many decisions made

# - Accuracy achieved

# - Latency

# - Resources used

# - Any errors

Step 3: Dashboard Setup

# Start monitoring dashboard

dashboard = monitor.create_dashboard(

    port=8000,

    refresh_interval=30  # seconds

)



# Access at: http://localhost:8000

# See real-time metrics, historical trends, anomalies

Step 4: Alert Configuration

# Set alert thresholds

monitor.set_alert_threshold(

    metric="accuracy",

    warning_level=0.85,

    critical_level=0.80,

    alert_on_trend=True

)



monitor.set_alert_threshold(

    metric="cost_per_decision",

    warning_level=0.015,

    critical_level=0.020

)

Step 5: Integration with Team

# Send alerts to team

monitor.connect_slack(

    webhook="https://hooks.slack.com/services/..."

)



monitor.connect_email(

    recipients=["*Emails are not allowed*"]

)



# PagerDuty for critical alerts

monitor.connect_pagerduty(

    api_token="..."

)

Dashboard Features
Real-Time Status
Live view of all agents:

Agent Status (Live):

┌─────────────────────────────────────┐

│ RecommendationAgent ✓ HEALTHY │

│ Accuracy: 89% | Latency: 245ms │

│ Cost: €12.50/min | CPU: 45% │

├─────────────────────────────────────┤

│ FilterAgent ✓ HEALTHY │

│ Accuracy: 95% | Latency: 120ms │

│ Cost: €2.50/min | CPU: 20% │

├─────────────────────────────────────┤

│ RankerAgent ⚠ DEGRADED │

│ Accuracy: 82% | Latency: 890ms │

│ Cost: €5.00/min | CPU: 85% │

├─────────────────────────────────────┤

│ PersonalizerAgent ✓ HEALTHY │

│ Accuracy: 87% | Latency: 150ms │

│ Cost: €3.50/min | CPU: 35% │

└─────────────────────────────────────┘
Historical Trends
View metrics over time:

Accuracy Trend (Last 30 days):

90% ┤ ╱╲

89% ┤ ╱ ╲ ╱╲

88% ┤ ╱ ╲ ╱ ╲

87% ┤╱ ╲╱ ╲

86% ┤ ╲ ╱

85% ┤ ╲ ╱

84% ┤ ╲╱

└──────────────────────────

 May 1         May 15      May 31

Current: 89%

Average: 87.2%

Trend: ↗️ Improving

Projection: Will reach 91% in 2 weeks
Anomaly Timeline
When problems occurred:

Anomaly History (Last 7 days):

May 8 10:45 AM ⚠️ Accuracy spike (negative)

   11:20 AM  ✓  Resolved

   Duration: 35 minutes

May 10 3:15 AM ⚠️ API cost spike

    3:45 AM  ✓  Resolved

    Duration: 30 minutes

May 12 2:00 PM 🔵 Unusual traffic pattern

    3:00 PM  ✓  System handled well

    No resolution needed

May 14 2:32 PM ⚠️ Agent conflict detected

    2:40 PM  ✓ In progress

    Currently resolving...

Cost Analysis
Where is money going?

Daily Cost Breakdown:

API Calls: €50.00/day (40%)

├─ DataService: €20.00

├─ VectorDB: €18.00

└─ LLM: €12.00

Compute: €60.00/day (48%)

├─ CPU: €35.00

├─ GPU: €20.00

└─ Memory: €5.00

Storage: €14.00/day (12%)

├─ Database: €8.00

└─ Cache: €6.00

Total: €124/day (€3,720/month)

Budget: €5,000/month

Status: ✓ On track
The Philosophy: Transparency Enables Action
You can’t manage what you can’t see.

Traditional monitoring made infrastructure visible. But AI systems need a different kind of visibility.

You need to see:

Are my agents working correctly?
Are they working together?
Is the system still aligned with what I want?
What’s degrading and why?
Socratic-performance provides this transparency.

With transparency, you can act. Fix problems before they become disasters. Optimize before costs spiral. Scale before bottlenecks form.

Use Socratic-Performance for System Monitoring
The real-time monitoring system described in this post is production-ready and open source:

GitHub: https://github.com/Nireus79/Socrates
PyPI Package: https://pypi.org/project/socratic-performance/
Documentation: https://github.com/Nireus79/Socrates/tree/main/socratic-performance

Quick Start

# Install the monitoring system

pip install socratic-performance



from socratic_performance import PerformanceMonitor



monitor = PerformanceMonitor(

    system_name="MySystem",

    tracking_enabled=True

)



# Connect your agents

agent = MyAgent(monitoring=monitor)



# View dashboard

dashboard = monitor.create_dashboard(port=8000)

# Access at: http://localhost:8000



# Get current metrics

metrics = monitor.get_metrics()

print(f"System accuracy: {metrics.accuracy:.1%}")

print(f"Cost rate: €{metrics.cost_per_minute:.2f}/min")



# Get anomalies

anomalies = monitor.get_anomalies()

for anomaly in anomalies:

    print(f"⚠️ {anomaly.description}")

Full examples and documentation: https://github.com/Nireus79/Socrates

The Complete Socratic Ecosystem
Socratic-performance is part of the complete Socrates AI system (11 modules, 2,300+ tests):

Socratic-nexus: Multi-provider LLM client
Socratic-morality: Constitutional governance with 13 modules
Socratic-agents: Multi-agent orchestration with conflict resolution
Socratic-knowledge: Enterprise RAG with multi-tenancy
Socratic-learning: Self-improving agents
Socratic-analyzer: Code quality analysis
Socratic-performance: Real-time monitoring(this post)
Socratic-workflow: Workflow orchestration
Socratic-conflict: Conflict resolution between agents
Socratic-docs: Auto-documentation
Socratic-maturity: Project maturity tracking
All modules work together seamlessly. Use individual packages or the complete platform.

Available on PyPI under MIT License: https://pypi.org/user/Nireus79/

4 things writing an HTTP prober in Rust taught me

Tue, 07 Jul 2026 14:10:44 +0000

TL;DR. An uptime prober is a strange HTTP client, so I swapped reqwest for raw hyper. It needs the opposite of what a normal client gives you: no connection pool, so the cold path gets measured instead of hidden; every phase timed, even when connect fails; a hard SSRF block on the URL the user typed; and exact failure reasons instead of one flat error. Four tricks below, each worth copying even if you never build a monitor.

I built an uptime checker in Rust. Like most people, I used reqwest first. Then I dropped it and moved down to raw hyper.

Not because reqwest is slow. It is great, and for app code you should use it. But a prober is a strange kind of HTTP client. It makes one request, and its whole job is to measure that request and report exactly what happened. A normal client is built to do the opposite: make requests fast and hide the messy details. Every trick below is a messy detail I needed to keep.

Here are four things building it taught me. Each one is a habit worth copying, even if you never write a monitor.

1. A connection pool lies to a monitor

The first thing reqwest gives you is a connection pool, and it is the first thing I had to remove.

A pool keeps TCP connections open and reuses them. The second request to a host skips DNS, skips the TCP handshake, skips TLS, and comes back much faster. For app code that is a free speed boost. For a monitor it is a lie. If I reuse a warm connection, the "connect time" I report is near zero, because I did not connect, I borrowed. A user watching a slow TLS handshake slowly rise in one region would see nothing, because the pool hid the handshake.

So the connector makes exactly one request per connection, then drops it:

/// No pooling: the caller drives exactly one request over the returned
/// stream, then drops it.
pub(crate) async fn timed_connect(
    p: &ConnectParams<'_>,
    host: &str,
    port: u16,
    is_https: bool,
) -> Result<TimedConnection, ConnectError> {

Every check pays the full cost of a fresh connection, because the full cost is the thing I am trying to measure. The lesson is general: if you are timing a request, a warm pool is measuring the wrong thing. Reuse is a feature that erases exactly the numbers a monitor exists to show.

2. Put the timings inside the error

Splitting a request into DNS, TCP, TLS, and time to first byte is easy on the happy path. The interesting question is what happens when connect fails.

The naive design loses everything on failure. You get an Err and no numbers, so a hung TLS handshake looks the same as a dead DNS server. Both are just "failed." That is the worst moment to have no data.

The fix is small, and I now use it everywhere: make the error carry the phases that finished before the one that broke.

pub(crate) enum ConnectError {
    Dns(anyhow::Error),
    NoAddrs { dns_ms: u16 },
    Connect { err: io::Error, dns_ms: u16 },
    Tls { err: io::Error, dns_ms: u16, connect_ms: u16 },
}

A TLS failure still reports how long DNS and TCP took. So when a cert handshake hangs, the chart still shows DNS was 12ms, connect was 45ms, and then TLS took the rest. The shape of the error tells you what broke. You are never left guessing which layer stalled, even on the path where everything went wrong. reqwest gives you one flat Error at the very end. Here the error type holds the answer.

one probe, one fresh connection, each phase on its own clock:

  DNS         TCP connect      TLS handshake     request + TTFB
  +-----------+----------------+-----------------+---------------->
   12ms        45ms             (hung)            never reached

  TLS failed. The error still carries dns_ms=12, connect_ms=45,
  so you know exactly which layer stalled, not just "it failed".

3. The SSRF check goes after DNS, not before

This is the one most people building a fetch-any-URL feature get wrong, and it is the scary one.

A checker fetches whatever URL the user typed. That is the feature. It is also a classic SSRF engine. Someone signs up, points a check at http://169.254.169.254/latest/meta-data/, and now your prober is reading cloud credentials from inside your own network and showing them the reply. Point it at http://10.0.0.5:6379 and it is a port scanner for your private subnet.

The trap is thinking you can block this by checking the hostname. You cannot. The classic bypass is DNS rebinding: the name looks public and passes your check, but it resolves to 127.0.0.1. The block has to happen after resolution, on the real IP you are about to dial:

let addrs: Vec<SocketAddr> = p
    .resolver
    .resolve_addrs(host)
    .await
    .map_err(ConnectError::Dns)?
    .into_iter()
    .filter(|ip| p.ssrf_guard.allow(*ip))   // block loopback, private, link-local, metadata
    .map(|ip| SocketAddr::new(ip, port))
    .collect();

Resolve first, filter every resolved IP, then connect only to what survives. And because each redirect hop reconnects through this same guarded path, a Location: header pointing at 169.254.169.254 gets rejected exactly like a directly typed one. The safety sits in the single place all connections pass through, so there is nothing to forget on the redirect path. If you ever fetch a user-supplied URL server-side, this is the check that matters, and the hostname version is not it.

4. Stop grepping your error strings

When a check goes red the user gets one line, and that line has to be right. "Something went wrong" trains people to ignore alerts. "Certificate expired" tells them what to fix in five seconds.

The tempting way to produce that line is to match on error message text. It is a trap: those strings change between library versions and read differently on every OS. The message is not an API. So instead the connector downcasts to the real error and reads the reason straight out of it:

fn tls_reason(io: &io::Error) -> &'static str {
    use rustls::CertificateError;
    let Some(rustls::Error::InvalidCertificate(cert)) =
        io.get_ref().and_then(|e| e.downcast_ref::<rustls::Error>())
    else {
        return "tls";
    };
    match cert {
        CertificateError::Expired => "certificate expired",
        CertificateError::UnknownIssuer => "certificate not trusted",
        CertificateError::NotValidForName => "certificate hostname mismatch",
        CertificateError::Revoked => "certificate revoked",
        _ => "certificate invalid",
    }
}

An expired cert, an untrusted issuer, and a hostname mismatch are three different problems with three different fixes, and the user should never have to guess which one they have. The io::ErrorKind and the rustls error already carry the truth. The trick is to refuse to flatten it into a string and then re-parse the string you just made.

So, reqwest or not?

For your program? Almost certainly reqwest. Going down to hyper means you own redirects, decompression with a size cap, timeout budgeting per hop, and the request-target rules for HTTP/1 versus HTTP/2. That is real code with real edge cases, and every one is a place reqwest would have been correct for free. If I needed to fetch a config file at startup, writing this connector would be a mistake.

It is worth it here for one reason: measuring and guarding the request is the product, not a detail of it. When the request itself is the thing you sell, you want to own every millisecond and every IP it touches.

Uptimepage is open source, AGPL-3.0, and the whole probe path is in the repo: github.com/uptimepage/uptimepage. If you want the wider build story, one binary and two databases, that is a separate post. Or just start free on the hosted tier and point a check at something.

Email bombing through uptime monitoring pages

Mon, 06 Jul 2026 13:14:50 +0000

Originally published on uptimepage.dev.

I build an uptime monitoring tool. Part of my job is to attack it before someone else does. This post is about one attack that most people in this space do not talk about: using monitoring tools to flood someone's inbox with email.

I will show you how it works from the attacker side. Then I will show you how to stop it from the builder side. If you run any tool that sends email to an address a user typed in, this is your problem too.

Cover photo by Brian J. Tromp on Unsplash.

TL;DR. Any tool that emails an address you type in can be turned into an inbox flood, and monitoring tools are ideal for it: many tools, many monitors per account, and real senders that pass SPF and DKIM so the mail lands in the inbox, not spam. If you build one, cap confirm emails by target address across the whole platform, say who added the address, add a one-click stop link (RFC 8058), block role and own-domain addresses, and skip the confirm email when the user is already signed in with that address.

What email bombing is

Email bombing is simple. You flood one inbox with so many messages that the person cannot use it. Hundreds of emails in a few minutes. When the attacker uses signup forms to do it, people call it subscription bombing or list bombing. Same idea. Anti-abuse groups describe it as a denial-of-service attack on your inbox, and IT teams saw a rise in 2026.

The goal is usually not the flood itself. The flood is a cover.

Say an attacker just stole your card and bought something. Your bank sends an email or a text: "Did you make this payment?" If your inbox has hundreds of new emails in a few minutes, you will never see that one bank message. By the time you dig it out, the money is gone. The flood buys the attacker time and quiet. Security teams have watched attackers use this trick to hide real fraud.

It is not always about money. Some ransomware crews use email bombing as step one. They flood a worker with mail, then call and pretend to be the IT help desk: "We see the spam, let us help you fix it." The panic is the way in. The flood softens the target for the real attack. The US cyber agency CISA documented this play: email bombing plus a fake help-desk chat on Microsoft Teams, used by the Black Basta ransomware crew to get in. Former members kept using it through 2025.

Why uptime pages make a good weapon

Photo by Moritz Erken on Unsplash.

Here is the part that surprised me the first time I looked.

Most uptime and status tools let you add an email address for alerts. You type an address, the tool sends a "confirm your email" message, and you click a link to turn alerts on. Normal stuff.

Now look at it as an attacker. The tool will send an email to any address I type. I do not need to own that address. I type the victim's email and hit save. The confirm email goes to the victim, not to me.

One email is not a bomb. But there are many of these tools. And inside each tool I can make many monitors, each with its own alert email. So I write a small script. Sign up, add the victim's address to fifty monitors, move to the next tool, repeat. The victim gets a wave of "please confirm" emails from services they have never heard of.

The mean part is where these emails come from. They come from real companies with a good name. They pass the checks that tell Gmail a sender is real (SPF and DKIM, if you want the terms). So they land in the main inbox, not in spam. One study in 2026 found that most of these emails were not marked as spam and reached the inbox. A normal spam filter usually will not save the victim, because on paper none of it is spam.

The builder side: how not to be the tool that floods people

I will be honest. When I first tested my own tool, it had this hole. The confirm email had no real limit and no clear owner. So I fixed it. Here is what actually helped, in order.

Rate limits are the floor. Cap how many confirm emails one address can get per day across your whole platform, not just per account. If one attacker opens twenty accounts, a per-account limit does nothing. Count by the target address.
Say who and why. My old confirm email just said "confirm this address." Now it says who added it and to what: "Acme Inc added this address to a monitor." If a stranger gets one, at least they can see what is happening.
Give a fast way out. Put a one-click "this was not me, stop" link in the email. One click kills that address on your tool. No login, no support ticket. There is a standard for this, RFC 8058, so mail apps can show a stop button right in the message.
Block the obvious traps. Do not allow role addresses like postmaster@ or abuse@. They are shared mailboxes, not a person who asked for alerts. And do not allow your own company domains, because a tool that emails itself can start a mail loop.
Skip the email when you can. If the person adding the address is logged in with that same email through Google or GitHub, you already know they own it. Mark it confirmed and send nothing.

None of this is clever. It is boring plumbing. But boring plumbing is what keeps your name off the list of services used in someone's email bomb.

If you are the one getting bombed

Do not delete everything in a panic. That is exactly what the attacker wants. Somewhere in that pile there may be one real email that matters, like a bank alert or a password reset you did not ask for.

Find the real one first. Search your inbox for words like "payment", "login", "password", or your bank's name. Handle that one before anything else, and call your bank if you see a charge you did not make.
Cut it off at the source. Use the stop and unsubscribe links in the junk. Most real services put them there. It is slow work, but it is what actually reduces the flood.
Escalate if it keeps coming. Tell your email provider. Gmail and others can filter a flood once they see the pattern. And if someone calls you "from IT" right after the flood starts, be careful. That call can be part of the attack.

One last thing

I like uptime tools. I build one. But any tool that will email a stranger on request can be turned into a small weapon, and most builders never test for it. If you run one of these, spend an afternoon trying to bomb yourself. You will learn more from that hour than from any security checklist, and you will find the holes before someone else does.

More reading: how I built this monitor, and how to report abuse to us.

Dashboard Design for Incident Response

Thu, 02 Jul 2026 15:12:56 +0000

Most dashboards answer one question: Is everything okay?

During an incident, nobody's asking that.

The real question: What broke, where, and what changed?

Most dashboards fail at incidents because they were built for monitoring, not troubleshooting.

The Problem

A typical dashboard shows CPU, memory, disk, network, requests, uptime.

Useful for routine checks.

During an outage? Just noise.

You're not looking for reassurance. You're looking for evidence.

Two Different Jobs

Most teams put everything on one dashboard. That's a compromise that doesn't work for either job.

Monitoring dashboard: Is the platform healthy? SLAs being met? Resources used correctly?

Incident dashboard: What failed? When? What changed? Where do I look next?

Same tools, different purposes.

What Works During an Outage

Error rate front and center. 5XX errors, exceptions, failed transactions. Failures tell the story faster than CPU metrics.

Timeline on the graph. Mark deployments, infrastructure changes, scaling events. Most incidents start right after something changed. Make this visible in one second.

Dependency health. A healthy app talking to a dead database is not healthy. Dependencies often point to root cause faster than app metrics.

Golden signals. Latency, traffic, errors, saturation. These beat hundreds of infrastructure metrics.

Logs visible. Top exceptions, error spikes, failed endpoints. Reduce tab-switching during incidents.

Service map. Which services depend on the failing one? Visual dependency maps answer this instantly.

Alert state. Which alerts fired? Which started first? First alert usually beats alert #100 for root cause.

The Test

For every panel: How does this help me resolve the incident faster?

If the answer isn't obvious, remove it.

Example

EKS outage. Don't show cluster CPU and memory.

Show:

Failed requests by service
Pod restarts
Readiness failures
Recent deployments
HPA scaling events
Dependency latency
Top exceptions
Queue backlogs

One tells you the cluster exists. The other helps you fix it.

The Point

Monitoring dashboards tell you something broke.

Incident dashboards help you figure out why.

During an outage, only the second one matters.

DevOps #SRE #Monitoring #Kubernetes #IncidentResponse #Dashboard

Building an uptime monitor in Rust: one binary, two databases, 130K checks/sec per core

Tue, 30 Jun 2026 17:20:57 +0000

I spent the last few months building Uptimepage, an open-source uptime monitor and status page written in Rust. This post is the build story: the decisions that shaped it, the parts I rewrote, and the numbers that came out the other side.

The whole thing ships as one self-contained binary of about 23 MB, plus Postgres and ClickHouse. You can docker compose up and self-host it, or use the hosted tier. Source is AGPL-3.0 on GitHub.

Why one binary

A status page is a small surface with a lot of moving parts behind it: a scheduler, probe workers, an HTTP client, a time-series writer, an incident detector, an alerting fan-out, a web UI, a JSON API. The usual answer is a handful of services and a message bus between them.

I went the other way. One process, one binary, the same image whether it runs the control plane or a remote probe. No queue to operate, no version skew between services, no "which container is wedged" at 3am. The cost is that you have to be careful about what shares a thread and what can stall what. Most of the engineering below is about keeping those boundaries clean inside a single process.

Stack: Rust 1.95 (edition 2024), Tokio, Axum 0.8, Askama for compile-time HTML templates, HTMX 2 for partial swaps so there is no SPA framework to ship. The API stays the single source of truth because every UI mutation hits the same /api/v1/* endpoint a script would.

Two databases, on purpose

Monitors are low-cardinality relational data that gets mutated by API calls: targets, regions, channels, incidents, plans. That is Postgres. Check results are append-only, high-cardinality, and almost always queried by time range. That is ClickHouse.

Trying to force one of those into the other is where uptime monitors usually fall over. Putting billions of check results in Postgres turns every dashboard query into a sequential scan. Putting your relational config in ClickHouse means fighting its update model forever. So: Postgres 18 for the world, ClickHouse 26.3 for the firehose. Both run their migrations at process startup, so there is no separate migrator to forget.

The HTTP client I did not want to write

The first version used a popular high-level HTTP client. It worked, but a monitor is a weird HTTP workload: you connect once per target per interval, you never reuse the connection, and you care about the timing of each phase more than the body.

So I dropped down to hyper and hyper-util with rustls and built a connector that times DNS resolution, TCP connect, and the TLS handshake as separate numbers, then runs the request over hyper::client::conn and aborts the connection task the moment the body is read. Each result carries dns_ms, connect_ms, tls_ms, and ttfb_ms as distinct columns, which is what makes "it got slow but it is the DNS, not your server" a thing the dashboard can actually say.

The rewrite paid for itself. On a single core the client sustains around 130K checks/sec at saturation, roughly 7.7 microseconds per check. That was a 44 to 56 percent throughput gain over the old path. A chunk of it was a url::parse call hiding in the redirect policy that cost 7.5 percent on its own and just vanished. Two cores get to about 153K. Scaling goes sub-linear past four cores because of shared HTTP/2 connection state and the pool mutex, which is a fine problem to have for this workload.

These numbers come from a laptop and a loadtest binary, so I treat them as regression detection, not capacity planning. The headline in-process run pushed 252K requests/sec sustained at p99 393ms. The design goal is around 50K concurrent in-flight checks per node with under 50ms p99 of per-check overhead on top of the network.

One heap, not a timer per target

The naive scheduler spawns a timer task per monitor. That falls apart at fleet size: thousands of tasks, thousands of wakeups, memory that grows with the number of targets.

Instead there is a single driver task that owns one BinaryHeap<Reverse<Due>>, a min-heap keyed by the next due Instant for the whole fleet. Memory stays flat in fleet size. Each target gets a deterministic jitter offset hashed from its UUID so a thousand monitors on a 60s interval do not all fire on the same tick. Generation and sequence counters mean a re-scheduled target supersedes its stale heap entry instead of double-firing. The registry refresh that pulls config from Postgres runs on its own task with exponential backoff, so a Postgres hiccup never stalls dispatch: the scheduler keeps running on what it last knew.

Failure isolation inside one process

Because everything is one process, one bad target cannot be allowed to take down the rest. Three patterns do most of that work. Per-host circuit breakers trip when a host keeps failing, so it fails fast with circuit_open instead of tying up a worker on a timeout, then probes half-open after a cooldown. A per-tenant host throttle bulkhead caps how many checks can be in flight against one host at once; over the cap, a check is recorded as throttled and degraded rather than piling on, and it never pages. And singleflight on RDAP collapses domain-expiry checks for the same domain across many tenants into one upstream probe, with sticky last-good state so a flaky registrar does not flip the monitor red.

The worker pool itself is a task-per-dispatch gated by a semaphore sized to a max-concurrency setting, with an SSRF guard filtering resolved IPs before any connect.

Modeling time series in ClickHouse

The check_results table is a MergeTree ordered by (org_id, target_id, region, timestamp). The org_id leads the sort key so each tenant gets a sparse-index slice, but it is deliberately kept out of the partition key (partition is by day) so we do not end up with millions of partitions.

The columns are where the storage savings live:

Timestamps are DateTime('UTC') with CODEC(DoubleDelta, ZSTD(1)). Check intervals are near-constant, so DoubleDelta crushes the gaps to almost nothing.
Numeric phase columns (duration_ms, dns_ms, connect_ms, tls_ms, ttfb_ms, response code and size) use CODEC(T64, ZSTD(1)).
region and agent_id are LowCardinality(String), status is an Enum8.

Retention is per row. There is a ttl_days column with TTL timestamp + toIntervalDay(ttl_days), stamped from the org's plan at write time. A plan that buys longer history needs zero schema change: new rows just carry a bigger number.

On top of the raw table sit two AggregatingMergeTree materialized views: a per-minute rollup kept 30 days and an hourly rollup kept 13 months, both holding quantilesState for p50/p95/p99 and per-status counts. Reads route by range: anything inside 30 days hits the minute rollup, older ranges hit the hour rollup, and raw reads are capped at a 90-day span. A dashboard asking for "last 24h p99 latency" never touches a raw row.

Regional probes without a second brain

You can run probes in multiple regions, but I did not want each region to be its own little system. So an agent is the same binary in agent mode, running as a stateless probe with no database, no web, no alerting. Adding a region adds execution capacity, never a second control plane.

An agent pulls its region's config from the control plane with ETag/304 handling, serves its last-known config if the control plane blips, and pauses if its token is revoked. It ships results back in batches that reuse one UUID across retries so a lost ack cannot double-count. Region and agent identity are derived server-side from the bearer token, never sent in the payload, so a probe cannot claim to be somewhere it is not. A separate long-poll loop handles interactive "check now" so a button press in the UI runs on a real remote probe within milliseconds.

Region is the partition dimension end to end, so every read can slice by region: per-region latency series, per-region incident scope, "down in Singapore, up in Helsinki."

Incidents as a follower, not a gatekeeper

The incident detector is a background task that follows the check_results stream and writes into the Postgres incidents table. The rule I held to: it never touches the hot write path, never gates check execution, and never produces alerts directly.

The detection itself is boring on purpose: two or more consecutive unhealthy results with no open incident opens one, two or more consecutive healthy results closes it. A small flap threshold absorbs single-result blips. The cross-tenant walk is keyset-paginated so memory stays bounded as tenants grow, and a unique index on open incidents resolves the race when two ticks try to open the same one: only the winner pages.

Opening or resolving fires a non-blocking signal to the escalation engine, which does repeat-until-acknowledged paging on a per-monitor cadence across about fourteen transports (Slack, generic signed webhooks, Telegram, PagerDuty, ntfy, Pushover, Discord, email, and more), with sharded per-incident locks so a reconcile and an inbound signal can never double-page the same episode. Channel secrets are sealed at rest with AES-GCM and never echoed back.

Automation as a first-class surface

Because the API is the single source of truth, the rest came almost for free:

A self-describing OpenAPI 3.1 spec with Swagger UI.
An official Terraform provider so you can manage monitors and notification channels as code.
An MCP server so an LLM client can query your monitors and incidents over OAuth 2.1, scope-gated and audited, with per-action confirmation on the few write tools.

Where it is

Uptimepage is live, free to start with no card, and AGPL-3.0 open source. The core is not paywalled: checks, status pages, subscribers, the API, and every alert channel are in the free tier. It monitors itself and serves its own status badges from the running binary, which is the most honest dogfood I could think of.

Site and hosted tier: https://uptimepage.dev
Source: https://github.com/uptimepage/uptimepage
More build notes: https://uptimepage.dev/blog

Happy to answer anything about the Rust internals in the comments.

Conversation monitoring for voice agents: the six metrics that matter

Tue, 30 Jun 2026 07:35:54 +0000

The six conversation metrics that catch what voice agent dashboards miss

Most teams have shipped voice agents and most are not happy with them. In Hamming AI's State of Voice AI 2026, 87 percent of companies have deployed voice agents but only 12 percent are satisfied with the quality. The gap is not latency or uptime. It is conversation-level quality: contradictions across turns, missed intents, escalations that fire too late, and sentiment that slides before a hang-up. Standard dashboards do not surface any of that.

The dashboards from a couple of years ago stopped at three numbers: latency, completion rate, and sentiment. Those catch the obvious failures, the calls that time out, do not resolve, or end with an openly angry customer. What they miss is the long tail where every number reads green and the experience still degrades. These six conversation-level metrics catch that long tail. You do not need all six on day one. Start with completion and turn coherence, then add the rest as call volume grows.

Turn coherence. Does the assistant stay consistent across turns. It catches a fact confirmed in turn 3 and contradicted in turn 7, a tool result that comes back but never gets used, and a reference like "the second option you mentioned" the assistant has forgotten. Each turn reads fine on its own; the failure is in the connective tissue between them, so a single-turn check never sees it. Score it with a multi-turn coherence rubric over the full message history.
Intent confidence. Did the system identify what the caller actually asked at the entry point. It catches a vague opening routed to a default flow, a mistranscribed accent that feeds the classifier a garbled input, and two requests in one turn where only one gets handled. Intent taxonomy is org-specific, so this is usually a custom check, and it pairs with audio-transcription scoring when STT drift is the real cause rather than the classifier.
Completion rate. Did the call resolve the caller's goal. Track two versions: customer-perspective resolution (did they get what they came for) and agent-perspective task completion (did the assistant do the right thing). The split is the signal. A correct out-of-policy refusal scores high on task completion and low on resolution, which tells you the failure was policy-induced, not a capability gap. The delta between the two is more useful than either alone.
Sentiment trend. The slope across turns, not the static end-of-call sentiment. It catches a caller who enters neutral, sours by turn 3, and hangs up before the agent notices, and it credits the agent that de-escalates a frustrated caller back to positive. Score sentiment per turn and alert on negative slopes that cross a threshold; on outbound calls a falling slope often predicts a hang-up a couple of turns out.
Escalation triggers. When and why a call hits a policy or scope boundary that needs a human. It catches escalate-out-of-caution, where the agent could have handled it but handed off anyway, and the reverse, where the agent should have escalated and tried to muscle through instead. Pair a refusal-justification check with a handoff-quality check that scores whether the human got enough context to avoid re-collecting the basics.
Repeat-question signal. Did the caller ask the same thing twice because the first answer was not useful. This is the one that catches silent CSAT decline: intent is right, coherence is high, completion fires positive, but the caller had to ask three times. Score it with a custom signal that compares semantic similarity between the caller's turns and flags pairs above a threshold.

The metrics are most useful in combination, not one at a time. Group calls by the set of verdicts rather than by a single score. Low completion plus high repeat-question plus neutral sentiment is the silent-degradation pattern worth fixing. High completion plus declining sentiment plus one escalation is usually recovered-after-friction and fine to leave alone. Clustering the low-scoring calls into named issues, each with a root cause and a fix, is what turns six metrics into an actionable backlog instead of six dashboards to scan.

A few things that go wrong in practice. Running all six on 50 calls is noise, so start with two and add the rest once each score is stable. Treating sentiment as a single end-of-call label throws away the slope, which is the part that carries the signal. Conflating an escalation trigger (the agent decided to hand off) with an escalation outcome (the human resolved it) hides a real failure cluster. And skipping repeat-question because it needs a custom evaluator drops the highest-signal metric for quiet CSAT decline.

The point is that the 12 percent satisfaction gap lives at the conversation level, and infrastructure dashboards never reach it. Completion and turn coherence get you started; the other four cover the failures that only show up once you are looking at the whole conversation rather than a single turn.

Two device stories, one URL: why averages lie

Mon, 29 Jun 2026 19:35:23 +0000

One checkout URL can look healthy in a blended score while mobile passes and desktop fails. Portfolio averages hide the same split. Here is how we report paired device data instead.

The account manager pasted a portfolio screenshot into Slack: average Core Web Vitals pass rate across twelve client sites was 81 percent, up three points from last month. Engineering opened the same week’s export and found checkout on one retailer at 94 on mobile and 41 on desktop, both from the same URL, both in field data. The average looked like progress; the paired rows looked like a revenue problem waiting for a marketing campaign to send desktop traffic to a route that could not convert.

Same URL, two device stories. That is normal for Core Web Vitals, not a tooling glitch. Google’s own field datasets already treat mobile and desktop as separate populations: the Chrome UX Report (CrUX) stores experiences by form factor, and Search Console’s Core Web Vitals report labels each URL group independently for mobile and desktop. The gap is not that vendors lack the split; it is that internal dashboards and client decks collapse the split back into one comforting number.

Why mobile and desktop Core Web Vitals diverge on one URL

Largest Contentful Paint often tracks different elements after responsive layout: a hero image on desktop and a text block or product tile on mobile. Interaction to Next Paint reflects touch versus pointer paths, menu density, and how much main-thread work survives on lower-end hardware. Cumulative Layout Shift spikes when ads, banners, or consent UI appear only under certain viewports. None of that shows up if you only store a single “page score” per URL.

Lab runs make the split obvious when you compare mobile and desktop emulation on the same template. Field data from CrUX does the same by form factor, which is why PageSpeed Insights shows separate rows and accepts strategy=mobile or strategy=desktop in its API. Google’s web.dev guidance on lab versus field data adds a further wrinkle: Lighthouse returns the same LCP candidate every time, while field data on one URL often spans several LCP elements depending on viewport and scroll depth. Stopping at whichever row looks better, or averaging them for a client deck, hides that variance. For metric definitions and fix paths, our LCP, INP, and CLS guide on the Watcher blog stays the reference.

Where blended scores and portfolio averages mislead

We see three patterns that produce misleading comfort in dashboards and decks.

Single-number client reports. A weekly PDF that lists one LCP or one “performance score” per URL without a form-factor column implies one experience. Sponsors assume the number covers how most users browse; in our experience mobile share is often higher on the URLs that matter, but desktop still carries high-value sessions on B2B and admin-heavy flows. Public CrUX aggregates show the same skew: mobile origin pass rates typically trail desktop by several points, so a blended headline can flatter phone-heavy sites even when mobile is the weaker side.

Portfolio rollups. Agency dashboards that average pass rates across domains treat a mobile fail on checkout and a desktop pass on a blog post as partial credit. The rollup climbs while the conversion path stays broken. We prefer worst-of or paired counts: how many priority URLs pass on both mobile and desktop, not how many pass on either. Search Console labels URL groups separately per device; a group can be Good on mobile and Need improvement on desktop. Exporting one “site health” percentage without that dimension is how an eighty-one percent portfolio average coexists with checkout at forty-one on desktop.

“Good enough” thresholds on the stronger device. Teams sometimes ship when desktop clears a budget while mobile INP stays red because desktop was the default Lighthouse profile in continuous integration. The deploy log shows green; support tickets mention “laggy on phone.” A related trap is treating PageSpeed Insights origin-level field data as proof that a specific URL is fine: when URL-level CrUX samples are thin, PSI falls back to origin aggregates, which can mask a single high-traffic template that fails on one form factor only.

Each pattern shares the same fix: store and show mobile and desktop as separate series for the same URL, then alert on the weaker side.

How we report paired device data to clients and internally

We stopped leading with averages in QBR slides. The first table for priority URLs lists mobile LCP, INP, and CLS beside desktop values for the same route, with a column for “both pass” rather than “either pass.” When one side lacks field data, we label it lab-only and do not merge it into a blended pass/fail. For alerts, device-specific thresholds beat a single budget line: a regression that hits mobile INP on checkout triggers the same channel as a desktop LCP fail on the homepage, but the message names the form factor so triage does not waste a day reproducing on the wrong profile.

Internal stand-ups that read “mobile fail / desktop pass” in the subject line have saved us from closing tickets too early, because the split is visible before anyone opens the chart. When a client asks for one headline number, we give them a paired summary: “ten of twelve money URLs pass on both mobile and desktop; two fail on mobile only.” That sentence is less flattering than an eighty-one percent average, and it is the one that gets engineering budget for the checkout script audit. Schedules, budgets, and portfolio views are in mobile versus desktop Core Web Vitals monitoring on the Watcher blog; Watcher holds the checklist, while Hashnode holds the reporting argument.

What to change in monitoring this week

If your stack stores one row per URL, add form factor to the grain before the next client send. Re-run baselines on five conversion URLs with mobile and desktop explicitly selected in PageSpeed Insights or your monitoring product, and note where field data exists for each. When PSI shows origin-level field data only, tag those rows as origin fallback so nobody mistakes them for URL-level proof on both devices. Rewrite one alert policy so a mobile-only breach on a tagged “money URL” cannot be drowned out by desktop green scores on the same path, and add a “both pass” metric for priority tiers if portfolio reporting is automated; the first time that metric drops while the average rises, you will know the blended number was hiding a split.

Next step

Pick one checkout or lead-form URL and publish a paired mobile/desktop baseline internally before the next client-facing report goes out. Use the mobile and desktop monitoring guide for schedules and budgets, and the LCP, INP, and CLS explainer when someone asks what the columns mean. Two device stories on one URL are expected; one averaged story is the number that usually costs you the argument.

Originally published on Hashnode.

First two weeks of a performance onboarding (realistic scope)

Sun, 28 Jun 2026 16:03:42 +0000

New monitoring clients often expect fixes in week one. Realistic onboarding scope is narrower: access, URLs, owners, baseline, and thresholds before anyone promises remediation.

A new client signed the monitoring retainer on Monday and asked on Wednesday when checkout would be green in Lighthouse. Fair question: the SOW mentioned Core Web Vitals, and their last agency had sent a PDF full of red bars.

We had not finished agreeing which checkout URL counted, who owned first response on alerts, or whether mobile field data even existed for that template yet. The work was not missing; the calendar was wrong.

Most onboarding failures we see are scope failures, not skill failures. Teams either treat week one like a mini audit with twenty remediation tickets, or they spend fourteen days in tooling setup and call it done without a baseline the client can repeat. Below is the two-week frame we use with agencies: what we promise, what we defer, and how it lines up with the longer checklists on our main blog.

What goes wrong when onboarding scope is vague on day one

Without a written two-week boundary, three collisions are common. Sales language drifts into delivery: "We will monitor performance" becomes "we will improve Core Web Vitals" in the client's head while engineering still waits for staging access. The URL list balloons when someone exports a sitemap, adds every campaign landing page from the last three years, and schedules tests before anyone marks which routes actually convert. Remediation starts before measurement is trusted when a developer fixes hero images on the homepage while checkout still lacks field data and nobody has agreed on alert severity.

Procurement and delivery can both be happy if week one and week two have named outputs. Those outputs are not "site is fast"; they are "we know what we watch, who acts, and what broke first."

Week one: what to lock before you run PageSpeed tests

Week one is contracts and context, not optimisation. We aim to close five agreements in writing (email is fine):

Primary domain and environments (production first; staging only if the client expects pre-release checks).
Five to ten priority URLs: homepage, main conversion path, one high-traffic template, and commerce or form routes if they exist.
Mobile and desktop coverage (both, unless the client has a documented mobile-only audience).
Alert recipients and a named first responder (one person, not a channel full of @here).
Reporting rhythm: weekly internal scan, monthly client-facing review, and who owns each.

If access is slow, week one still produces a blocker log: who owes credentials, which firewall ticket is open, which analytics consent rule blocks synthetic tests. That log is a deliverable, and it prevents week two from quietly slipping to week four.

Our site audit checklist for onboarding new clients into performance monitoring has the full tick list for access, URL inventory, and handover. On Hashnode we are only arguing for order: scope and owners before quotas and schedules.

Week two: baseline, thresholds, and the first client-safe summary

Week two is when tools earn their place. Run initial tests on the priority URL set from week one, record LCP, INP, and CLS for mobile and desktop where data exists, and mark pages with no field data so nobody interprets a lab-only score as a production guarantee.

Set conservative thresholds on those same URLs. Tight budgets on day ten usually mean alert noise by day seventeen, so we would rather start slightly loose, watch one full weekly cycle, and tighten after we see variance.

Wire alerts to the routes agreed in week one: one internal channel for regressions that need same-day eyes, and a separate, slower route for client-visible summaries if the contract includes them.

Draft a short "what we monitor and why" note the account lead can send without engineering in the thread. Three paragraphs cover which URLs, which metrics, what happens when a threshold breaks, and what is explicitly out of scope for the monitoring retainer.

End week two with three actions from the baseline, each with an owner and a due date. They can be investigative ("confirm whether tag X loads on checkout") rather than shipped fixes; the client should see prioritisation, not a promise that all three close before month end.

What we deliberately defer past the first two weeks

Saying no early protects the retainer later. Full template coverage across hundreds of URLs belongs in month two unless the contract priced it; CI gate design, CrUX versus lab reconciliation workshops, and third-party script triage across every tag manager container are real work, but they are not week-two work.

We also defer client-facing dashboard polish, because a screenshot of a green score without context creates the same false confidence as the PDF they received from the previous vendor. Remediation sprints stay separate from monitoring setup unless the SOW merged them with hours attached; mixing the two in fortnight one is how agencies absorb unpaid engineering.

If the client pushes for fixes immediately, we point to the three actions from the baseline and offer a change order for implementation. Monitoring onboarding is complete when everyone can answer what broke first, who got pinged, and what we do next.

How the two-week frame maps to ongoing CWV operations

Fortnight one is intake; the recurring habit starts afterward. After week two, we hand teams the Core Web Vitals monitoring checklist for agencies for weekly scans, monthly deep dives, and deploy checks. That document assumes monitoring is live and scoped. Without the two-week frame, teams treat the checklist like a one-off audit and abandon it when the first sprint ends.

In practice the split looks like this:

Phase	Question it answers	Typical owner
Weeks 1–2 (this post)	What do we watch, who responds, what is the baseline?	Delivery lead + technical lead
Week 3 onward (Watcher checklist)	What do we do every Monday and every month?	Account manager + engineering

Agencies managing several clients benefit from the same shape every time. Portfolio onboarding is repetitive; the client's internal politics are not. Reusing the two-week boundary makes it easier to compare sites without pretending each onboarding is bespoke heroics.

What to put in the client email at the end of week two

We keep the end-of-fortnight email short enough to forward inside the client's organisation:

Confirmed URL list (with one-line business reason per URL).
Baseline table for LCP, INP, CLS on those URLs (note where field data is missing).
Alert routes and first responder name.
Top three findings with owners (not necessarily fixes).
Explicit "starts week three" items: expanded coverage, remediation quotes, workshop dates.

That email sets expectations better than another Lighthouse export: procurement sees deliverables, engineering sees assigned tickets, and sponsors see that monitoring is operational rather than a slide in the pitch deck.

Next step

If you are onboarding a client this month, write the two-week outputs before you buy more test quota. Use the site audit checklist for ticks inside each week, then move into the agency CWV checklist once alerts have survived a full weekly cycle without being muted. Realistic scope is not pessimism; it is how monitoring retainers survive contact with real clients.

Originally published on Hashnode.

When to Use Synthetic vs Real User Monitoring for Performance

Sun, 28 Jun 2026 08:53:55 +0000

Performance monitoring conversations often collapse two different questions into one: "Are we fast in a controlled test?" and "Are we fast for real visitors?" Synthetic monitoring answers the first; real user monitoring (RUM) answers the second. Teams that pick only one layer usually discover the gap after a deploy, when lab scores look fine but conversion drops on mobile, or when CrUX turns red a week later. What follows is a practical split for web teams and agencies: what each method measures, when each is worth the operational cost, and how scheduled PageSpeed monitoring with CrUX context fits beside first-party RUM without forcing a rip-and-replace.

What is synthetic monitoring for web performance?

Synthetic monitoring runs automated tests against URLs you choose, on a schedule you control, in a lab environment, loading each page with a fixed device profile, simulated network, and throttled CPU (for example Lighthouse mobile emulation through PageSpeed Insights) so results stay comparable run to run.

Typical outputs include:

Core Web Vitals from the lab run (LCP, INP, CLS where Lighthouse reports them).
Lighthouse Performance score and supporting timings (FCP, TBT, Speed Index).
Diagnostics and opportunities (render-blocking resources, image sizing, third-party cost).
Chrome User Experience Report (CrUX) percentiles when Google publishes field data for that URL and strategy.

Synthetic monitoring is not a one-off PSI paste into a browser tab. It is the same class of measurement, repeated and stored so you can trend, alert, and report. Our PageSpeed Insights vs automated monitoring guide explains why spot checks fail once you manage more than a handful of URLs. Agencies use synthetic schedules to cover client portfolios (homepage, pricing, lead forms, checkout paths, and campaign landers), usually on both mobile and desktop strategies.

What is real user monitoring (RUM)?

Real user monitoring collects performance data from actual browser sessions on your site. A JavaScript snippet, or tag manager injection, records timings, navigation, and often custom attributes (logged-in state, plan tier, locale) as users browse, which is why RUM reflects real networks, devices, cache states, ad blockers, and consent banners rather than a fixed lab profile.

First-party RUM products (PageVitals, DebugBear, SpeedCurve, Datadog RUM, and others) let you:

Segment by page, device, geography, and custom tags.
See distributions and percentiles from your traffic, not only Google's aggregate.
Monitor routes CrUX rarely covers well (authenticated app shells, low-volume templates).
Tie performance to business events when you connect analytics or product data.

That traffic mix is also why RUM is harder to compare week to week unless you control for seasonality and campaigns. For metric definitions shared by both lab and field programmes, start with What Are Core Web Vitals? and LCP, INP, and CLS explained.

CrUX vs first-party RUM: both are field data, but not the same thing

Teams comparing RUM vs synthetic monitoring sometimes treat CrUX as "free RUM," yet it is field data without session-level detail, and that distinction matters when you choose tooling or write a client report.

Source	What it is	Strength	Limit
CrUX	Google's aggregated Chrome experience for URLs with enough traffic	Aligns with Search Console and SEO reporting; no snippet required	Rolling window; URL-level; limited segmentation
First-party RUM	Your snippet on your origin	Session paths, custom tags, logged-in views	Consent, tag governance, maintenance per site
Synthetic lab	Scheduled Lighthouse-style runs	Same-day, repeatable, works on staging	Simulated users, not real cache or ad mix

PageSpeed Insights and many monitoring tools show CrUX beside lab results, which helps public marketing URLs, but that pairing does not replace RUM when stakeholders ask what paying customers on 4G in Germany experience on a specific checkout step. Mobile vs desktop Core Web Vitals monitoring matters here as well: CrUX and lab both split by strategy, yet RUM segmentation is where device and connection reality show up for product decisions.

Synthetic vs RUM: comparison for web teams

Question	Synthetic (scheduled lab)	First-party RUM
Needs a script on the site?	No	Yes
Works on staging / pre-release URLs?	Yes	Usually no (no real traffic)
Comparable numbers every run?	Yes	Depends on traffic mix
Catches deploy regressions same day?	Yes, if scheduled	Yes, with enough sessions
Good for SEO / CrUX alignment?	Lab + CrUX in PSI output	CrUX separate; RUM is your own dataset
Portfolio scale without per-site snippets?	Favourable for agencies	Harder (consent and client approvals)
Multistep flows (checkout, signup)?	Per-URL unless tool supports journeys	Strong when tagged correctly

Neither approach wins outright because they answer different risk questions: synthetic shows whether yesterday's release broke lab thresholds on URLs you list, while RUM shows whether real users felt it across segments you care about.

When synthetic monitoring is the right choice

Choose synthetic as your primary web performance layer when you manage many sites or URLs and need one schedule, one history store, and one alert policy without installing tags on every client origin. That is the common agency constraint on marketing sites, brochureware, and public app URLs where the client will not approve another analytics script.

Synthetic also fits when you need repeatable baselines before and after a change. Lab runs on the same URL, strategy, and daypart make comparisons honest, whereas field data drifts when traffic mix shifts. The same logic applies to low-traffic or new pages where CrUX may show "no data" for months while scheduled lab runs still return LCP, INP, and CLS every time.

Use synthetic for staging, preview, or password-protected builds before production, because RUM cannot run where users do not visit yet. Pair schedules with performance budgets and alerts when you need a steady measurement source, and with portfolio reporting when clients want trend lines instead of a PDF assembled from manual PSI tabs; Automated vs manual PageSpeed testing shows the time cost when that layer is missing. Synthetic monitoring still does not replace backend APM, error tracking, or session replay: it measures web URLs in the browser lab, plus CrUX where published, not API latency inside your cluster.

When real user monitoring is the better fit

Choose first-party RUM when authenticated experiences dominate the product. Dashboards, account settings, and checkout after login often lack stable CrUX samples, and RUM with custom tags is how you prove INP on those routes. The same applies when stakeholders require segment-level proof by plan tier, locale, experiment bucket, or campaign source, because CrUX will not break down your traffic that way.

RUM also belongs in the stack when you already operate observability tooling and need web vitals beside traces and errors, when CI/CD or feature flags need session-level validation after release, or when multistep journeys must be measured as flows rather than isolated URLs. Legal and client approval for another script varies by estate, and if RUM is already installed you can add synthetic for URLs RUM under-samples or for pre-production checks, while teams with synthetic schedules add RUM where lab and CrUX disagree with support tickets or conversion data.

How agencies combine synthetic and RUM across client portfolios

Agencies rarely get a single answer across thirty clients, but a workable default is to run scheduled synthetic on mobile and desktop for public marketing and lead-gen URLs, read CrUX from the same test results, and keep budgets and alerts in one place without a snippet on every domain. For flagship SaaS or ecommerce clients with RUM already installed, keep their RUM vendor for logged-in app routes and use synthetic for the URLs the retainer covers and for prospects during performance audit outreach; when clients refuse third-party scripts, synthetic plus CrUX is often the entire programme, so say that explicitly in proposals rather than implying session replay is included.

In client reporting, separate slides for lab trends on priority URLs and for RUM segment summaries from the analytics team. Blending both into one green chart invites arguments about methodology. For tool selection by client shape, see comparing PageSpeed monitoring tools for agencies and PageVitals vs Apogee Watcher when RUM and CI/CD are part of the brief. SaaS product teams face a similar split at single-company scale; our SaaS performance monitoring guide covers URL priorities without repeating the full decision tree here.

How scheduled PageSpeed monitoring fits beside RUM vendors

Apogee Watcher is built for scheduled synthetic monitoring across many sites: PageSpeed Insights-backed lab runs, CrUX when Google returns field data, vitals and optional FCP/TBT budgets, portfolio alerts, and multi-tenant organisations for agencies. We do not ship first-party RUM; we sit beside RUM, APM, and Lighthouse CI gates rather than replacing them, so when a client needs session-level RUM, PageVitals, DebugBear, or an observability vendor still belongs in the stack.

Watcher is often the better primary monitor when you run many client domains under one subscription without per-site RUM rollout, when automated page discovery should keep new templates on the schedule without spreadsheet maintenance, when retainers promise ongoing CWV oversight on public URLs rather than full product instrumentation, or when you want prospecting and free evaluation via the domain PageSpeed check before a client signs. Configure mobile and desktop strategies per URL, set budgets, route alerts, and review history in the app, and use RUM elsewhere when the question requires your traffic, your tags, and your logged-in routes.

FAQ

What is the difference between RUM and synthetic monitoring?

Synthetic monitoring runs controlled lab tests on chosen URLs on a schedule. RUM records performance from real user sessions in the browser, usually via a script you install. Synthetic is repeatable and works without site traffic; RUM reflects actual devices, networks, and behaviour, which is why most mature teams treat the two as complementary rather than interchangeable.

Is CrUX the same as RUM?

No. CrUX is Google's aggregated field dataset for URLs with sufficient Chrome traffic, while RUM is first-party data you collect from your visitors. Both describe real loads in the field sense, but CrUX is not session-level RUM and cannot replace custom segmentation on your origin.

Should I use synthetic or RUM for Core Web Vitals?

Use both when you can. Synthetic and CrUX (via scheduled PSI runs) cover public URLs, staging, and low-traffic pages; first-party RUM covers segmented and authenticated experiences CrUX does not explain. SEO and client reporting often lean on CrUX, while product teams often lean on RUM for route-level proof.

Do agencies need RUM on every client site?

No. Many retainers cover public marketing and key templates where synthetic plus CrUX is enough, and you add RUM when the contract includes logged-in app routes, the client already owns a RUM tool, or stakeholders require session-level segmentation that CrUX cannot supply.

Can synthetic monitoring replace RUM?

Not fully. Synthetic answers whether lab thresholds breached on listed URLs, while RUM answers what users experienced broken down by segment, so use synthetic for portfolio scale and pre-release checks and RUM where traffic truth and custom tags matter.

Does Apogee Watcher include RUM?

No. Watcher schedules PageSpeed lab tests and shows CrUX where available, and you should layer it with your RUM or observability vendor when you need first-party session data on authenticated or heavily segmented routes.

How often should synthetic tests run compared to RUM sampling?

Synthetic frequency is a schedule you set (daily, weekly, or higher on risky URLs), while RUM sampling is continuous or rate-limited by your vendor. Match synthetic cadence to release risk, use RUM for always-on field truth where installed, and see test frequency and priority across a portfolio for scheduling patterns across many sites.

Pick synthetic when you need repeatable lab coverage at portfolio scale without a snippet on every domain, and RUM when session segments and authenticated routes drive decisions. Most mature stacks use both, with clear owners for each signal, rather than debating which single tool should own performance end to end.

Start a free Apogee Watcher account to schedule mobile and desktop tests with vitals budgets across your sites, or run a free domain PageSpeed check to baseline a URL before you add RUM or expand monitoring.

OpenTelemetry-native, all-in-one, simple

Mon, 15 Jun 2026 23:34:33 +0000

The standard observability stack:
Grafana + Loki + Tempo + Prometheus. Four services to deploy, four configs to learn, dashboards to set up before you see a single trace.

What I actually needed:
One place to see traces, logs, and metrics from my services.

What I built:
One container. docker compose up. Done.

I work with .NET microservices and at some point I went through the full Grafana setup. And look, it works fine, it's a solid stack. But for what I needed it was way too much. I spent more time configuring things than actually looking at my data. The .NET Aspire dashboard is on the other end: you get a nice UI, but nothing is persisted. Close the browser and everything's gone. No customization either.

So I built something in between.

What it is

OtlpDashboard is a single container that receives OpenTelemetry data and stores it. It speaks standard OTLP on port 4317 (gRPC) and 4318 (HTTP/Protobuf). If your app already exports telemetry, you just point it here. No collector needed (works behind one too if you already have it).

Storage is SQLite by default, so zero config, just a file. You can switch to PostgreSQL or SQL Server if you want.

The frontend is a Vue 3 / Nuxt 4 SPA served on the same port. You get a grid of widgets you can customize: stat cards, line charts, sparklines, gauges, heatmaps, log streams, trace viewers, and a few more. Auth is on by default.

How AI helped me ship this in under a month

I want to be upfront about this. I used Claude Code a lot, especially for the frontend. I'm a backend dev. I know .NET, EF Core, gRPC. But Vue and Nuxt? Not really my thing. Without Claude Code I would have spent days stuck on CSS and component wiring. Instead I could focus on what I actually care about: the OTLP ingestion, the storage layer, how widgets work.

All the architecture and decisions are mine. The AI did the typing.

And honestly, I think that's the best use of these tools right now if you're a solo dev. They don't replace you. They cover the parts where you're slow, so you can ship something that would normally need a small team.

Some stuff I like about how it turned out

You can install widget packs from Git repos. There's a pack.json that bundles widgets and dashboards together, so if you have a team you can share a curated setup.

There's also a spec engine for building custom widgets with Mustache templates. It's sandboxed with DOMPurify, no JS eval. I wanted people to be able to build their own cards without opening security holes.

Retention is configurable per signal. Set 7 days for traces, 30 for metrics, whatever you want. Old data gets cleaned up on its own.

And there's an MCP server at /mcp if you use Claude Code or similar tools. You can query your telemetry from the LLM directly. I didn't expect to use this much but it's actually pretty handy when debugging.

Try it

Quickest way:

docker compose up --build

This starts everything including a sample workload that generates real data, so you're not staring at an empty dashboard.

Or just check the live demo. It's a static build with mock data, login accepts any password: Live demo

The repo is here: GitHub

It's GPLv3. If you try it out let me know what you think, I'm curious what's missing or broken.

From Reactive to Proactive: How Smart Alerts Change Performance Monitoring

Mon, 01 Jun 2026 11:05:46 +0000

An account manager forwarded us a Search Console screenshot at 4:47 p.m. on a Friday. LCP on the homepage had been in the red for eleven days. Engineering had not been in the thread because nobody was assigned to watch the numbers until a sponsor noticed.

That is reactive performance monitoring in practice: fast tools, slow humans, and a calendar that only opens when someone else raises the alarm.

Proactive monitoring is the opposite shape. Scheduled lab tests run whether or not anyone remembered to open PageSpeed Insights. Performance budgets turn history into rules. Alerts fire when those rules break, with enough context to triage and enough restraint that people still read them. The word “smart” in product copy usually means the second part: alerts that match how teams actually work, not a louder siren.

This post is for agency leads who already automate some tests but still learn about regressions from client inboxes. We walk through what reactive looks like, what proactive requires, and how to wire alerts so they change behaviour instead of training everyone to mute the integration.

What reactive performance monitoring looks like on agency teams

Reactive does not mean “no tools.” Most agencies we speak with have PageSpeed Insights bookmarks, a monthly spreadsheet, and a standing item on the retainer review slide. Reactive means the signal arrives after the business event you wanted to prevent.

Common patterns:

Spot checks after deploy. Someone runs PSI on staging, ships, and assumes production matches until a complaint arrives.
Quarterly or monthly report pulls. CrUX and Lighthouse exports go into a deck. Useful for storytelling; too slow to catch a plugin update that moved INP on Tuesday.
Client-led discovery. The sponsor pastes a screenshot, asks “is this new?”, and the team scrambles to reproduce on a laptop that may not match the lab conditions you use in production monitoring.
Alerting without policy. Webhooks fire on every metric twitch. By week three the channel is muted and the organisation is back to reactive, only now with guilt about ignored notifications.

Each pattern shares a cost: engineering time spent proving what happened instead of fixing it, and account teams explaining delays that sound like neglect even when the underlying fix is straightforward.

Reactive monitoring also scales badly across portfolios. At five client sites you can hero your way through manual checks. At twenty-five, the maths stops working unless you hire a role whose entire job is tab discipline. Scaling a digital agency without headcount is really a story about moving discovery, testing, and notification into infrastructure that does not grow linearly with site count.

What proactive performance monitoring means in practice

Proactive monitoring answers a narrower question on a fixed schedule: did any URL we care about cross a line we agreed in advance?

That sentence has three concrete parts.

Scheduled tests replace memory. Mobile and desktop lab runs hit the same URL set on a cadence you can defend in a statement of work: daily for commerce, weekly for brochure sites, and so on. Results land in a time series so “was it always this slow?” becomes a chart, not a debate.

Performance budgets replace vibes. Budgets store maximum LCP, INP, CLS, score floors, and supporting metrics per site and strategy. They are the contract line between “watching numbers” and “doing something when numbers fail.” Our complete guide to performance budgets for web teams and the performance budget thresholds template give starting numbers; you still tune per client.

Alerts connect breaches to people. Without alerts, budgets are a dashboard ornament. With naive alerts, budgets become spam. Proactive monitoring needs the third layer to be designed as carefully as the first two.

Proactive is not the same as “more notifications.” It is earlier signal with clearer ownership. The client email on Friday becomes a digest on Wednesday that engineering already acknowledged.

Automated performance alerts: budgets, cooldowns, and digests

Teams that call alerts “smart” usually mean a bundle of behaviours that protect human attention:

Behaviour	What it prevents
Budget-driven triggers	Alerts on metrics nobody agreed to care about
Cooldowns	The same homepage LCP breach paging you every run until someone ships
Digest-style delivery	Forty separate messages for forty URLs on one bad deploy
Resolution when metrics recover	Stale “open” fires that no longer match production
Severity and routing policy	Critical meaning “drop everything” instead of “anything red”

Cooldowns deserve a full sentence because agencies misunderstand them. A cooldown is not permission to ignore a sustained failure. It is how you keep the first breach credible while the fix is in progress. If checkout LCP fails Monday and still fails Tuesday, you still need visibility; you do not need forty identical Slack posts that teach the channel to tune out.

Digest-style email is the pattern we ship today in Apogee Watcher: one mail per site per budget-check run, worst pages first, totals for how many URLs breached, and a cap on detail rows so the message stays scannable. The product spotlight on performance budgets and email alerts walks through what lands in the inbox after each scheduled run. Slack and additional channels are on the roadmap; the policy thinking below applies regardless of transport.

Smart alerts also fail when routing is vague. If #general receives every severity, proactive monitoring reverts to reactive noise. The fix is usually a one-page policy, not a new tool. Our Slack alert policy template for web performance teams defines channels, P1–P4 examples, and cooldown language you can paste into client documentation.

Reactive vs proactive monitoring: what changes for agencies

Dimension	Reactive	Proactive
Trigger	Client email, ranking drop, manual curiosity	Scheduled test + budget breach
Evidence	Screenshot, single PSI run	Time series, URL list, metric vs threshold
Timing	Days to weeks after change	Next run after deploy (hours to days, per cadence)
Owner	Whoever saw the message first	Named triage role in alert policy
Client conversation	“We are investigating”	“We detected X on URL Y; fix in progress”
Scale limit	Heroics and spreadsheets	Automation with policy guardrails

The shift is operational, not philosophical. You do not need a new performance practice religion. You need infrastructure that runs when people are in meetings, plus rules that tell them which meetings to skip when a digest arrives.

Agencies selling retainers get a second benefit: proactive proof. A digest with timestamps and agreed thresholds is easier to defend than a Lighthouse capture taken after the client complained. That lines up with why agencies need automated performance monitoring in 2026 and with prospecting workflows that lead with evidence rather than generic pitches.

How to move one client site from reactive to proactive

Pick the noisiest or highest-revenue property first. A pilot beats a big-bang policy rewrite.

1. Lock the URL set

List money URLs: homepage, primary conversion path, top landing pages, and any commerce routes that bypass cache. If you onboard clients with a checklist, reuse the URL section from our site audit checklist for performance monitoring onboarding. Discovery tools can expand the list later; start with templates that actually ship revenue.

2. Set budgets before alert channels

Write mobile and desktop thresholds that match the contract or your internal standard. Separate client-facing numbers from internal early warning if you use both; only one layer should interrupt people at night. Mixing them in a single channel produces either false calm or false panic.

3. Choose delivery to match how the team works

Email digests suit account leads who live in inboxes. Slack suits engineering-heavy teams with on-call rotation. Pick one primary interrupt path per severity; duplicating the same alert to Slack and email “just in case” wastes attention.

4. Write a one-page alert policy

Document: which channel is P1, what “two consecutive breaches” means for money URLs, who triages before client comms, and cooldown expectations. If you cannot classify last month’s real alerts into P1–P3 in under a minute, the levels are decorative.

5. Run the loop once on purpose

Lower a threshold on a URL you control, wait for the scheduled run, confirm the digest or channel message lists the expected metric, then restore the threshold. You are validating routing, not performance bragging rights.

6. Fold results into the client rhythm

Point the monthly review at open breaches and fixes verified by the next green run. Monthly performance review templates for agency teams work better when the agenda is five live issues, not fifty green checks.

For portfolio-wide setup, follow how to set up automated PageSpeed monitoring for multiple sites and the Core Web Vitals monitoring checklist for agencies so proactive habits match the tooling.

Choose one client whose monitoring channel is noisiest or whose sponsor is most likely to forward Search Console screenshots. Document money URLs, mobile and desktop budget numbers, and who triages alerts before account management replies. Turn on scheduled tests if they are not already running, set budgets to match the contract, point alert delivery at inboxes or channels people actually read, and run one controlled breach to confirm the message shape.

Performance alert policy: fix alert fatigue before you retune thresholds

When teams say alerts do not work, they usually mean one of three things:

Severity inflation. Everything is critical, so critical means nothing.
Route sprawl. Alerts go to channels nobody owns, or to clients before engineering confirms cause.
Metric sprawl. Every Lighthouse sub-score pages someone, even when the retainer only mentions LCP and INP.

Fix routing and severity before you argue about whether LCP should be 2.4 or 2.5 seconds. Export a week of notifications, count how many were labelled urgent, and count how many got a human reply in-thread. High urgent count with low reply count means policy, not tuning.

We have seen agencies cut noise by half without touching budgets: one urgent channel, one team channel, everything else to weekly review. The Hashnode companion piece on alert policies people actually follow goes deeper on routing; the headline is the same: fewer routes, clearer meaning.

How Apogee Watcher fits the proactive loop

Apogee Watcher is built around the boring loop agencies need at scale: organisations, sites, scheduled PageSpeed tests, budgets per strategy, and alerts when stored lab results breach those budgets. Page discovery and multi-tenant dashboards matter, but proactive monitoring lives or dies on whether anyone hears about a breach in time to act.

Today, reliable delivery for budget violations is email digest to organisation admins: per site, per run, structured summary, cooldowns to limit repeat noise while a URL stays red. In-product signals complement mail for teams that live inside the app. Additional channels (Slack and webhooks) extend the same budget and cooldown model as they ship; the product page for Smart Alerts states what is live versus planned so copy stays honest.

Watcher does not replace your alert policy. It gives you scheduled evidence and a default notification shape that respects inboxes. You still name who triages, what counts as client-ready communication, and which URLs belong in the contract.

If you are comparing manual PSI habits to automation, read PageSpeed Insights vs automated monitoring for where each tool belongs. PSI remains excellent for deep dives on one URL; it is a poor system of record for twenty client properties.

Proactive performance monitoring is not a dashboard vanity metric. It is the habit of learning about regressions from your own schedule, with alerts quiet enough that someone will still answer when the digest says checkout LCP crossed the line you wrote down.

Create a free account to run scheduled PageSpeed tests, set performance budgets, and receive budget-violation digests across your client portfolio.

FAQ

Is proactive monitoring the same as real-user monitoring (RUM)?

No. Proactive monitoring in this sense is usually scheduled lab testing against budgets, so you catch regressions soon after deploys even when field data lags. RUM shows what visitors experience in the wild. Strong agency programmes use both; they answer different questions. Lab alerts are faster to wire consistently across many sites.

How often should scheduled tests run for a proactive setup?

Match cadence to risk and plan limits. High-traffic commerce often warrants daily runs on money URLs; lower-risk brochure sites may be weekly. The point is regularity you can cite in a retainer, not hourly noise unless you have a clear incident process.

Will proactive alerts guarantee better Core Web Vitals rankings?

Alerts do not change rankings by themselves. They change time to fix, which protects SEO and conversion work you already sold. Rankings still depend on what you ship after the alert.

Do I need Slack to be proactive?

No. Email digests, ticketing webhooks, and in-app review can be proactive if someone owns triage. Slack helps when engineering already rotates on-call there; it hurts when alerts land in channels nobody monitors.

What is the minimum team size for this to pay off?

One technical owner can run proactive monitoring across many sites if budgets and policies are written once and reused. The break-even is usually well before you would hire a full-time “performance monitoring” role, especially compared with manual PSI tours each week.

How does this relate to performance budgets in CI/CD?

CI budgets catch regressions before deploy; scheduled monitoring catches production changes, third-party scripts, and CMS edits that never touched your pipeline. Agencies often need the second layer because client sites change outside your release process. See performance budgets in CI/CD pipelines for the build-time side.

Your Server Is at 97% CPU Right Now. Would You Know?

Fri, 22 May 2026 11:00:00 +0000

Here's how it usually goes:

You deploy something. Traffic is light. Server load sits at 15% and you move on to the next thing. Then traffic grows, or a cron job stacks on itself, or a memory leak slowly eats through your RAM over 72 hours. By the time you notice, the server is thrashing, responses take 8 seconds, and your app is effectively dead.

The frustrating part is that the tools to catch this have been on your server the entire time. top and free ship with every Linux distribution ever made. Nobody installs them. They're just... there. Waiting for someone to actually ask.

So I wrote a script that asks every hour and logs a warning when the answer is bad.

The Script

#!/bin/bash

CHECK="✓"
CROSS="✗"

# --- Configuration ---
THRESHOLD=80                    # Alert when usage exceeds this %
LOG_FILE="/var/log/resource-monitor.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')

# --- CPU Usage ---
CPU=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1 | cut -d',' -f1 | xargs printf "%.0f")

# --- RAM Usage ---
RAM=$(free | awk '/Mem:/ {printf "%.0f", $3/$2*100}')

echo "[$DATE] CPU: ${CPU}% | RAM: ${RAM}%"

# --- CPU Alert ---
if [ "$CPU" -gt "$THRESHOLD" ]; then
  echo "$CROSS [$DATE] WARNING: CPU at ${CPU}% (threshold: ${THRESHOLD}%)" | tee -a "$LOG_FILE"
else
  echo "$CHECK CPU OK: ${CPU}%"
fi

# --- RAM Alert ---
if [ "$RAM" -gt "$THRESHOLD" ]; then
  echo "$CROSS [$DATE] WARNING: RAM at ${RAM}% (threshold: ${THRESHOLD}%)" | tee -a "$LOG_FILE"
else
  echo "$CHECK RAM OK: ${RAM}%"
fi

Runs in under a second. Zero dependencies. Works on Ubuntu, Debian, CentOS, RHEL, Arch — anything with top and free, which is everything.

How the CPU Check Actually Works

The CPU line looks intimidating, so let me walk through it:

top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1 | cut -d',' -f1 | xargs printf "%.0f"

top -bn1 — runs top in batch mode (-b) for exactly one iteration (-n1). Batch mode dumps the full output to stdout instead of opening the interactive TUI. This is the only way to use top in a script.

grep "Cpu(s)" — grabs the line that shows aggregate CPU stats.

awk '{print $2}' — pulls the user CPU percentage (the second field).

cut and xargs printf — strips the percent sign and any comma decimal separator, then rounds to an integer. You can't do integer comparison in bash with 2.5 — it needs a clean number like 3.

The RAM check is simpler: free shows total and used memory, and awk divides used by total and multiplies by 100.

The Thing About `tee -a`

You'll notice the script uses echo ... | tee -a "$LOG_FILE" for warnings but plain echo for healthy checks. This is intentional.

tee -a writes to the terminal AND appends to the log file simultaneously. When everything is fine, there's nothing to log — you don't want a log file full of "CPU OK" lines every hour for three years. You only want entries when something is actually wrong. So the log file becomes a clean history of every resource spike your server has had, with timestamps.

When something breaks at 2 AM and you're debugging at 9 AM, you can cat /var/log/resource-monitor.log and see exactly when resources started climbing.

Schedule It

crontab -e

Hourly checks (what I use for most servers):

0 * * * * /home/user/monitor.sh >> /var/log/monitor-cron.log 2>&1

Every 5 minutes (for production servers where you need tighter visibility):

*/5 * * * * /home/user/monitor.sh >> /var/log/monitor-cron.log 2>&1

Not sure about the cron syntax? I have a cron job builder tool that generates the line visually.

Variations That Are Worth Adding

Add an email alert when thresholds are breached:

if [ "$CPU" -gt "$THRESHOLD" ]; then
  MSG="$CROSS WARNING: CPU at ${CPU}% on $(hostname) at $DATE"
  echo "$MSG" | tee -a "$LOG_FILE"
  echo "$MSG" | mail -s "[ALERT] High CPU on $(hostname)" *Emails are not allowed*
fi

I have a full email alert script that covers the mail setup if you haven't configured it before.

Check disk space in the same script:

DISK=$(df / | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$DISK" -gt "$THRESHOLD" ]; then
  echo "$CROSS [$DATE] WARNING: Disk at ${DISK}%" | tee -a "$LOG_FILE"
fi

Now you've got CPU, RAM, and disk in one pass. I keep disk in a separate script because I use a different threshold for it (90% vs 80%), but combining them works fine if you want fewer cron entries.

Log to a CSV for trending:

echo "$DATE,$CPU,$RAM" >> /var/log/resource-history.csv

Run this for a week and you'll see patterns. Maybe your app spikes every day at 2 PM when a batch job runs. Maybe RAM creeps up 1% per day, which means you have a memory leak that'll hit the wall in three months. You can't see these patterns without historical data.

When This Isn't Enough

This script is a notification system, not a monitoring platform. It tells you "something is wrong right now" but doesn't give you graphs, dashboards, or historical trending out of the box.

If you need that level of visibility, tools like Netdata (free, runs locally) or Grafana + Prometheus are the next step. But for a single VPS or a handful of servers, a cron script that logs warnings and optionally emails you is 90% of what you need — and it takes 2 minutes to deploy instead of 2 hours.

Full script, line-by-line breakdown, cron setup, and more variations:

bashsnippets.xyz/snippets/monitor-cpu-ram-usage.html

Every script on the site is free, copy-paste ready, no signup required: bashsnippets.xyz

Changelog #8: Quota accuracy and deeper lead analyses

Fri, 15 May 2026 21:29:05 +0000

Late April was mostly about counting tests the way your charts already do, giving sysadmin leads a denser board and a proper read-only analysis page, and wiring related tags so thin blog archives still send readers somewhere useful. We also lined up marketing anchors and small UI fixes so the path from a feature page into signup or the waitlist behaves the same everywhere.

Monthly quota now follows scored PageSpeed runs

Your plan limit is PageSpeed tests per month. We only decrement the allowance when a run finishes with a Lighthouse performance score (the 0–100 headline number you see on site tests). HTTP failures, timeouts, or responses with no score no longer eat quota. That matches how we store a site test as worth charting only when there is a score to plot.

Customer dashboard numbers

On normal customer accounts (not sysadmin), the API usage strip at the top of the dashboard still shows three figures: successful runs today, this calendar month (we abbreviate thousands as “2.4k”), and errors today. Successful uses the same scored rows as quota; errors groups failed attempts so you can tell whether the lab or the URL is flaking without guessing from gaps in history. The strip still links to the day-by-day API usage table. The same ideas are spelled out for prospects on the API access and quota visibility feature page.

Pricing FAQ and product behaviour

The pricing page FAQ already described which runs count toward quota. Counters in the app now match that wording: success means a scored result you can act on, not a bare HTTP 200 with empty metrics.

When a scheduled test has no performance score

If automation gets a payload that cannot produce a score, we treat it as failed for quota and record an explicit failure instead of a row that looks healthy with no number. You spend less time hunting empty rows when PageSpeed Insights is intermittent or the URL blocks the lab.

Leads board and full-page analysis

The admin leads management and prospecting area is still in development with a small beta group. It is where agencies will run prospecting against sites that are not yet on a customer organisation: discovery jobs, scheduled lab tests, and the public report link you might send before someone becomes a paying tenant. The marketing page describes the intended workflow; the April changes already change how you work the queue inside the app.

Index table: scan before you open a lead

The index now shows enough signal on the first screen that you can sort and skim without opening every lead. Extra columns are on by default:

Pages discovered tells you how much of the site the crawler actually mapped (see automated page discovery for how that fits the wider product). A low number next to a large marketing site is often the first hint that auth walls, aggressive bot blocking, or a shallow sitemap will skew later scores.
Completed tests and pending tests compare what we have already run on mobile and desktop with what discovery says we should expect. Pending stays visible when a device lane has not caught up yet, so you are not guessing from a single “last tested” date in the row title.
Report shows the date of the latest public report plus a link that opens in a new tab, so you can sanity-check the customer-facing page before you paste it into an email.
Mobile and desktop render the latest analysis performance score as colour-coded badges: green from 90, amber from 50, red below, and “N/A” when we never got a scored run for that device.

Platform or CMS and priority remain in the column manager. Turn them on when you need taxonomy or routing metadata; leave them off when you want a wide table that still wraps long headers instead of crushing column widths.

Lead analysis: same report layout as monitored sites

Choosing View on a lead analysis opens a full-page, read-only record headed with the lead name and the URL we tested. Layout and sections match what you already know from monitored site tests: status and device, timestamps, a full-page or viewport screenshot when the stored response includes image data, and the headline 0–100 performance score.

Under that you get the same lab detail we surface for production monitoring: core timings (LCP, INP, CLS, and the rest of the Lighthouse set), optional Chrome UX Report field data when Google returns it, then diagnostics and optimisation opportunities, passed audits, and resource summaries. Nothing on the page is editable on purpose. When you are reconciling a PDF or a prospect email against what the crawler saw, you want a frozen snapshot of our store, not a second editor that could drift from the run you quoted.

If you already rely on the monitored-site test view for client conversations (the same framing as on the Core Web Vitals monitoring page), you should not have to learn a parallel “lead-only” dialect. Same sections, same ordering, different object in the database.

Blog archives and marketing pages

Thin blog tag pages can show related posts at the bottom when a tag only lists a few articles, so you still have onward reading. Across the public site, we cleaned up internal sign-up links, so you are less likely to land in the wrong place when you move from a post or feature page into the waitlist or account flow.

What to do next

If you are on a paid plan (pricing), run your next batch of tests and watch the API usage strip: successful totals should move with the scores you see on site tests, and error spikes should show up the same day. If you work leads in admin, sort by pages discovered or completed tests before opening a single row. For the public overview of that workflow, see leads management and prospecting.

The next changelog will cover the first half of May.

LLM Observability tool

Mon, 11 May 2026 06:50:41 +0000

I just created an open source LLM observability named "TraceLM" that can track the following activities:

token cost
token length
latency information etc

The following tool can be found here:

https://github.com/unnivm/tracelm-java-agent

This tools offers a very cool dashboard to track the above information. The user is required to login to access this dashboard.

This tool currently supports only Java based applications as it is basically working based on instrumentation technique. But, this tool will be extended ti support other programming languages as well.

This project is under development and am going to add more features to to so that the idea is to make it an enterprise ready product.

Watch your CrewAI agents in real-time with PiQrypt Vigil

Tue, 21 Apr 2026 11:45:45 +0000

You've added cryptographic audit trails to your crew (part 1)
and co-signed the handoffs between agents (part 2).

You now have tamper-evident history. But that's retrospective — you find out
something went wrong after the fact, when you run verify_chain().

Vigil is the real-time layer. It watches your agents while they run and raises
alerts the moment behavioural anomalies appear in the chain.

What Vigil is

Vigil is a local HTTP dashboard that launches on http://localhost:8421.
No external dependency. No data leaves your machine. It reads the same
AISS event chains your agents are writing and computes a VRS
(Verifiable Risk Score) in real time.

One command:

pip install piqrypt

Vigil is included. To start it:

from vigil import start_vigil
start_vigil()
# Dashboard running at http://localhost:8421

Or from the CLI:

piqrypt vigil start

What VRS measures

VRS is a composite risk score — a number between 0 and 1, where 1 means
the chain is healthy and 0 means it's compromised. It's computed across
five cryptographic dimensions:

Integrity — are all hashes in the chain consistent?
Verified interactions — are A2A handoffs co-signed as expected?
Diversity — is the event type distribution normal, or is the agent
doing something anomalous?
Finalization — are events being properly closed?
Rotation health — if the agent rotated its key, is the PCP continuity
intact?

Critically: VRS is deterministic. No ML, no probabilistic scoring, no model
that could drift. Given the same chain, any third party running the same
verification gets the same score.

A full CrewAI setup with Vigil

from crewai import Crew, Task
from piqrypt.bridges.crewai import AuditedAgent as Agent
from vigil import start_vigil

# Start Vigil before the crew runs
start_vigil()  # → http://localhost:8421

researcher = Agent(
    role="Researcher",
    goal="Find competitive pricing data",
    backstory="Expert at finding and analyzing market data.",
    agent_name="researcher_01"
)

writer = Agent(
    role="Writer",
    goal="Produce a pricing analysis report",
    backstory="Turns raw data into clear executive summaries.",
    agent_name="writer_01"
)

research_task = Task(
    description="Research competitor pricing for product X",
    agent=researcher
)

write_task = Task(
    description="Write a pricing analysis based on the research",
    agent=writer
)

crew = Crew(
    agents=[researcher, writer],
    tasks=[research_task, write_task]
)

result = crew.kickoff()
# While the crew runs, Vigil shows VRS and chain health per agent in real time

Open http://localhost:8421. You'll see a dashboard per agent — event count,
VRS, chain health, and a SOC-style timeline of events as they're stamped.

What a CRITICAL alert looks like

If researcher_01's VRS drops below the critical threshold, Vigil raises an
alert:
[CRITICAL] researcher_01 — chain anomaly detected
VRS: 0.21 (threshold: 0.30)
Event #34: previous_hash mismatch
Affected events: 34 → end of chain

This means: someone (or something) modified event 34 after it was written.
Every subsequent hash is now invalid. The chain is compromised from that
point forward.

Vigil alerts. It never acts. The human decides what to do — that's the
design principle. Automated blocking without human decision is not in scope.

A real example: the PixelFlow agency

The PiQrypt repo includes a runnable demo with a digital content agency —
three CrewAI agents running on Claude Haiku, all monitored live in Vigil.

# From repo root
.\demos\start_families.ps1 pixelflow
# Opens Vigil automatically at http://localhost:8421

Three agents, three distinct behavioural profiles:

pixelflow_content (Content Creator) — watch profile, TSI drift signal.
Runs 3 content sessions per day. Vigil tracks the drift in event diversity
over time.
pixelflow_seo (SEO Analyst) — safe profile, A2C concentration signal.
88% of its interactions go to pixelflow_content. Vigil flags this
concentration pattern.
pixelflow_scheduler (Social Scheduler) — alert profile, temporal sync
signal. Shares the same Claude Haiku backend as the other two — Vigil
detects the timing correlation as a CRITICAL anomaly.

All three interact with external peers: anthropic_api, google_search_console,
instagram_api, twitter_api, analytics_ga4. Every interaction is
hash-chained. Every A2A handoff between agents carries a co-signed
interaction_hash in both chains.

The demo runs in loop mode, injecting events every 5 seconds. Vigil
recalculates VRS in real time. You can watch the pixelflow_scheduler
trust score degrade as the temporal sync pattern accumulates — without
touching a single line of your crew definition.

Full demo code: github.com/piqrypt/piqrypt/demos

Checking chain integrity programmatically

You don't have to use the dashboard. You can verify a chain directly:

import piqrypt as aiss
from aiss.exceptions import InvalidChainError

events = aiss.load_events("researcher_01")

try:
    aiss.verify_chain(events)
    print(f"Chain intact — {len(events)} events verified")
except InvalidChainError as e:
    print(f"Chain compromised: {e}")

verify_chain() is binary — it returns cleanly if the chain is intact,
or raises InvalidChainError the moment a broken link is found. No score,
no partial result. The failure message tells you which link broke and why.

The VRS score with thresholds (CRITICAL below 0.30, healthy above 0.70)
is what Vigil computes continuously from the live chain and displays in
the dashboard. These are two different tools for two different moments:
verify_chain() is a point-in-time integrity check you call explicitly;
Vigil is the continuous watch that alerts you before you think to check.

Standard vs Pro

The Vigil included in pip install piqrypt (standard tier) gives you:

Full dashboard UI
VRS with 7-day history
CRITICAL alerts
Up to 2 bridge types monitored
Chain health per agent

Vigil Pro adds 90-day VRS history, all alert levels (WARNING, INFO, DEBUG),
unlimited bridge types, and TrustGate integration — the governance layer
that intercepts agent actions before execution and routes to a human when
policy requires it.

For most development and staging use cases, the standard tier is enough.

The full picture

Three articles, one integration pattern:

Part 1 — drop-in bridge, 3 lines, every agent action is now
Ed25519-signed and hash-chained.

Part 2 — A2A co-signing, the handoffs between agents carry
cryptographic proof that neither side can deny.

Part 3 — Vigil, real-time monitoring of chain health and behavioural
anomalies while the crew runs.

Together: your CrewAI crew has tamper-evident history, non-repudiable
inter-agent accountability, and live anomaly detection. None of it requires
a server, an account, or sending data anywhere.

Part 1: How I added cryptographic audit trails to any CrewAI crew in 3 lines
Part 2: Multi-agent accountability: who co-signs the handoff between your CrewAI agents?

GitHub: github.com/piqrypt/piqrypt
pip install piqrypt
Protocol spec (MIT): github.com/piqrypt/aiss-standard

Your API is down - now what? Capturing failure context in Node.js

Sun, 19 Apr 2026 05:55:20 +0000

Most monitoring tools tell you one thing

"Your API is down."

That's useful. But only partially.

I wanted to go beyond "it's down" and understand why — specifically, where in the request lifecycle things actually broke.

The frustrating part of debugging failures

The first time this really hit me, I spent almost an hour digging through logs after a 3am alert — only to realize the issue had already disappeared. No trace of what went wrong. Just a gap in the metrics and a resolved status.

The typical workflow looks like this:

You get an alert
You SSH into your server
You check logs
You try to reproduce the issue

And in many cases… the issue is already gone.

The failure might have lasted only a few seconds:

A DNS resolution issue
A TLS handshake problem
A temporary upstream timeout

By the time you investigate, there's no trace left.

Logs don't always tell the full story

Logs are helpful, but they have real limitations:

They only capture what your application explicitly logs
They often miss network-level issues entirely
They don't show how a request failed step by step

What I kept missing was the network-level picture — not what the app logged, but what actually happened between the client and the server at each phase of the request.

Breaking down an HTTP request

When you make a request to an API, a lot happens under the hood:

DNS lookup — translating the domain name into an IP address
TCP connection — establishing a connection to the server
TLS handshake — negotiating encryption (for HTTPS)
Server processing — the server does its work and prepares a response
First byte received — also called TTFB (Time To First Byte), this is the total time from sending the request to getting the first byte back. It's not a separate step — it's a cumulative measurement across all the steps above.

If any one of these steps fails or slows down, the entire request is affected.

But most monitoring tools treat the request as a black box. You get a status code. Sometimes a timeout. Rarely anything more.

The idea: capture the failure at the moment it happens

Instead of trying to debug after the failure has resolved itself...

What if we could capture useful diagnostic signals during the failure?

So I started experimenting with capturing:

DNS lookup timing
TLS handshake duration
Time to first byte
The first part of the response body

At the exact moment a request fails.

I ended up building a small tool around this idea.

What this actually changes

To see why this matters, here's what a typical monitoring alert looks like:

ALERT: API DOWN
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
endpoint:  api.myapp.io/payments
status:    DOWN
started:   2026-03-26 09:05:25
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

That's everything you get. Now you start the investigation — SSH, logs, curl, check DNS, check TLS. By the time you look, the evidence is usually gone.

Now here's what it looks like when you capture diagnostic data at each failure. The tool waits for three consecutive failures before alerting (more on why below), and captures a snapshot at each one silently:

Example A — Server degradation into timeout

--- SNAPSHOT #1 ---                         ap-southeast-1
───────────────────────────────────────────────────────────
  dns_lookup       1ms                       (~estimated)
  tls_handshake    5ms                   (SSL negotiation)
  ttfb             4,007ms                     ↑ critical
                   server is very slow or overloaded
  total_time       4,312ms
───────────────────────────────────────────────────────────

--- SNAPSHOT #2 ---                         ap-southeast-1
───────────────────────────────────────────────────────────
  dns_lookup       2ms                       (~estimated)
  tls_handshake    4ms                   (SSL negotiation)
  ttfb             10,002ms                    ↑ critical
                   server is very slow or overloaded
  total_time       10,374ms                      ↑ worse
───────────────────────────────────────────────────────────

--- SNAPSHOT #3 ---                         ap-southeast-1
───────────────────────────────────────────────────────────
  error            TIMEOUT
                   Your server did not respond within
                   the allowed time. May be overloaded,
                   crashed, or unreachable.
  total_time       10,000ms
───────────────────────────────────────────────────────────

TTFB went from 4,007ms to 10,002ms to a full timeout. The server was degrading under load until it stopped responding entirely. You can see the failure developing across three snapshots.

Example B — Upstream dependency failure

--- SNAPSHOT #1 ---                         ap-southeast-1
───────────────────────────────────────────────────────────
  dns_lookup       3ms                       (~estimated)
  tls_handshake    12ms                  (SSL negotiation)
  ttfb             1,847ms                        ↑ high
  status_code      502
                   Bad Gateway — upstream service
                   returned an invalid response
  response_body    {"error":"upstream_timeout",
                    "service":"payment-gateway"}
───────────────────────────────────────────────────────────

--- SNAPSHOT #2 ---                         ap-southeast-1
───────────────────────────────────────────────────────────
  ttfb             1,923ms                        ↑ high
  status_code      502
  response_body    {"error":"upstream_timeout",
                    "service":"payment-gateway"}
───────────────────────────────────────────────────────────

--- SNAPSHOT #3 ---                         ap-southeast-1
───────────────────────────────────────────────────────────
  ttfb             2,104ms                        ↑ high
  status_code      502
  response_body    {"error":"upstream_timeout",
                    "service":"payment-gateway"}

  ┌─────────────────────────────────────────────────────┐
  │  DNS ✓  TLS ✓  Server responded — upstream failed.  │
  │  Payment gateway timed out. Your server is fine.     │
  └─────────────────────────────────────────────────────┘
───────────────────────────────────────────────────────────

DNS fine. TLS fine. Your server responded — but the payment gateway behind it timed out. No SSH session required.

The difference: instead of starting an investigation after the alert, the evidence is already there when the alert arrives.

The tricky part: getting this data in Node.js

This turned out to be harder than expected.

Most high-level APIs like fetch don't expose DNS timing or TLS handshake duration at all. They give you a response, but no visibility into the individual phases of how that response was obtained.

To get this data, I ended up using undici (the HTTP client that powers Node.js fetch under the hood) combined with Node.js diagnostics_channel. This lets you subscribe to events that fire at specific moments during a request:

import diagnostics_channel from 'node:diagnostics_channel'

const requestStartMs = Date.now()
let connectStartMs = null
let connectEndMs = null
let headersMs = null

diagnostics_channel.subscribe('undici:client:beforeConnect', () => {
  connectStartMs = Date.now()
})

diagnostics_channel.subscribe('undici:client:connected', () => {
  connectEndMs = Date.now()  // TCP + TLS handshake complete
})

diagnostics_channel.subscribe('undici:request:headers', () => {
  headersMs = Date.now()  // First byte received
})

// After the request completes:
// connectEndMs - connectStartMs = real TCP + TLS handshake time
// headersMs - requestStartMs     = real TTFB (total time to first byte)
// Remember to unsubscribe after the request completes

Here's what each metric actually gives you — and whether it's a real measurement or an estimate:

Metric	Source	Real or estimated?
connect_ms	`undici:client:connected` - `beforeConnect`	Real (TCP + TLS)
ttfb_ms	`undici:request:headers` - request start	Real
total_ms	`Date.now()` delta	Real
tls_ms	Rough split of connect_ms (HTTPS only)	Estimated
tcp_ms	Rough split of connect_ms	Estimated
dns_ms	Derived heuristic	Estimated

A few things worth noting honestly:

DNS timing is estimated, not exact. Node.js doesn't expose real DNS resolution timing through diagnostics_channel. DNS resolution happens before the beforeConnect event fires, so there's no direct event to hook into. The values I capture are heuristic-based — useful for spotting anomalies, but not precise measurements. That's why the examples above show (~estimated) next to DNS values.
The diagnostics_channel events are semi-internal. They work well today, but undici doesn't guarantee these channel names will stay stable across Node.js versions. It's a tradeoff — you get real timing data, but you're relying on an API that could change.
The capture is a separate request. When a check fails, the tool makes a second HTTP request using undici to capture the timing breakdown. This means the snapshot might not match the original failure exactly — the server could have recovered, or the error could be different. It's a real tradeoff I'm still thinking about. The alternative would be to instrument the original request, but that adds complexity to the main check path.

I wanted real timing data, even if it meant accepting these tradeoffs.

One more design decision: don't alert on the first failure

As you saw in the examples above, the tool waits for three consecutive failures before sending an alert — and captures a snapshot at each one silently:

Check 1 fails → snapshot #1 captured silently
Check 2 fails → snapshot #2 captured silently
Check 3 fails → snapshot #3 captured + alert sent with all three

Why three? It's enough to filter out transient blips (a single DNS hiccup, a momentary network glitch), but short enough to catch real outages within minutes at typical check intervals.

The result:

Fewer false alarms — transient issues don't wake you up at 3am
A progression timeline — you can see the failure developing across three snapshots (like the degradation in Example A above)
Evidence arrives with the alert — all three snapshots are attached before you even open it
Recovery is handled too — two consecutive successes resolve the incident and send a recovery notification with the total downtime duration

A small shift in thinking

What changed for me was this:

Don't just detect failures — capture context when they happen.

That one shift made debugging much faster and less frustrating. By the time the alert arrives, the evidence is already there.

Still exploring

There are still open questions here — especially around DNS timing accuracy and how closely the snapshot matches the original failure when it's captured as a separate request.

I'm curious how others handle this:

How do you usually debug short-lived or transient API failures?

Do you rely on logs, APM tools, distributed tracing, or something else entirely? Have you found a good way to capture context at the moment of failure rather than investigating after the fact?

Would love to hear what works for you.

Your Website Looks Fine. This Free Health Scanner Disagrees.

Fri, 17 Apr 2026 07:06:19 +0000

Most websites have problems their owners don't know about. Not the dramatic kind that crashes a server. The quiet kind — a broken script, an expiring SSL certificate, missing security headers, images that silently fail to load.

These issues don't trigger uptime alerts. They just make the experience worse for visitors while every dashboard says "all green."

We built a free scanner that finds them in 20 seconds: getsitewatch.com/scan

No signup. No email. Just paste a URL and get a full health report.

What It Checks

Broken Assets — Every image, script, and stylesheet your page references. Do they actually load, or do they return 404? A missing JavaScript file can break your entire page while the server happily returns 200 OK.

SSL Certificates — Is your cert valid? When does it expire? Is the chain complete? An incomplete chain means your site works in Chrome but throws warnings in Safari and on mobile. 88% of companies have had an outage from an expired cert in the past two years.

Mixed Content — Does your HTTPS page load anything over HTTP? Browsers block this silently. No error. Images just don't render. Common on WordPress sites after an SSL migration.

Security Headers — CSP, HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy. Most sites are missing at least two. These are basic protections against XSS and clickjacking that take 15 minutes to set up.

Schema & SEO — JSON-LD validation, Open Graph tags, canonical URLs. If these are broken, your links look bare when shared and Google can't generate rich snippets for your pages.

Discoverability — Does your robots.txt exist? Is there a sitemap? Surprisingly often the answer is no, especially on SPA and Jamstack sites.

What We Typically Find

We've run this on hundreds of sites. The pattern is consistent:

Almost every site is missing security headers
Roughly 3 out of 10 have at least one broken asset
SSL issues are more common than anyone expects
SEO metadata is frequently broken or missing after CMS updates

Most of these sites look perfectly fine when you open them in a browser. The problems are invisible until you actually check.

How It Works

The scanner fetches your page, enumerates every subresource, validates each one returns a successful response, inspects headers for security policies, checks SSL certificate validity and chain completeness, and validates structured data and SEO metadata.

Results come back as a prioritized report: critical issues first, info-level findings last. Every finding includes a plain-language explanation of what it means and how to fix it.

Try It

getsitewatch.com/scan

20 seconds. Completely free.

If you want these checks to run continuously and alert you when something breaks, that's what Sitewatch does — website monitoring that catches what uptime tools miss. But the one-time scan is yours with no strings attached.

Curious what your site's health rating looks like.