Skip to content

feat(on-call): page again when an acknowledged incident or alert is not resolved in time (#5123) - #5173

Merged
simlarsen merged 15 commits into
masterfrom
claude/ack-timeout-repage
Oct 22, 2026
Merged

simlarsen merged 15 commits into
masterfrom
claude/ack-timeout-repage

Conversation

@simlarsen

@simlarsen simlarsen commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

For issue #5123 (a suggested reply to it is at the end).

The problem

Acknowledging a page stops every on-call policy for that incident, alert or episode, for good. When the person who acknowledged it is pulled away, falls back asleep or forgets, the record can sit in Acknowledged with nobody reminded of it. The policy's Repeat only covers the case where nobody acknowledges. The customer's workaround (a workflow that sleeps, re-reads the incident and pages) has to be built per project, and cannot show in the feed or the execution logs that the timeout fired.

The recommendation

Build it, as an advanced, opt-in policy setting with nothing else to decide: it is a real gap that teams moving from PagerDuty expect to be closed (its service-level Acknowledgement Timeout works this way), and the workaround is fragile. What we do not do is move the record back to un-acknowledged: OneUptime never moves a record back up its states (Common/Utils/StateMove), and doing so here would write a state change into its history, email status page subscribers, post to its channels and start every workflow and SLA that waits on a state, all for a reminder. A page again is a run of the policy of its own instead, visible in the feed and the execution logs, and acknowledged, escalated or left alone like any page.

What changed

One policy setting, folded away. OnCallDutyPolicy.acknowledgementTimeoutInMinutes (nullable, no default: off on every policy there is). In the Dashboard it is the Acknowledgement Timeout card under More settings at the bottom of the policy's Escalation Rules page, folded by default (More settings names the card, and says "30 min" while it is on). Its dialog asks one question, If it is acknowledged but not resolved: Don't page again (what every policy does today) or Page again, and only then After how many minutes? (5 to 1440, 30 to start with). Everything else is decided once for everybody, so there is nothing else to pick:

  • The clock starts at each acknowledgement, from anywhere (an answered page, the Dashboard, the mobile app, Slack, Teams, the API, a workflow), and starts again at every new one. It is counted from when OneUptime records the acknowledgement, so a back-dated state change does not page anybody at once.
  • When it runs out with the record still acknowledged and not resolved, the policy pages again: a new run (pageAgainNumber 1..5) that starts at the level that was paging when it was acknowledged, pages whoever acknowledged it and whoever is on call there now (a rotation that changed hands pages its new person), and goes on through the levels like the first run.
  • At most 5 pages again per policy and record, so a record somebody keeps acknowledging is not paged about forever.
  • It stops when the record is resolved or deleted, the policy is archived or deleted, or the timeout is turned off; a changed number of minutes applies to the clocks already running.

Server. Common/Types/OnCallDutyPolicy/OnCallAcknowledgementTimeout.ts holds the rule and its words; Common/Server/Utils/OnCall/OnCallAcknowledgementTimeout.ts holds the clock (started from the four state timeline services and from answering a page), the claim (a compare-and-set, so each timeout pages again exactly once across workers), who is paged and the stops. A new minute worker, OnCallDutyPolicyExecutionLog:PageAgainAfterAcknowledgementTimeout, reads only running clocks through a partial index. The run and person-page workers learned that a page again is not over because its record is acknowledged (it was all along), only once the page again is acknowledged or the record resolved. The API and Terraform refuse anything but null or a whole number from 5 to 1440.

What people see. The feed says "On-call policy Payments is paging again (1 of 5): the incident is still not resolved 30 minutes after Dana Kim acknowledged it." The run's status message and its Execution Logs row say "Paged again (1 of 5)". The run's Slack/Teams message says the policy is paging again, and, when nobody answers, raises the same alarm as a first page. The person paged again gets an Acknowledge again button on the record's page beside Escalate (the record's own Acknowledge is gone since it is acknowledged); it answers their page and starts the clock again. Their own call, email, push, Slack, Teams and Telegram message say the record is still not resolved so long after it was acknowledged, and the policy's Escalation summary and Policy at a glance say it pages again (see UI review below). The Escalate wording from #5167 is kept.

Migrations. 1805960000000-AddOnCallAcknowledgementTimeout adds three nullable columns (no rewrite, no backfill); 1805970000000-AddOnCallAcknowledgementTimeoutIndex builds the partial index online. Both numbered after master's newest (renumbered at the last merge, after the code production review's 1805950000000); the drift check is clean.

Docs and words. On-Call Policies gains Page again if it is not resolved and the API field, in all 17 docs languages, line for line, naming each language's Dashboard labels. The 18 new Dashboard strings are translated in all 16 locales.

What changes for existing customers

Nothing until somebody turns it on: every policy starts at Don't page again, and the setting is folded under More settings. The new worker reads an index that is empty until then.

Left out, on purpose

  • Moving the record back to un-acknowledged (see above).
  • A timeout per severity or per record: one per policy, like Repeat. A severity can use its own policy through on-call rules.
  • More choices (who is paged, how many times): fixed, to keep it one decision.
  • An Acknowledge for a page again on the mobile app's record page: the person paged again answers it from its email, text or call like any page; a button in the app is a follow-up.

Tests

  • Common/Tests/Types/OnCallDutyPolicy/OnCallAcknowledgementTimeout.test.ts: the rule, the bounds, the clock to the millisecond, the cap, the words.
  • Common/Tests/Server/Utils/OnCall/OnCallAcknowledgementTimeout.test.ts and OnCallAcknowledgementTimeoutRecords.test.ts (an in-memory world, AcknowledgementTimeoutWorld.ts): fires once per timeout, restarts on a new acknowledgement, stops on resolve, delete, archive and turning it off midway, a changed timeout, minute-edge clock reads, who is paged after the rotation changed hands, the cap, two policies on one record, every record kind, the feed entry, and never throwing.
  • OnCallDutyPolicyExecutionLogPageAgain.test.ts, AcknowledgementTimeoutHooksGuard.test.ts (an AST guard that the four state timelines and answering a page start the clock), OnCallPolicyRunMessagePageAgain.test.ts, OnCallPageAgainAcknowledgeAPI.test.ts, OnCallAcknowledgementTimeoutApiAndTerraform.test.ts.
  • App workers: PageAgainAfterAcknowledgementTimeout.test.ts and the two ExecutePendingExecutionsPageAgain.test.ts.
  • Dashboard (jsdom): the card, the choice, the Escalation page's fold, the Execution Logs row, the record header's Acknowledge.
  • Docs: OnCallAdministrationDocsFacts holds the new section to the code; the translation suites hold the 16 translations to it.
  • Browser: packages/E2E/AcknowledgementTimeout (offline, real page): folded by default, the dialog, what each save sends, validation, a refused save, 1280 and 390 pixels, dark and German. Wired into Dashboard Offline UI group 9.
  • Terraform E2E test 31 now sets the timeout and changes it on update.

What ran locally

  • Common npx jest Guard: 162 suites, 5,061 tests passed.
  • Common, every suite that imports or reads a changed file, plus the named guards (RequiredColumnsWithDefaultsGuard, RelationListReadCoverage, the credential settings suites, PlanGatedTableLeftovers, TestSendRoutesAskTheRule, PageSearchIndexCovers*, PatternWorkAllowanceGuard, UpdateChecksReadHeldRows, WrongComponentKeyGuard, MoreSettingsOneCard, CardInCardGuard): 249 files, 242 passed and 7 skipped (Postgres-only), 11,053 tests. The 202 server suites among them again with BILLING_ENABLED=true (12 of them need the plans CI sets with billing on, and pass with them).
  • App, every suite that imports or reads a changed file, the root-level tests, ShortDescriptionsGuard, CardDescription*, OnCallSimplicityWiring, MoreSettingsOneCardDocs and DashboardLocalesGuard: 88 suites, 6,243 tests; the 41 worker and notification suites again with billing on.
  • Docs: Tests/FeatureSet/Docs/Docs*, OnCallAdministrationDocsFacts and Translations, CriteriaGroupsPostmortemsDocsFacts (32 suites, 2,919 tests), docs:check-anchors, i18n:validate, i18n:check, and the Locale suites in Common (37) and App (10).
  • The offline browser suite: 9 tests in 18 seconds with its bundle, also with ee/ removed as CI runs it.
  • Every migration from an empty Postgres 15, then the schema drift check: clean.
  • npm run compile in Common, App and E2E, compile-tests in Common, and tsc --listFiles in Common, App, the Dashboard and E2E (nothing from the root Scripts/ or another package); eslint and prettier on every changed file; the ReDoS checker (no new pattern that is not linear).

Not run locally: the Terraform E2E (needs the full stack), the real-stack Playwright suites, and the whole App and Common suites (only the affected ones, as listed).

UI review

A second agent reviewed every surface this change adds, looked at each one for real, and polished it before merging.

What was looked at. The policy's Escalation Rules page (More settings folded and open, the card, the dialog with each answer, every refusal, a refused save) at 1280 and 390 pixels, light and dark, English and German, through the branch's offline fixture and the real On-Call route group in a harness. The record header for the person paged again (incident and alert) at 1280 and 390, light and dark, English and German, including a failed acknowledge, through the EventOverview fixture with an open page again. The Execution Logs (one policy's and all policies') at 1280 and 390, dark and German. The Escalation summary and Policy at a glance. The run's Slack Block Kit and Teams Adaptive Card messages (paging again, answered, nobody answered, the alarm) and the paged person's Slack/Teams DM and Telegram message, rendered at desktop and phone widths. The call script, the text message, the push and the email (the real template, rendered) that the person paged again receives. The feed entry, the run's status message and the docs section.

What changed, and why.

  • The person paged again was told it was a new incident. A page again goes out through the person's own notification rules, worded as a first page: the call said "A new incident has been created", the email's subject "Incident INC-42 created", the push "New Incident", Slack, Teams and Telegram "New incident assigned to you" - to somebody who acknowledged it half an hour ago. Each now says what it is: the call opens "An incident is still not resolved 30 minutes after it was acknowledged", the email's subject is "ACTION REQUIRED: Incident INC-42 still not resolved - ..." and its first line says why, the push is "Incident INC-42 still not resolved: ...", and Slack, Teams and Telegram head it "Incident still not resolved" with the reason under it. Same for alerts and both kinds of episode. The delivery reads once whether its run is a page again (and the policy's minutes); a run that cannot be read is worded as a first page and the page still goes out. Left as they are: the text message (one 160-character part, and it never calls the record new) and WhatsApp (its words are a template Meta approved).
  • "Acknowledge" beside "Acknowledged". The header offered the person paged again a bare Acknowledge right next to the record's Acknowledged pill, which read as a button that does nothing. It is Acknowledge again. A failure no longer repeats the button's word as its title: "Could not acknowledge" - "You are still being paged. Please try again."
  • The minutes box spoke in the form's words. "After how many minutes? should not be less than 5." and "After how many minutes? is required.", and a fraction (7.5) went through to the server, which refused it. One sentence now covers everything the box may not hold, before anything is sent: "Enter a whole number of minutes between 5 and 1440."
  • A policy that pages again now says so where it is read. The setting stays folded under More settings, as asked. But once it is on, it pages people, and the two places a policy is read in a few seconds said nothing about it. The Escalation summary at the top of the same page (updated the moment the card is saved) and Policy at a glance on the policy's Overview now end with "If it is acknowledged but not resolved within 30 min, it pages again, up to 5 times." Durations read like the rest of the summary ("1 hr 30 min"), and the numbers stay whole on a phone. A policy that does not page again reads exactly as before.
  • Docs (17 languages): Acknowledge again, and the summaries saying it, each language naming its own Dashboard labels.
  • Strings: 7 new keys translated in all 16 locales (a dedicated "up to {{count}} times" plural, since the shared "{{count}} times" reads as a count of occurrences in Russian); the orphaned "Could not acknowledge. Please try again." is gone.

Tests added or changed in the review.

  • Common/Tests/Server/Services/OnCallPageAgainChannelMessages.test.ts (49 tests): the words; for each of incident, alert and both episodes, the call, email, Telegram, Slack/Teams message and push of a page again and of a first page; reading whether a delivery is a page again (with and without the policy's minutes, a first run, a direct page that is never read, a read that fails quietly); and a whole delivery through executeNotificationRuleItem handing the notice to each channel, once.
  • App/Tests/Notification/OnCallPageAgainEmailTemplates.test.ts: the four page emails open with the page-again line instead of "A new ... has been created", a first page as before, and the line is escaped text.
  • Common/Tests/App/Dashboard/AcknowledgementTimeoutPolicySummaries.test.tsx (14): the sentence, its bold and unbreakable numbers, the Escalation summary's row (durations, placement after the last step, absent when off) and Policy at a glance (what it reads, the sentence, absent when off, an out-of-range stored value read as the worker reads it).
  • AcknowledgementTimeoutCard.test.tsx: one refusal for 4, 1441, 7.5, 0 and empty; 5, 1440 and 45 taken as typed. AcknowledgementTimeoutEscalationPage.test.tsx: the page hands the card's minutes to the summary, and a save reaches it at once. OnCallPageAgainHeader.test.tsx: Acknowledge again, never a bare Acknowledge, and the failure dialog's title and sentence.
  • OnCallAdministrationDocsFacts: Acknowledge again and the summaries sentence pinned to the code.
  • Offline browser suite (packages/E2E/AcknowledgementTimeout): the refusal for 4, 1441, 7.5 and empty, and the Escalation summary's line - absent while off, there at once after a save, gone again when turned off. Still 9 tests (assertions added to existing ones), about 25 seconds with its build.

Left alone, on purpose. The all-policies Execution Logs table scrolls sideways at 1280 pixels (its Status pill was clipped the same way before this change). The token pages a text or email link opens ("Do you want to acknowledge this Incident?") read correctly for a page again. The mobile app's Acknowledge for a page again stays the follow-up the builder named.

What ran for the review (after its last edit and its merge of master):

  • Common: the 233 suites that name a changed file (222 passed, 11 Postgres-only skipped), npx jest Guard (162 suites), UpdateChecksReadHeldRows, DeleteChecksReadHeldRows, HookWritesCallersMayMake, ListOrderColumnModels, PermissionCatalogueCoverage, and the 37 Locale suites; the notification suites again with BILLING_ENABLED=true and CI's plans.
  • App: the 76 suites that name a changed file (Notification, Workers, Docs facts and translations, Dashboard wiring), the root-level tests, DashboardLocalesGuard, NormalizeListOrder, the 10 Locale suites, Tests/FeatureSet/Docs/Docs* (29 suites) and the policies page's own docs suites; docs:check-anchors, i18n:validate, i18n:check (no translation lost).
  • ee: the TeamCompliance suites that import the notification service, with ee/ present.
  • The added and changed suites again with ee/ hidden, as Common Test and App Test run them.
  • The offline browser suite: 9 of 9, about 25 seconds with its build, also with ee/ hidden.
  • Type-checks: tsc and compile-tests in Common, npm run compile in App and E2E, the Dashboard's tsc; eslint and prettier on every changed file; the ReDoS checker (no new pattern).

Suggested reply to the issue

Thanks for the detailed write-up. This is now in review as an Acknowledgement Timeout on the on-call policy: under More settings on the policy's Escalation Rules page, choose Page again and after how many minutes (5 to 1440). The clock restarts on every acknowledgement; when it runs out with the incident or alert still unresolved, the policy pages again (the person who acknowledged it, and whoever is on call at that level now), and the feed and execution logs say "Paged again (1 of 5)". It pages again at most 5 times and stops when the incident is resolved. It is off by default. One difference from your proposal: the incident stays acknowledged, because moving it back would notify status page subscribers and channels of a state change; the page again is its own page that can be acknowledged or escalated.

🤖 Generated with Claude Code

valeriamustafaeva and others added 10 commits October 22, 2026 04:47
…ared rule (#5123)

An on-call policy can page again when an incident, alert or episode it
paged about stays acknowledged without being resolved. This adds the
policy's acknowledgementTimeoutInMinutes, the run's clock
(acknowledgementTimeoutStartedAt, with a partial index built online) and
pageAgainNumber, and Types/OnCallDutyPolicy/OnCallAcknowledgementTimeout:
the rule and its words.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… in time (#5123)

The clock starts when a record moves into an acknowledged state (every
state timeline asks OnCallAcknowledgementTimeout) and again when a page
of the policy's latest run is acknowledged; a resolve stops it. A new
worker job pages again once the policy's timeout runs out: a run of its
own (pageAgainNumber), starting at the rule that was paging, paging the
person who acknowledged first, at most five times per record and policy.

A page again is over when it is acknowledged or its record resolved: the
run worker, the person's page worker and Escalate read it that way. The
feed and the run's Slack and Microsoft Teams message say it pages again.
The Dashboard gets the setting under More settings on the policy's
Escalation Rules page, and the logs table marks a page again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…PI, workers and Dashboard (#5123)

AcknowledgementTimeoutWorld drives the real clock, page again, escalation
and acknowledge hook in memory: pages again once per timeout (two ticks at
once page once), restarts on every new acknowledgement, stops on resolve,
delete, archive or turning it off, follows a changed timeout, pages the
acknowledger first and whoever is on call now at the rule that was paging,
caps at five, and can be escalated. Plus the per-kind reads, the state
timeline hooks guard, the API and Terraform shape, startPageAgain, the
run's chat message, both workers, the Dashboard card, its More settings
fold and the logs table note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A page again's record was acknowledged all along, so the record's own
Acknowledge is not in its header: the person paged again opening it from
their push could escalate but not say they are still on it. The record's
header now shows them an Acknowledge of their own, before Escalate, which
answers their page through POST /on-call-page/acknowledge/<type>/<id>.

UserOnCallLogTimelineService.acknowledgePage is the one place a page is
acknowledged (the token links' hook calls it too), and a person's open
pages say which is a page again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… language (#5123)

The card, its choices, the folded More settings value, the Execution Logs
line and the record page's Acknowledge failure, translated in the 16
Dashboard locales. The card's summary is one PluralTemplate, so en.json
holds it once with its _one form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… resolved (#5123)

On-Call Policies gains a section after Repeat a policy: where the
Acknowledgement Timeout card waits (More settings on the Escalation Rules
page), its two answers, what a page again does and what stops it, and the
acknowledgementTimeoutInMinutes API field. In all 17 docs languages, line
for line, naming each language's Dashboard labels.
OnCallAdministrationDocsFacts holds the new words to the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… More settings (#5123)

An offline Playwright suite of a policy's real Escalation Rules page:
More settings folded at first and naming the card, the card's one row,
the dialog's two answers and the minutes box (30 to start with, 5 to
1440), what each save sends (the minutes, or null to turn it off), a
refused save, and all of it at 1280 and 390 pixels, dark and German.
Wired into Dashboard Offline UI group 9 with its upload paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…files (#5123)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…repage

The acknowledgement timeout's migrations move to 1805920000000 and
1805930000000, after master's AddIncidentPostmortemStatus pair took
1805900000000 and 1805910000000. Locale files merged with
npm run i18n:extract; Postmortems and Acknowledgement Timeout both join
Dashboard Offline UI group 9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d whole (#5123)

AddOnCallAcknowledgementTimeoutMigration holds the two migrations to the
models: three nullable columns with no default and nothing written, the
partial clock index online under TypeORM's name, registration and order.
Terraform E2E test 31 sets acknowledgement_timeout_in_minutes and changes
it on update. The feed test compares the whole entry, links included, and
the header test reads the posted route as an address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@simlarsen

simlarsen commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor Author

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

valeriamustafaeva and others added 5 commits October 22, 2026 07:37
…ssage (#5123)

UI review. A page again reaches people through their own notification rules,
worded as a first page: the person who acknowledged the incident half an hour
ago was called with "A new incident has been created", emailed "Incident
INC-42 created", pushed "New Incident" and messaged "New incident assigned to
you".

Each channel now says what it is: the call opens "An incident is still not
resolved 30 minutes after it was acknowledged", the email's subject and first
line say it, the push title is "Incident INC-42 still not resolved", and Slack,
Teams and Telegram head it "Incident still not resolved" and say why. The
delivery reads once whether its run is a page again (and the policy's minutes);
a run that cannot be read is worded as a first page and still goes out. The
text message (one 160-character part, never calls it new) and WhatsApp (an
approved template) are unchanged, and a first page reads exactly as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…summaries say it (#5123)

UI review of the acknowledgement timeout's Dashboard surfaces:

- The record's header showed "Acknowledge" right beside its "Acknowledged"
  pill to the person paged again, which read as a button that did nothing.
  It is "Acknowledge again"; a failure says "Could not acknowledge" and that
  they are still being paged, rather than repeating the button's word.
- The minutes box said "After how many minutes? should not be less than 5."
  and "After how many minutes? is required.", and let a fraction through to
  the server. One sentence now covers every value it may not hold: "Enter a
  whole number of minutes between 5 and 1440."
- The setting stays folded under More settings, but once a policy pages
  again, the two places it is read in a few seconds say so: the Escalation
  summary at the top of the same page (updated the moment the card is
  saved) and Policy at a glance on its Overview - "If it is acknowledged but
  not resolved within 30 min, it pages again, up to 5 times." A policy that
  does not page again reads as before.

Seven new strings in all 16 locales (the orphaned "Could not acknowledge.
Please try again." is gone), the docs in 17 languages, the jsdom suites, the
docs facts and the offline browser suite follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…repage-ui-review

The acknowledgement timeout's migrations are renumbered 1805960000000 and
1805970000000, after master's newest (1805950000000, the code production
review). Locale files merged with i18n:extract.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@simlarsen
simlarsen merged commit 7cf1f5d into master Oct 22, 2026
1 of 66 checks passed
@simlarsen
simlarsen deleted the claude/ack-timeout-repage branch October 22, 2026 08:23
simlarsen pushed a commit that referenced this pull request Oct 22, 2026
…ry; a page-again count is not a record number

Common Test went red on master 7cf1f5d in two suites:

- AIKillSwitchBackstop: #5172's ProductionReviewAI calls
  AIService.executeWithLogging and was not in the registry of call sites.
  It belongs in the skip lane: ProductionReviewRunner reads the project's
  AI switch first (isProjectAIEnabled) and, when AI is off, the review
  notes it and goes out with the production facts alone; a refusal while
  it reads becomes the review's AI note, never an error. It is registered
  with that reason.
- RunRecord: #5173 gave OnCallDutyPolicyExecutionLog a numeric
  pageAgainNumber, and the suite takes any numeric "...Number" column of a
  model a workflow can watch for a record number that needs its
  "...WithPrefix". It counts which page again a run is (1 to 5) within one
  record's paging; the product's getRecordNumberColumns already reads no
  record number without a "...WithPrefix" beside it. The suite records
  that decision in NOT_RECORD_NUMBERS, as its own comment asks.

Both suites pass (110 tests); Common's compile-tests type-check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant