Repository navigation
feat(on-call): page again when an acknowledged incident or alert is not resolved in time (#5123) - #5173
Merged
Conversation
…ared rule (#5123) An on-call policy can page again when an incident, alert or episode it paged about stays acknowledged without being resolved. This adds the policy's acknowledgementTimeoutInMinutes, the run's clock (acknowledgementTimeoutStartedAt, with a partial index built online) and pageAgainNumber, and Types/OnCallDutyPolicy/OnCallAcknowledgementTimeout: the rule and its words. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… in time (#5123) The clock starts when a record moves into an acknowledged state (every state timeline asks OnCallAcknowledgementTimeout) and again when a page of the policy's latest run is acknowledged; a resolve stops it. A new worker job pages again once the policy's timeout runs out: a run of its own (pageAgainNumber), starting at the rule that was paging, paging the person who acknowledged first, at most five times per record and policy. A page again is over when it is acknowledged or its record resolved: the run worker, the person's page worker and Escalate read it that way. The feed and the run's Slack and Microsoft Teams message say it pages again. The Dashboard gets the setting under More settings on the policy's Escalation Rules page, and the logs table marks a page again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…PI, workers and Dashboard (#5123) AcknowledgementTimeoutWorld drives the real clock, page again, escalation and acknowledge hook in memory: pages again once per timeout (two ticks at once page once), restarts on every new acknowledgement, stops on resolve, delete, archive or turning it off, follows a changed timeout, pages the acknowledger first and whoever is on call now at the rule that was paging, caps at five, and can be escalated. Plus the per-kind reads, the state timeline hooks guard, the API and Terraform shape, startPageAgain, the run's chat message, both workers, the Dashboard card, its More settings fold and the logs table note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A page again's record was acknowledged all along, so the record's own Acknowledge is not in its header: the person paged again opening it from their push could escalate but not say they are still on it. The record's header now shows them an Acknowledge of their own, before Escalate, which answers their page through POST /on-call-page/acknowledge/<type>/<id>. UserOnCallLogTimelineService.acknowledgePage is the one place a page is acknowledged (the token links' hook calls it too), and a person's open pages say which is a page again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… language (#5123) The card, its choices, the folded More settings value, the Execution Logs line and the record page's Acknowledge failure, translated in the 16 Dashboard locales. The card's summary is one PluralTemplate, so en.json holds it once with its _one form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… resolved (#5123) On-Call Policies gains a section after Repeat a policy: where the Acknowledgement Timeout card waits (More settings on the Escalation Rules page), its two answers, what a page again does and what stops it, and the acknowledgementTimeoutInMinutes API field. In all 17 docs languages, line for line, naming each language's Dashboard labels. OnCallAdministrationDocsFacts holds the new words to the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… More settings (#5123) An offline Playwright suite of a policy's real Escalation Rules page: More settings folded at first and naming the card, the card's one row, the dialog's two answers and the minutes box (30 to start with, 5 to 1440), what each save sends (the minutes, or null to turn it off), a refused save, and all of it at 1280 and 390 pixels, dark and German. Wired into Dashboard Offline UI group 9 with its upload paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…files (#5123) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…repage The acknowledgement timeout's migrations move to 1805920000000 and 1805930000000, after master's AddIncidentPostmortemStatus pair took 1805900000000 and 1805910000000. Locale files merged with npm run i18n:extract; Postmortems and Acknowledgement Timeout both join Dashboard Offline UI group 9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d whole (#5123) AddOnCallAcknowledgementTimeoutMigration holds the two migrations to the models: three nullable columns with no default and nothing written, the partial clock index online under TypeORM's name, registration and order. Terraform E2E test 31 sets acknowledgement_timeout_in_minutes and changes it on update. The feed test compares the whole entry, links included, and the header test reads the posted route as an address. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
…ssage (#5123) UI review. A page again reaches people through their own notification rules, worded as a first page: the person who acknowledged the incident half an hour ago was called with "A new incident has been created", emailed "Incident INC-42 created", pushed "New Incident" and messaged "New incident assigned to you". Each channel now says what it is: the call opens "An incident is still not resolved 30 minutes after it was acknowledged", the email's subject and first line say it, the push title is "Incident INC-42 still not resolved", and Slack, Teams and Telegram head it "Incident still not resolved" and say why. The delivery reads once whether its run is a page again (and the policy's minutes); a run that cannot be read is worded as a first page and still goes out. The text message (one 160-character part, never calls it new) and WhatsApp (an approved template) are unchanged, and a first page reads exactly as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…summaries say it (#5123) UI review of the acknowledgement timeout's Dashboard surfaces: - The record's header showed "Acknowledge" right beside its "Acknowledged" pill to the person paged again, which read as a button that did nothing. It is "Acknowledge again"; a failure says "Could not acknowledge" and that they are still being paged, rather than repeating the button's word. - The minutes box said "After how many minutes? should not be less than 5." and "After how many minutes? is required.", and let a fraction through to the server. One sentence now covers every value it may not hold: "Enter a whole number of minutes between 5 and 1440." - The setting stays folded under More settings, but once a policy pages again, the two places it is read in a few seconds say so: the Escalation summary at the top of the same page (updated the moment the card is saved) and Policy at a glance on its Overview - "If it is acknowledged but not resolved within 30 min, it pages again, up to 5 times." A policy that does not page again reads as before. Seven new strings in all 16 locales (the orphaned "Could not acknowledge. Please try again." is gone), the docs in 17 languages, the jsdom suites, the docs facts and the offline browser suite follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…repage-ui-review The acknowledgement timeout's migrations are renumbered 1805960000000 and 1805970000000, after master's newest (1805950000000, the code production review). Locale files merged with i18n:extract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
simlarsen
pushed a commit
that referenced
this pull request
Oct 22, 2026
…ry; a page-again count is not a record number Common Test went red on master 7cf1f5d in two suites: - AIKillSwitchBackstop: #5172's ProductionReviewAI calls AIService.executeWithLogging and was not in the registry of call sites. It belongs in the skip lane: ProductionReviewRunner reads the project's AI switch first (isProjectAIEnabled) and, when AI is off, the review notes it and goes out with the production facts alone; a refusal while it reads becomes the review's AI note, never an error. It is registered with that reason. - RunRecord: #5173 gave OnCallDutyPolicyExecutionLog a numeric pageAgainNumber, and the suite takes any numeric "...Number" column of a model a workflow can watch for a record number that needs its "...WithPrefix". It counts which page again a run is (1 to 5) within one record's paging; the product's getRecordNumberColumns already reads no record number without a "...WithPrefix" beside it. The suite records that decision in NOT_RECORD_NUMBERS, as its own comment asks. Both suites pass (110 tests); Common's compile-tests type-check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For issue #5123 (a suggested reply to it is at the end).
The problem
Acknowledging a page stops every on-call policy for that incident, alert or episode, for good. When the person who acknowledged it is pulled away, falls back asleep or forgets, the record can sit in Acknowledged with nobody reminded of it. The policy's Repeat only covers the case where nobody acknowledges. The customer's workaround (a workflow that sleeps, re-reads the incident and pages) has to be built per project, and cannot show in the feed or the execution logs that the timeout fired.
The recommendation
Build it, as an advanced, opt-in policy setting with nothing else to decide: it is a real gap that teams moving from PagerDuty expect to be closed (its service-level Acknowledgement Timeout works this way), and the workaround is fragile. What we do not do is move the record back to un-acknowledged: OneUptime never moves a record back up its states (
Common/Utils/StateMove), and doing so here would write a state change into its history, email status page subscribers, post to its channels and start every workflow and SLA that waits on a state, all for a reminder. A page again is a run of the policy of its own instead, visible in the feed and the execution logs, and acknowledged, escalated or left alone like any page.What changed
One policy setting, folded away.
OnCallDutyPolicy.acknowledgementTimeoutInMinutes(nullable, no default: off on every policy there is). In the Dashboard it is the Acknowledgement Timeout card under More settings at the bottom of the policy's Escalation Rules page, folded by default (More settings names the card, and says "30 min" while it is on). Its dialog asks one question, If it is acknowledged but not resolved: Don't page again (what every policy does today) or Page again, and only then After how many minutes? (5 to 1440, 30 to start with). Everything else is decided once for everybody, so there is nothing else to pick:pageAgainNumber1..5) that starts at the level that was paging when it was acknowledged, pages whoever acknowledged it and whoever is on call there now (a rotation that changed hands pages its new person), and goes on through the levels like the first run.Server.
Common/Types/OnCallDutyPolicy/OnCallAcknowledgementTimeout.tsholds the rule and its words;Common/Server/Utils/OnCall/OnCallAcknowledgementTimeout.tsholds the clock (started from the four state timeline services and from answering a page), the claim (a compare-and-set, so each timeout pages again exactly once across workers), who is paged and the stops. A new minute worker,OnCallDutyPolicyExecutionLog:PageAgainAfterAcknowledgementTimeout, reads only running clocks through a partial index. The run and person-page workers learned that a page again is not over because its record is acknowledged (it was all along), only once the page again is acknowledged or the record resolved. The API and Terraform refuse anything but null or a whole number from 5 to 1440.What people see. The feed says "On-call policy Payments is paging again (1 of 5): the incident is still not resolved 30 minutes after Dana Kim acknowledged it." The run's status message and its Execution Logs row say "Paged again (1 of 5)". The run's Slack/Teams message says the policy is paging again, and, when nobody answers, raises the same alarm as a first page. The person paged again gets an Acknowledge again button on the record's page beside Escalate (the record's own Acknowledge is gone since it is acknowledged); it answers their page and starts the clock again. Their own call, email, push, Slack, Teams and Telegram message say the record is still not resolved so long after it was acknowledged, and the policy's Escalation summary and Policy at a glance say it pages again (see UI review below). The Escalate wording from #5167 is kept.
Migrations.
1805960000000-AddOnCallAcknowledgementTimeoutadds three nullable columns (no rewrite, no backfill);1805970000000-AddOnCallAcknowledgementTimeoutIndexbuilds the partial index online. Both numbered after master's newest (renumbered at the last merge, after the code production review's 1805950000000); the drift check is clean.Docs and words. On-Call Policies gains Page again if it is not resolved and the API field, in all 17 docs languages, line for line, naming each language's Dashboard labels. The 18 new Dashboard strings are translated in all 16 locales.
What changes for existing customers
Nothing until somebody turns it on: every policy starts at Don't page again, and the setting is folded under More settings. The new worker reads an index that is empty until then.
Left out, on purpose
Tests
Common/Tests/Types/OnCallDutyPolicy/OnCallAcknowledgementTimeout.test.ts: the rule, the bounds, the clock to the millisecond, the cap, the words.Common/Tests/Server/Utils/OnCall/OnCallAcknowledgementTimeout.test.tsandOnCallAcknowledgementTimeoutRecords.test.ts(an in-memory world,AcknowledgementTimeoutWorld.ts): fires once per timeout, restarts on a new acknowledgement, stops on resolve, delete, archive and turning it off midway, a changed timeout, minute-edge clock reads, who is paged after the rotation changed hands, the cap, two policies on one record, every record kind, the feed entry, and never throwing.OnCallDutyPolicyExecutionLogPageAgain.test.ts,AcknowledgementTimeoutHooksGuard.test.ts(an AST guard that the four state timelines and answering a page start the clock),OnCallPolicyRunMessagePageAgain.test.ts,OnCallPageAgainAcknowledgeAPI.test.ts,OnCallAcknowledgementTimeoutApiAndTerraform.test.ts.PageAgainAfterAcknowledgementTimeout.test.tsand the twoExecutePendingExecutionsPageAgain.test.ts.OnCallAdministrationDocsFactsholds the new section to the code; the translation suites hold the 16 translations to it.packages/E2E/AcknowledgementTimeout(offline, real page): folded by default, the dialog, what each save sends, validation, a refused save, 1280 and 390 pixels, dark and German. Wired into Dashboard Offline UI group 9.What ran locally
npx jest Guard: 162 suites, 5,061 tests passed.BILLING_ENABLED=true(12 of them need the plans CI sets with billing on, and pass with them).Tests/FeatureSet/Docs/Docs*, OnCallAdministrationDocsFacts and Translations, CriteriaGroupsPostmortemsDocsFacts (32 suites, 2,919 tests),docs:check-anchors,i18n:validate,i18n:check, and the Locale suites in Common (37) and App (10).ee/removed as CI runs it.npm run compilein Common, App and E2E,compile-testsin Common, andtsc --listFilesin Common, App, the Dashboard and E2E (nothing from the rootScripts/or another package); eslint and prettier on every changed file; the ReDoS checker (no new pattern that is not linear).Not run locally: the Terraform E2E (needs the full stack), the real-stack Playwright suites, and the whole App and Common suites (only the affected ones, as listed).
UI review
A second agent reviewed every surface this change adds, looked at each one for real, and polished it before merging.
What was looked at. The policy's Escalation Rules page (More settings folded and open, the card, the dialog with each answer, every refusal, a refused save) at 1280 and 390 pixels, light and dark, English and German, through the branch's offline fixture and the real On-Call route group in a harness. The record header for the person paged again (incident and alert) at 1280 and 390, light and dark, English and German, including a failed acknowledge, through the EventOverview fixture with an open page again. The Execution Logs (one policy's and all policies') at 1280 and 390, dark and German. The Escalation summary and Policy at a glance. The run's Slack Block Kit and Teams Adaptive Card messages (paging again, answered, nobody answered, the alarm) and the paged person's Slack/Teams DM and Telegram message, rendered at desktop and phone widths. The call script, the text message, the push and the email (the real template, rendered) that the person paged again receives. The feed entry, the run's status message and the docs section.
What changed, and why.
Tests added or changed in the review.
Common/Tests/Server/Services/OnCallPageAgainChannelMessages.test.ts(49 tests): the words; for each of incident, alert and both episodes, the call, email, Telegram, Slack/Teams message and push of a page again and of a first page; reading whether a delivery is a page again (with and without the policy's minutes, a first run, a direct page that is never read, a read that fails quietly); and a whole delivery throughexecuteNotificationRuleItemhanding the notice to each channel, once.App/Tests/Notification/OnCallPageAgainEmailTemplates.test.ts: the four page emails open with the page-again line instead of "A new ... has been created", a first page as before, and the line is escaped text.Common/Tests/App/Dashboard/AcknowledgementTimeoutPolicySummaries.test.tsx(14): the sentence, its bold and unbreakable numbers, the Escalation summary's row (durations, placement after the last step, absent when off) and Policy at a glance (what it reads, the sentence, absent when off, an out-of-range stored value read as the worker reads it).AcknowledgementTimeoutCard.test.tsx: one refusal for 4, 1441, 7.5, 0 and empty; 5, 1440 and 45 taken as typed.AcknowledgementTimeoutEscalationPage.test.tsx: the page hands the card's minutes to the summary, and a save reaches it at once.OnCallPageAgainHeader.test.tsx: Acknowledge again, never a bare Acknowledge, and the failure dialog's title and sentence.OnCallAdministrationDocsFacts: Acknowledge again and the summaries sentence pinned to the code.packages/E2E/AcknowledgementTimeout): the refusal for 4, 1441, 7.5 and empty, and the Escalation summary's line - absent while off, there at once after a save, gone again when turned off. Still 9 tests (assertions added to existing ones), about 25 seconds with its build.Left alone, on purpose. The all-policies Execution Logs table scrolls sideways at 1280 pixels (its Status pill was clipped the same way before this change). The token pages a text or email link opens ("Do you want to acknowledge this Incident?") read correctly for a page again. The mobile app's Acknowledge for a page again stays the follow-up the builder named.
What ran for the review (after its last edit and its merge of master):
npx jest Guard(162 suites), UpdateChecksReadHeldRows, DeleteChecksReadHeldRows, HookWritesCallersMayMake, ListOrderColumnModels, PermissionCatalogueCoverage, and the 37 Locale suites; the notification suites again withBILLING_ENABLED=trueand CI's plans.Tests/FeatureSet/Docs/Docs*(29 suites) and the policies page's own docs suites;docs:check-anchors,i18n:validate,i18n:check(no translation lost).ee/present.ee/hidden, as Common Test and App Test run them.ee/hidden.tscandcompile-testsin Common,npm run compilein App and E2E, the Dashboard'stsc; eslint and prettier on every changed file; the ReDoS checker (no new pattern).Suggested reply to the issue
🤖 Generated with Claude Code