[simlarsen]

Why

It's very hard to understand this step. Why should I pick between oneuptime AI and Runbook? Can oneuptime AI do things by default, and I can whitelist some Runbook for AI to pick and run? If I do not whitelist any Runbook, it can basically run any Runbook by default. [...] Please do this for alerts as well. In fact, in the auto remediation rule, you could ask which runner we should use to run the remediation. If no runner is selected, run on the global runner, but all of these should be in a more option. They should not be visible by default.

remove Run AI remediation commands as well, this should be in Incident / Alert remediation settings on which runner to run from (global runner by default) should be an optional setting.

What changed

An auto remediation rule's Fix step (Incidents → AI → Settings and Alerts → AI → Settings, under More settings → Auto remediation rules)

• The "Fix With" choice between OneUptime AI and Runbooks is gone. OneUptime AI fixes every match.
• The step asks one question, Approval: Ask before fixing or Fix without asking.
• More fields, folded, holds what most rules never change. Its folded line says what the defaults do ("OneUptime AI may run any of your runbooks, and fixes through the AI agent on the affected cluster or host"), or names what was picked:
  • Runbooks: the runbooks OneUptime AI may run. None picked shows Any runbook.
  • Run them every time: shown once the rule names runbooks. It starts every named runbook on every match without OneUptime AI deciding, which is what a "Runbooks" rule did before.
  • Runner: where OneUptime AI runs its own commands for this rule. It uses the existing Runner picker (RunnerPicker + RunnerChoicesLoader). The first choice, and the default, is the AI agent on the affected cluster or host. The picker never offers a global Runner or the in-cluster Runner installed by the Kubernetes agent chart.
• The rules table names each rule's fix in one Fix column: whether it asks first, and, in a quiet line, how it fixes for a rule that does not fix the way new rules do. Alert rules get the same form.

Runners

• The Runs AI Remediation Commands switch is gone from Create Runner and Runner Details. Create Runner is now a single page: Name, Description, and More fields (Runs Runbooks, labels).
• Picking a Runner, either as a rule's Runner or for a Kubernetes cluster's AI access, is now what lets OneUptime AI's commands run on it (RunnerAiCommandsPick.allowPickedRunners). It runs the same check the switch ran, under the project's credential-reach lock: a member who may not read runbook credentials cannot pick a Runner that holds SSH credentials. The canRunAiCommands column stays, and only OneUptime writes it.
• A Kubernetes cluster's AI page reads its Runners through RunnerChoicesLoader.

Engine

• A OneUptime AI rule picks a runbook from its own runbooks, or from any enabled runbook of the project when it names none. When the rule picks a Runner, OneUptime AI also composes commands on that Runner.
• A pick from the runbooks a Fix without asking rule names starts right away, as long as no AI agent round is already fixing the signal and the rule's hourly breaker allows it. Every other pick is proposed for one-click approval.
• The project's own pick (when no rule is set up) is recorded as AiPickingRunbook or AiRunbookPickNotStarted, classified as acted and needs attention.

Decisions made on your behalf

1. "Global runner by default" means the AI agent on the affected cluster or host, for OneUptime AI's own commands. Global Runners run JavaScript steps in a sandbox every project shares and refuse AI commands (GlobalRunnerMode.getJobRefusal); this PR does not widen that sandbox. So with no Runner picked, no Runner of your own is needed: OneUptime AI fixes through the AI agent on the affected cluster or host, as it does today. Runbooks OneUptime AI picks still run where each of their steps says, which is the Global Runner for a step that names none (feat(runners): global Runners, on by default like global probes; custom Runners for more security #5062). The rule's Runner applies only to OneUptime AI's own Bash and SSH commands; it never overrides a runbook step's Runner, since a step may need a particular host.
2. No runbooks named means any runbook, but OneUptime AI's pick from "any" is always proposed, never started on its own. That pick is driven by incident and alert text, which can come from outside systems, and "any" includes runbooks written for a person to run on purpose (failover, wipe). Before this PR every AI-picked runbook was proposed. Now a Fix without asking rule starts the AI's pick only from the runbooks it names. The Fix without asking card says so ("Runbooks start on their own only if you pick them under More fields"), and so do the docs.
3. "Run them every time" stays, folded and shown only once a rule names runbooks. A "Runbooks" rule runs every named runbook deterministically, with no LLM provider needed. Turning those rules into AI rules would stop them on installations without an LLM provider, start only one of several runbooks, and make the result depend on a model. The per-incident Runbook Rules don't replace it either: they have no approval, verification window or breaker.
4. Picking a Runner replaces the switch as consent, with the same credential check and lock as above. The Runner is allowed in the project of each rule the write saves: the request's project for a teammate, or each rule's own project for OneUptime's or a master admin's write.
5. Existing rules keep doing what they did (migrations 1804750000000-AddAutoRemediationRuleRunner and 1804800000000-AddAutoRemediationRuleRunnerIndexAndKey, renumbered past master's Twilio migration in the UI review):
   • A new nullable AutoRemediationRule.runnerId column. Its index and its key to Runner (ON DELETE SET NULL) are built online.
   • An older command rule that named no Command Runners ran on every Runner of its project with the switch on. Those Runners are now written out as its Command Runners. In-cluster agent Runners, global Runners, deleted Runners and other projects' Runners are left out.
   • A command rule that now names exactly one Runner it can run on picks that Runner as its Runner. A rule naming several keeps all of them.
   • Runbooks rules keep running their runbooks every time (the switch shows on).
   • A OneUptime AI rule drops runbooks it held out of sight. The old form hid the Runbooks field unless Fix With was Runbooks, but still saved what it held, so a rule switched from Runbooks to OneUptime AI kept them. Those runbooks now mean "the runbooks OneUptime AI may start without asking", so leaving them would have started runbooks nobody chose for that rule. They are dropped (DROP_RUNBOOKS_THE_FORM_HID); the rule reads as any runbook, always proposed. Rules that run their runbooks every time, and the two older AI shapes, keep theirs.
   • No Runner's switch changes. A cluster bound to a Runner whose switch was off keeps it off, and the cluster's AI page asks you to choose that Runner again.
6. Rules saved in the two older shapes (OneUptime AI composing commands on Runners, or OneUptime AI picking a runbook) keep working, and the table names them.

What changes for existing customers

• The rule form asks only Approval; runbooks and the Runner sit under More fields. Existing rules open with what they hold.
• New proposals: with Fix new incidents automatically (or alerts) on, an LLM provider and at least one enabled runbook, OneUptime AI now also proposes the runbook that fits each new incident or alert, when no rule is set up or for a OneUptime AI rule. Nothing runs without approval unless a Fix without asking rule names the runbook. This adds one AI planning run per such incident or alert.
• Runners have no Runs AI Remediation Commands switch. Over the API and Terraform, canRunAiCommands becomes read-only: a write that names it is refused (Terraform's attribute becomes computed, as feat(runners): a code repository picks its Runner; Runners lose Runs AI Code Fixes #5086 did for canRunCodeFixTasks).
• A cluster bound to a Runner with the switch off says so on its AI page and asks you to choose the Runner again.
• Docs in 17 languages: AI SRE (rules section), Runbook Agents, Runbook Credentials, Permissions, Workflow steps, Incident settings, and an upgrade note in Upgrading.

Left out

• No project-wide Runner for remediation: the Runner is a rule's choice, as the maintainer asked ("in the auto remediation rule"). A project with no rules uses the AI agent; to run OneUptime AI's commands on a Runner for every incident, add one rule for All incidents and pick the Runner there. A second, project-level place to choose a Runner would be one more setting to explain.
• The rule's verification window, auto-resolve on verified recovery and command allowlist stay API and Terraform only, as before.
• The table's Fixed by column became part of one Fix column in the UI review: "OneUptime AI" on almost every row said nothing, so only a rule that does not fix the way new rules do says how it fixes.

Tests, and what each proves

• Common, types: AutoRemediationFix (how every stored shape reads; allow-list ids; the Runner a rule's commands may use; when an AI pick may start), AutoRemediationDecision (the new reasons are acted / need attention).
• Common, engine: AutoRemediationDecisionRecording (+21: the no-rule pick for incidents and alerts, once per signal; a rule naming none always proposes; naming some and Fix without asking starts, unless another rule asks first, an AI agent round fixes the signal, or nothing is enabled; no LLM provider; a rule's Runner adds a command run; an every-time rule picks nothing), RemediationPlanRunner (+17: candidates are the rule's runbooks or every enabled one; the framing says proposed or started; a start re-reads the rule under the breaker lock; breaker, failed start and lost race propose or settle; a proposed pick or the project's pick never starts), RemediationCommandTools and RemediationExecutionRunner (commands target only the Runners a rule names, never "any Runner").
• Common, services: AutoRemediationRuleRunner (a rule's Runner: refused for an agent Runner, allowed by the pick, written as id or relation), AutoRemediationRuleUnattendedCommands (widening checks read the rule's Runner), RunnerAiCommandsPick (the pick turns the column on only for the project's own non-global, non-agent Runners, and refuses a Runner with SSH credentials to someone who may not read them, under the lock), the Kubernetes cluster binding suites (binding is the consent; the gap text), RunnerServiceGlobalRunner, OwnedScopeEveryKindWithOwners, GlobalRunnerChoiceGuard.
• Common, migration: AddAutoRemediationRuleRunnerMigration (the statements, their order and registration) and AddAutoRemediationRuleRunnerPostgres (on real rows: Runners written out and picked, agent rows by the server's own rule, every-time and older rules kept, hidden runbooks dropped only from OneUptime AI rules, no Runner's switch changed, the key validated online, rollback). Opt-in; the schema drift workflow runs it.
• Common, Dashboard: AutoRemediationRulesTable and AutoRemediationRuleFields (for incidents and alerts: no Fix With; Approval up front; More fields folded with its line; Runbooks none = any; every-time switch; the Runner picker; what a save sends), RunnerPickerModel (the remediation picker's first choice and helper text), RemediationDecisionLines, RunnerKubernetesAgentForm (the one-page Runner form without the switch), the Kubernetes AI page suites.
• App: docs suites, including the new AutoRemediationRunnerUpgradeDocs (the upgrade note in 17 languages), AISettingsSwitchesFirst, RunnerUi.
• E2E, offline: npm run test-auto-remediation-rules-ui (Dashboard Offline UI group 7 since the last master merge, about 2 minutes with its bundle): the real pages in Chromium - create and edit, More fields, the Runner choices, rules saved before, alert rules, a viewer, a phone, dark mode and German. GlobalRunners and the deployed RunnersUnderRunbooks spec check the Create Runner dialog has no switch.

Verified locally

After merging master at 387b345, with ee/ hidden as CI runs them and billing on:

• The full App suite: 1,928 suites pass (2 skipped).
• The full Common suite: 6,660 pass (108 opt-in suites skipped). It caught two repo-wide guards this branch broke, both fixed in 8cfe7d4 and re-run green with every related suite:
  • HookWritesCallersMayMake: RunnerService's create hook wrote canRunAiCommands, which no caller may write any more. The hook no longer writes it, and the column's default (off) applies.
  • UpdateChecksReadHeldRows: the rule service read an update's rules by its query. It now reads the projects through findProjectsToCheckUpdateIn / getProjectsToCheckRowsIn. As a result, a master admin's update made in another project allows the Runner in the rule's own project.
  • The third failure, MarkdownEditorLongLines (master's fix(markdown-editor): the editor reads and writes a note in time in line with its length #5091), is a 2-second timing check that ran 2.4 s under two full suites at once. It passes alone.

After merging master again at 3e2cf23 (#5097 nginx, then #5092 Profiles):

• Conflicts resolved: the Dashboard locales (with i18n:extract during the merge, keeping both sides' keys and translations; the extract also re-adds 43 keys from ee/ and other PRs' models that no committed locale carries, so those were dropped again), test.app.yaml group 6 (both suites; see below), and packages/E2E/package.json (both scripts).
• Type checks pass for Common, App (with its tests), the Dashboard and E2E, and Common's compile-tests passes. No package reads files outside itself and Common (tsc --listFiles).
• Every Common guard (151), Common's and App's locale suites (31 and 9), the 279 related Common suites and 223 App suites pass: the related ones, App's root-level tests, and the 167 docs suites that read the changed pages plus the general ones. Before this merge the related suites also passed with billing off. i18n:check passes, and the Ops suites that read test.app.yaml (E2ESuitesRunInCi and three more) pass.
• Seen failing, and failing on master the same way (not this branch): Tests/Server/API/ExpiredSessionAIRoutes.test.ts. feat(profiles): the Profiler answers where the time goes and what changed; Function Share monitors, Fix with AI, profile evidence for AI #5092 added /ai-investigation/create-profile-fix-task and /ai-investigation/profile-fix-suggestion without adding them to that suite's table. This branch has both files exactly as master does.
• CI time: group 6 of Dashboard Offline UI took 16.2 to 22.7 minutes on master's runs today (18.4 on the last run every group finished). The Profiler (feat(profiles): the Profiler answers where the time goes and what changed; Function Share monitors, Fix with AI, profile evidence for AI #5092, about 2 minutes) and this suite (about 1.5 minutes: 47 seconds of tests plus the bundle) joined it the same day, which puts it at about 22. The comment above it says to move one of the two to the lightest Chromium group if it passes that.
• On a fresh Postgres 15 with every migration run, the migration suites pass (37) and the schema drift check finds no drift.
• The offline Playwright suites pass: auto remediation rules (16) and global Runners (30).
• docs:check-anchors, i18n:validate and the Dashboard's i18n:check pass, and no anchors are left to localize. The ReDoS checker finds no new patterns. Prettier and eslint pass on every changed .ts and .tsx file.
• Not run: the deployed E2E spec RunnersUnderRunbooks.spec.ts, which needs a live stack. Its selectors drive the same Create Runner dialog as the offline GlobalRunners suite, which passes.

UI review

A second agent reviewed every surface of this PR with fresh eyes, rendered for real (the offline fixture's production pages in Chromium with Inter, plus a scratch harness of its own for languages, data shapes and states), and polished what kept it from great.

Looked at: the rule's create and edit dialog on every step state (folded and open More fields, Ask / Fix without asking, runbooks picked, Run them every time on and off, the Runner menu, no Runners, Runners loading, Runners unreadable, long Runner and runbook names), the rules table with rules saved before (all four shapes) and long rule names, both lanes (incidents and alerts), at 1280px, at 1000px (the card's real width beside the Dashboard side menu at a 1280px window) and at 390px, light and dark, in English, German, Japanese, Russian and Persian; the AI settings switches; Create Runner, Runner Details and the Runners list (GlobalRunners suite); the remediation feed items as Slack (Block Kit section) and Microsoft Teams (Adaptive Card TextBlock) draw them, built with the product's own converters, at desktop and phone widths; the Remediation card's suggestion lines.

Changed, and why:

• "Fix without asking" no longer says it "starts the runbooks you picked" when nobody picked any - with none picked, OneUptime AI's pick is proposed. It now reads "OneUptime AI fixes right away where the AI agent's settings allow it. Runbooks start on their own only if you pick them under More fields." (the handoff's open copy point).
• One Fix column instead of "Fixed by" + "Approval". "Fixed by" said OneUptime AI on almost every row, and with the older shapes' long pills the table ran past its card: Status slid under the sticky row actions at 1280px. The Fix column shows the approval pill and, only for a rule that does not fix the way new rules do, a quiet line that wraps ("Its runbooks, every time", "OneUptime AI: commands on Runners", "OneUptime AI: picks a runbook" - the docs' names). The table now fits a 940px card with every shape and long names; a new offline test measures it.
• Rules saved before rules were simplified say what they do. Their edit dialog showed fields that did nothing for them and contradicted the table: an "AI picks a runbook" rule showed Fix without asking selected and a Runner that is never used, while the table said Asks first. Now the Fix step starts with one line ("This older rule only has OneUptime AI pick one of its runbooks, and always asks first." / "... run commands on its Runners."), hides what does nothing for that shape (Approval and Runner for a picking rule, Runbooks for a commands rule), and offers Fix like a new rule: it clears the two flags (the form now carries them, never drawn) and asks before fixing until someone chooses otherwise - the rule does more than it did, so nothing runs unattended from that one click. Cancel changes nothing.
• The Runner field says less. The helper under the picker repeated its own choice ("OneUptime AI fixes through the AI agent ... Pick one of your Runners to let it run commands there too." - "there" was ambiguous). With Runners to pick from there is no line now; in a project with none, it says where one is added ("Add a Runner under Runbooks › Runners ... then pick it here", in a new tab) - never to someone who could not read the Runners (new RunnerChoices.areProjectRunnersUnread).
• The folded line no longer names "Run them every time" while it is off ("Runbooks: 2, Run them every time, Runner" read as if it were on). New generic Field.isListedWhileFolded (Common, one if in getFoldedFormFieldItems); it is named while on ("Run them every time: On").
• AI settings: the fixing switch's note is one sentence: "Nothing runs without approval unless the AI agent's settings or a rule allow it." (was two, one of them "waits for yours").
• Remediation card: the project's own runbook pick (no rule set up) read "Rule: OneUptime AI", as if a rule had that name; it reads "OneUptime AI" now. A pick whose rule was deleted still names the rule. (DEFAULT_RUNBOOK_PICK_NAME moved to AutoRemediationFix with isProjectRunbookPick.)
• Kubernetes cluster → AI → Agent: a cluster bound to a Runner whose switch was off said "With Change below, clear the Runner and choose it again", which saved nothing when done in one dialog (the modal sends only changes), and its step linked to the Runner's page, where nothing turns it on any more. The step now says "Choose the Runner again to let OneUptime AI run kubectl on it." with a Choose it again button that writes the same Runner back through the server's own checks (credential read, binding), a notice when done and the server's refusal when not; a reader is told who to ask. The server's next step for other surfaces says the same.
• "Fix" is translated in 15 locales (it was English in all but Japanese): the dialog's step and the new column. ("Rule" stays: the SLO docs bold it as the Dashboard draws it.)
• Docs (ai/ai-sre, all 17 languages): one sentence on Fix like a new rule.
• The offline suite's default port is 4395: 4391 was the Profiles suite's default too, in the same CI group.

Left as they are, on purpose: the feed messages (bold headline plus one or two plain sentences, consistent with the other remediation items, and they read well in Slack and Teams at both widths); Create Runner's Description line ("Optional description for this agent" is the model column's, as on master); the Runner picker's folded value (it names the Runner field, not the Runner: the name is known only once the picker has loaded).

Tests added or changed: AutoRemediationRulesTable.test.tsx (+the Fix column, the older-rule note and its switch, every field's visibility for each shape, the folded listing; 70 tests), AutoRemediationRuleFields.test.tsx (no helper with Runners, where to add one with none, not when unreadable, the note component), new RemediationSuggestionCardSource.test.tsx (the project's pick, a rule's, a deleted rule's; isProjectRunbookPick), FoldedFormFields.test.ts (isListedWhileFolded), KubernetesClusterAiAgentStatus.test.ts and KubernetesClusterAiPageServerParity.test.ts (the step text and choose_runner_again), KubernetesClusterAiPage.test.tsx (+3: one click writes the same Runner, a refusal is shown, a reader is told who to ask), AISettingsSwitchesFirst.test.ts (the note), the offline Playwright suite AutoRemediationRules.spec.ts (now 20 tests: the Fix column, the table fitting its card at the real 1280px width, the no-Runners hint, the folded line, both older shapes kept, switched with Fix like a new rule, and left alone by Cancel; about 10 seconds more).

Verified after the UI review, on the branch merged with master twice:

• At 027bbf1 (Project Settings: Import from another tool moves from Basic to the end of Advanced #5098, AI / LLM: connect provider accounts and company AI tools without code; Setup becomes Connect #5095, fix(ai): OneUptime AI's own calls record what they said, one conversation per run #5100), conflicts in the locales (a three-way merge by key, both sides' keys and translations kept), the migrations index, the E2E scripts and test.app.yaml (master had moved the Profiler and LLM Connect to a new group 7 and this suite joined them, the lightest group).
• At 4b25a22 (feat(twilio): Twilio configs list their own SMS numbers, for countries like Singapore where no number does both calls and SMS #5099). Its AddTwilioSmsPhoneNumbers1804700000000 took a slot above this branch's two migrations, so they were renumbered past it: 1804750000000-AddAutoRemediationRuleRunner and 1804800000000-AddAutoRemediationRuleRunnerIndexAndKey, registered last, SQL unchanged. Every migration then ran on a fresh Postgres 15 in that order, the schema drift check found nothing, and the opt-in AddAutoRemediationRuleRunnerPostgres suite passed on the migrated database (21). The 17 Dashboard locale files merged on their own; they match a key-by-key three-way merge (31693 keys each, same order).

Checks run:

• Type checks: Common (tsc and compile-tests), the Dashboard, App with its tests, and E2E; --listFiles for Common, App and E2E lists nothing from another package or the repository's Scripts.
• Every Common guard (153 after the second merge), HookWritesCallersMayMake, UpdateChecksReadHeldRows, SchemaMigrationsOrdering, master's LongFormStepsGuard and SmallWizardsOnePageGuard, Common's and App's locale suites, every App guard (41; ShortDescriptionsGuard caught the two never-drawn flag fields showing their columns' API descriptions - fixed with hideColumnDescription; after the second merge it fails only on master's Twilio copy, below), App's root-level tests, AutoRemediationRunnerUpgradeDocs and CodeRepositoryRunnerUpgradeDocs.
• The suites that read the changed modules: 266 in Common (5 opt-in skipped) and 57 in App; 134 docs suites (the ones reading ai-sre, upgrading and the Runner pages, plus the general Docs* suites).
• i18n:check (sound, no translation lost; the extractor finds nothing missing), i18n:validate, docs:check-anchors, the Ops suites that read test.app.yaml (E2ESuitesRunInCi and three more), and the ReDoS checker (no new patterns).
• Offline Playwright: Auto Remediation Rules (20, about a minute plus the bundle) and Global Runners (30).
• The new and changed suites with ee/ hidden, as CI runs them.
• Prettier and ESLint on every changed .ts/.tsx file of the branch.

Not from this PR: ShortDescriptionsGuard fails on master since #5099 - the Twilio SMS numbers description is 192 characters (limit 150) in the Dashboard's Twilio config table and the Admin Dashboard's Call and SMS settings. This branch does not touch either, so it fails the same way here; the fix belongs with the Twilio feature.

Not run: the deployed RunnersUnderRunbooks.spec.ts against a live stack (its selectors match the offline suite, which passes), a live LLM and Runner, the Kubernetes AI page in a browser (its new button is the same Button, placement and flow as the existing Turn on in the same list, covered by three jsdom tests).

🤖 Generated with Claude Code

[simlarsen]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.