[simlarsen]

The problem

We have this button in all traces and not traces that are slow. Can you please fix it? Suggestions like this should be intelligent across the product. Please audit the whole product and make suggestions like these intelligently. Please make the product easy and simple for people to understand. Only show things when necessary.

Fix performance with AI sat in the header of every trace, fast or slow. Clicking it on most traces only got the server's refusal ("No deterministic performance pattern was found"), and even when the span-tree analysis found a "pattern", it was often a 3 ms query that was "most" of a 4 ms request, or a one-span health check that is "100%" of itself. The same shape repeated across the product: AI actions offered where they could not run (AI switched off, no repository for the pull request, a fix already on its way).

What changed

1. The trace page: "Fix performance with AI" only on a trace worth fixing, saying what it found

• The action moved out of the header's button row into a slim finding strip inside the header card, shown only on a trace worth fixing. It says what was found in plain words, what it costs, and offers the one next step:
  • "The same database query ran 40 times in ReportService.build" · SELECT * FROM line_items WHERE invoice_id = $1 · "Together they take 960 ms of this 1.72 s request. Show them" · [Fix performance with AI]
  • "4 calls to GET /inventory/{n} ran one after another in reserve stock" · "…and they could run at the same time."
  • "One database query took most of this request" / "POST /api/v1/import took most of this request" · "It takes 2.1 s of this 2.4 s request on its own."
• Show them puts the finding's spans in view in the waterfall (their shared name as the search, the first one selected).
• A trace with nothing worth fixing shows nothing: no disabled button, no "not slow" note. The header keeps only Copy link and Refresh.
• After a click the strip says "OneUptime AI is working on a fix. View task progress"; a refusal is shown beside the finding ("Could not start the fix: …") and the button stays to try again.

The rule (decided as a product designer): a trace is worth fixing when the span-tree analysis finds a known pattern that costs the request real time:

Pattern	Analysis finds it when	Worth a fix when it costs
The same call again and again (N+1)	5+ near-identical calls under one parent	100 ms or more together
Calls one after another that could run together	3+ same calls, strictly sequential, half of their parent or more	100 ms or more together
One call that takes most of the request	60%+ of the trace in its own work	500 ms or more on its own
Every pattern		and at least a tenth of the request

Why absolute floors and not "slower than this endpoint usually is": an endpoint that always runs the same query 40 times is the one most worth fixing, and it never looks slower than its own usual; the floors need no query beyond the trace itself, so the page's suggestion and the server's create gate decide from exactly the same findings. A slow trace with no known pattern gets no button: the AI would have nothing grounded to fix (the server refuses it too). The 500 ms floor for "one slow call" keeps ordinary calls (a 300 ms payment provider call, a 150 ms query) from asking for a pull request.

One analysis, shared by the page and the server. The analysis moved to Common/Utils/Traces/SpanTreeAnalyzer.ts (pure). The rule is Common/Utils/Suggestions/TracePerformanceFixSuggestion.ts. The page asks POST /ai-investigation/performance-fix-suggestion, which runs FixPerformanceTaskTrigger.getSuggestionForTrace — the same gates create-performance-fix-task checks a click against (AI on, a finding worth fixing, a repository, no fix already in flight). The analysis needs the spans' attributes (db statements group the N+1), which the trace page does not load, so the server answers; a project with AI off is not even asked. A test runs every fixture trace × AI on/off × repository/none × run in flight/none and holds that the page offers Fix exactly when the create route accepts the click.

AI off or no repository — decided:

• AI switched off: hide. The project chose no AI (Project.enableAi is its one AI switch); nothing on the page could start, and a nudge on every slow trace would be noise. Ask AI in the header stays — it is the quiet way back (it says AI is off and links to the switch).
• No repository connected: a quiet link — "Connect GitHub to fix this with AI" in place of the button. The finding is useful by itself, connecting a repository is the single step between the reader and the fix, and the link appears only on traces worth fixing, so it never nags. (For suggestions with no evidence on screen — Improve logging/tracing — a missing repository hides the suggestion instead; see below.)

The server's create route now also refuses with AI off (AI_DISABLED_MESSAGE) and keeps only the findings worth fixing in the task's evidence. Insight fix routing (the automatic latency path) follows the same rule. normalizeDbStatement keeps positional placeholders ($1) so the query reads as written (linear pattern, 400k-character timing test).

2. The audit: every suggestion shows only when it applies

Suggestion / AI action	Where	Before	After
Fix performance with AI	Trace page	Every trace, always	Only on a trace worth fixing, naming the finding; AI off → nothing; no repository → quiet "Connect GitHub to fix this with AI"; fix in flight → link to it
Improve logging with AI / Improve tracing with AI	Service → Logs / Traces	Every service, always (AI off, no repository, one already running)	AI on and a repository → the card; one in flight → "…improvement in progress · View task progress"; otherwise nothing (no leftover spacing). New POST /ai-investigation/telemetry-improvement-suggestion; create route refuses with AI off
AI Assistance (Fix with AI, Generate Regression Test, Improve Error Handling)	Exception side menu + page	Always in the menu; start buttons with AI off	AI off → not in the menu, the page says AI is off and who can turn it on, the server refuses; repository/provider/Runner setup checklist unchanged
Open Fix PR from this analysis	Incident/alert AI Investigation card	When the investigation recommended a code fix	+ only while AI is on (kept after a fix was started here, for its link)
Draft with AI (Generate Public/Private Note with AI)	Notes on incidents, alerts, maintenance, episodes	Whenever the feed supports AI	+ only while AI is on
Generate with AI	Incident and episode Postmortem cards	Always	Only while AI is on (Apply Template already only with a template)
Explain with AI	Log error pattern panel; investigation drawer (metric explorer, charts, dashboards)	Always	Only while AI is on
Insight fix routing (automatic)	AI Insights → latency regressions	Any span-tree finding	Only findings worth fixing (same rule as the trace page)
Ask AI (header, Cmd+K)	Everywhere	Always	Unchanged on purpose: the way back — opening it says AI is off and links to the switch
Ask AI suggested questions	Ask AI panel	Static per page type	Unchanged (follow-up: record-aware questions need a read per panel open)
AI Investigation card's "not investigated" reasons	Incident/alert	Explains why AI did not run, incl. AI off	Unchanged: deliberate design (#4181) — the maintainer's call
Operations view "×N from one parent"	Trace → Operations	≥10 calls from one parent	Unchanged: a description, not a suggestion
Monitor recommendations	Resource side menus	Catalog − covered − dismissed, telemetry-aware	Already applies
AI Insights "Investigate in Logs/Metrics/Traces/Exceptions"	Insight page	Only when the insight has that evidence	Already applies
Apply Template	Postmortem card	Only with a template (#4363)	Already applies
"Ask AI to do something"	Code	Code's blocker notices (CodeTaskGate)	Already applies
Investigate (window)	Metric explorer, charts, dashboards	Always	Unchanged: deterministic, applies to any window
Create monitor from this view	Metric explorer → Actions	In a menu	Unchanged
Recording health Explain it	Session replay	Local diagnostics explainer, not AI	Unchanged
Setup checklist, Home tiles, empty-explorer first steps	Home, User Settings, explorers	Earlier audits (#4162, #4972–#4990)	Unchanged
AI agent pages	Kubernetes, Host, Docker, Podman, Proxmox, Docker Swarm	Always in the side menu	Unchanged: where the agent is set up, not an action on a record (see Left out)

How AI-off is known without a request: the project list (list-user-projects) now selects enableAi, the Dashboard follows Enable AI's saves (SelectedProjectSwitches), and Utils/UseAiOffered (useAiOffered() / isAiOfferedForSelectedProject()) answers from the selected project through Common/Utils/Suggestions/AiSuggestion.isAiOffered — fail open (an unknown value reads as AI on; the server refuses either way).

The Common helpers (for later tasks — profiles-product-overhaul follows them)

packages/Common/Utils/Suggestions/ — pure, shared by server and Dashboard, one per kind of suggestion:

• SuggestionOffer — None | Act | SetUp | InProgress, and parseSuggestionOffer (unknown → None).
• AiSuggestion.isAiOffered(project) — the one AI switch (=== false, fail open).
• CodeFixSuggestion.getCodeFixOffer({isAiEnabled, applies, hasActiveFix, hasConnectedRepository, isBackedByEvidence}) — the shared decision for "AI opens a pull request" suggestions.
• TracePerformanceFixSuggestion — WORTH_FIXING_MIN_MS, WORTH_FIXING_MIN_PERCENT_OF_TRACE, isFindingWorthFixing, getFindingsWorthFixing, getTracePerformanceFixOffer, the wire format and its parser.
• TelemetryImprovementSuggestion — getTelemetryImprovementOffer and its wire format.

A new suggestion adds a helper here, its server route answers with it (the same gates as the action's create path), and its page draws only what the answer says. SuggestionRulesGuard holds it.

What changes for existing customers

• Traces: the AI button disappears from most traces; it shows only on traces with a finding worth fixing. The API's create-performance-fix-task now refuses traces with nothing worth fixing (same message the docs give) and refuses with AI off.
• Projects with AI switched off: no AI action buttons anywhere (notes, postmortems, Explain with AI, trace fix, improvement cards, exception AI Assistance, the investigation's Open Fix PR). Ask AI stays. The create routes for performance fixes, logging/tracing improvements and exception fixes refuse with AI off (before, they queued runs that failed at the first model call).
• Projects without a connected repository: no Improve logging/tracing cards; on a trace worth fixing, a quiet "Connect GitHub to fix this with AI".
• Automatic insight fix PRs for latency regressions are opened only for findings worth fixing.

Tests

• Common/Tests/Utils/Suggestions/TracePerformanceFixSuggestion.test.ts — realistic span trees through the real analysis: slow checkout with an N+1 (offered, named), fast request with the same loop, health check, fast clean request, slow request with no pattern, ordinary 300 ms payment call, long batch with a small N+1 (all not offered), calls one after another, one slow query, one slow span (offered); thresholds at their edges; AI off / no repository / fix in flight; wire format round trip and every garbled answer.
• SuggestionHelpers.test.ts, TelemetryImprovementSuggestion.test.ts — the shared offer, the AI switch, every combination of the code-fix inputs.
• Tests/Utils/Traces/SpanTreeAnalyzer.test.ts (moved) — plus positional placeholders and a 400,000-character linear-time test.
• FixPerformanceTaskTrigger.test.ts — AI off, nothing worth fixing, only worth-fixing findings stored, the suggestion's every branch, span loading under the caller's permissions, and the server agrees with the page (10 traces × AI × repository × run in flight).
• TelemetryImprovementTaskTrigger.test.ts — AI off refusal, every suggestion branch, the agreement matrix for both recipes.
• AIInvestigationPerformanceFix.test.ts — both trace routes end to end; ExpiredSessionAIRoutes covers the two new routes.
• FixRouting.test.ts, InsightFixRouting.test.ts, TelemetryExceptionCodeFixRun.test.ts — routing only worth-fixing findings; exception fixes refused with AI off.
• Dashboard: TraceExplorer.test.tsx (17 in a new block, replacing the old two: offers, refusals, AI off, garbled answers, Show them, refresh), TracePerformanceSuggestion.test.tsx, TracePerformanceSuggestionInLocale.test.tsx (German, Japanese, Russian from the shipped locales), TelemetryImprovementCard.test.tsx, UseAiOffered.test.tsx, ErrorPatternExplainWithAi.test.tsx, plus AI-off cases in EventNotes, PostmortemPageParts, InvestigationArtifacts, InvestigationPanel, ExceptionAIAssistance, ExceptionDetailSideMenu, SelectedProjectSwitches; ProjectSettingsSwitchesSaveOnFlip holds Enable AI among the selected project's switch columns and in the project list's select.
• Guard: Common/Tests/App/Dashboard/SuggestionRulesGuard.test.ts — helpers pure; thresholds written once; every Ask-AI-with-a-prompt opener, every Generate-with-AI renderer and every code-fix starter asks useAiOffered(); no page reads enableAi itself; every server code-fix creator asserts the AI switch.
• Offline E2E (packages/E2E/TraceDetail): the fixture answers the suggestion route with the real shared analysis on its own spans; new report trace; 10 new specs (no button on the checkout trace, the finding and the fix, task link, refusal, Connect GitHub, fix in flight, AI off, failed suggestion, Show them, phone layout). The whole suite: 42 passed in 1.3 min locally (+~16 s for the new specs).
• Docs (17 languages): traces explorer's tip and the Code page's "Where fix tasks start" (trace row + a paragraph on when the buttons show); TelemetryExplorersDocsFacts and AiDocsFacts pin the numbers and wording to the code.

Also fixed on the way

TraceSignals reset its tabs in a mount effect after the first paint, so a tab picked in that window snapped back to Logs (the metrics tests were timing-lucky). The reset now runs only when the trace changes.

Left out

• Ask AI's suggested questions stay per page type. Making them about the record on screen needs a read each time the panel opens; a follow-up.
• The AI Investigation card's "not investigated" reasons still show with AI off. That card was made one flat card on purpose (One AI Investigation card on incidents and alerts, with the Ask AI conversation in it #4181), and saying why nothing ran is its job; it is the maintainer's call whether AI off should hide it.
• Plan, LLM provider and Runner readiness are not part of the trace and logging/tracing suggestions. Their create routes do not check them either (the run fails at its first model call), so the suggestion asks exactly what the click is checked against. Adding them to both is a follow-up.
• The exception page's setup checklist (repository, provider, Runner) is unchanged. Only AI off is new there.
• The infrastructure AI agent pages (Kubernetes, Host, Docker, Podman, Proxmox, Docker Swarm → AI agent) are unchanged. They set the agent up and offer no action on a record. Saying there that AI is off is a follow-up.
• Three unused English keys of the old header button stay in the locale files ("Performance fix task created", "Could not create the performance fix task", the old tooltip). i18n:check reports the files sound.
• A full-stack run (real spans in ClickHouse through the real route) was not done. The offline suite runs the real analysis on its fixture's spans, and the API tests run both routes with the reads mocked.
• The notes and postmortem docs pages were not each given an AI-off line. The rule is written once, on the Code page ("These buttons show only where they can start a task").

What ran (after merging master)

• Common: 255 suites (every suite that reads a changed file, every *Guard* suite, the side-menu guards): 254 passed, 1 skipped; 8,899 tests.
• App: 91 suites (every suite that reads a changed file, App's root-level tests, the docs suites): all passed; 13,084 tests.
• Both runs were made with ee/ in place and again with ee/ moved aside. The first run found two things, both fixed here: the switch-columns wiring test now expects Enable AI, and a service worker my local build generated (public/sw.js, ignored by git) tripped a pattern guard until it was removed.
• Docs suites for the changed pages and their translations: 45 suites, 15,667 tests passed.
• Offline browser suite packages/E2E/TraceDetail: 42 passed (1.3 min).
• npm run compile-tests (Common), npm run compile (App, E2E) and the Dashboard's tsc: clean.
• prettier --check and eslint on every changed file: clean. The branch's ReDoS check found 0 patterns that are not proven linear (the moved analyzer's patterns were checked separately: all linear). npm run i18n:check: every locale sound.
• Not run: the full Common and App suites (only the suites that read what changed), and CI's other browser suites.

UI review

What I looked at. The real trace page (the offline fixture, with production's Inter font, Theme.css dark remaps and the Dashboard's locale files) at 390, 640, 768, 900, 1024, 1280 and 1440 px, light and dark: every finding's wording (the same query again and again, the same call, queries and calls one after another, one slow query, one slow call, a 120-character parent name, a long SQL statement), every state of the strip (the fix, its loading state, Connect GitHub, a fix on its way, after the click, a refusal), "Show them" at each width, and the strip in German, French, Spanish, Italian, Russian, Japanese, Chinese, Hindi and Persian (the fix button in all 17 languages, measured). The improvement card on a service's Logs and Traces pages (offered, its confirm dialog, created, in progress, refused) at 1280 and 390 px, light and dark, in English, German and Russian. An exception's AI Assistance page and side menu with AI off (the offline exception fixture). The AI-off removals (Draft with AI, Generate with AI, Explain with AI, Open Fix PR) in code: nothing is left behind (no empty toolbar, gap or card).

What I changed, and why.

1. The strip's layout holds at every width. From md up the next step sat beside the words and squeezed them into a narrow column (the German headline took four lines at 768-900 px). Now it sits under the words, lined up with them, until lg, and beside them from lg up. On a phone the fix button spans the strip, as the header's own buttons do (beside the clock its label broke in two); the working line and the Connect GitHub link line up with the words instead of starting at the strip's edge.
2. Clearer marks. A clock marks the finding (the bolt is the fix's own, and Critical path's); the GitHub mark is on "Connect GitHub to fix this with AI" (it was a code icon). "OneUptime AI is working on a fix" has Code's live dot and is a status a screen reader announces when it takes the pressed button's place; "View task progress" stays in one piece where there is room. The strip's links share one quiet style with a keyboard focus ring. The query shows on up to two lines (on a phone one line read "SELECT * FROM line_items WHER…").
3. A refused click is said in the strip's words. The page asks the server again after a refusal: a fix someone else started meanwhile is linked ("working on a fix"), and a repository disconnected since the page loaded turns the button into the Connect GitHub link - instead of "No GitHub-App-connected repository exists for this project, so the agent has nowhere to open the performance-fix pull request". Any other refusal (a daily limit, say) keeps its reason beside the button. The improvement card does the same: an improvement started meanwhile is linked, not reported as an error.
4. "Show them" shows them. Below xl (1280 px) the span panel is a sheet over the page, and "Show them" opened the first span in it - covering the very spans it was asked to show. There, the spans the search marks are now left in view; a single span ("Show it"), or spans with no shared name to search by, are still opened. At every width "Show them" lets go of what would hide them (errors only, service chips, an earlier search) and scrolls the waterfall into view.
5. The phone toolbar (a bug on master too, found while checking "Show them"): the span search was squeezed to nothing beside the two filters, and its match count and arrows spilled out of the card to the left. It has a row of its own on a phone.
6. Exception AI Assistance with AI off shows Code's own notice - the tasks this page starts are Code's: "AI is turned off for this project." with Turn AI on for whoever may change Enable AI, who can for everyone else, and the sentence alone until permissions arrive. It was an info alert with no way forward.
7. Every language. "Fix performance with AI", "Show them", "View task progress" and the improvement card's titles and states were still English in 12 to 15 locales (the strip read "… Show them [Fix performance with AI]" in Russian, Italian, Chinese and the rest); they are translated in all 16. The button label broke into two lines on a 390 px phone in German, French, Spanish, Italian, Portuguese and Dutch, so there it says "speed up with AI", as Russian does ("Mit KI beschleunigen", "Accélérer avec l'IA", …). The docs (Traces Explorer, Code) name the button as each language's Dashboard now draws it, in those 15 languages.

Tests added or changed.

• TracePerformanceSuggestion.test.tsx: where the next step sits for each offer (and only the fix button spans a phone), the query's two lines, the working line's status role, live dot and link.
• TracePerformanceSuggestionInLocale.test.tsx: every language words the strip's button, tooltip, links and states itself, and every language's button label fits a phone's button on one line (the six old labels fail it).
• TraceExplorer.test.tsx: three refusal re-reads (someone else's fix → its link; repository gone → Connect GitHub; the re-read fails → the reason stays) and a "Show them" block replacing the one test (wide: opens the first and scrolls; narrow: no sheet; one span; lets go of errors only and service chips).
• TelemetryImprovementCard.test.tsx: a refusal while the same improvement started meanwhile links it; the refused-start test checks the re-read.
• ExceptionAIAssistance.test.tsx: AI off for an owner (Turn AI on → AI Features), a member (who can) and before permissions arrive.
• Offline browser suites: TraceDetail +6 specs and a ?repo=gone fixture scenario (repository gone after load; the fix beside the finding on a wide screen; two tablet layouts; phone alignment of the link and the working line; phone "Show them" with the toolbar), about 15 s more in Dashboard Offline UI group 3. ExceptionDetail +1 spec and ?enableAi=off (no AI Assistance in the menu, the notice with Turn AI on), about 5 s more in group 2. No regrouping needed.

Left for later (not UI polish): once a performance fix has finished, the trace offers the fix again (the server's de-duplication looks at runs in flight only); linking the finished fix's pull request instead is a server change. Persian pages render left to right across the Dashboard, the strip included (pre-existing). Durations read "1.72 s" in every language (the trace page's formatter, pre-existing).

What ran for the review (after merging master at 603b51d): Common's guard suites (150 suites, 4,751 tests) and the suites that read what the review changed, the locale suites included (42 suites, 1,645 tests); App's root-level tests (17 suites), its locale and wiring suites (11 suites); the docs suites that read the two changed pages (13 suites, 9,126 tests), every docs translations suite (47 suites, 73,006 tests, as a Dashboard label the docs write changed in 15 locales) and the general docs suites (29 suites); the changed suites again with ee/ moved aside; docs:check-anchors, docs:localize-anchors (nothing to move), i18n:validate, i18n:check; the offline browser suites TraceDetail (48 passed) and ExceptionDetail (32 passed); compile-tests (Common), the App, Dashboard and E2E compiles, the --listFiles package check, prettier --check and eslint on every changed file, and the ReDoS check (no new pattern). Not run: the full Common and App suites, and the other browser suites.

🤖 Generated with Claude Code

[simlarsen]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.