Skip to content

fix: bound daily AIC quota recovery and snapshot discovery - #70968

Merged
pelikhan merged 4 commits into
mainfrom
pelikhan-literate-telegram
Oct 22, 2026
Merged

pelikhan merged 4 commits into
mainfrom
pelikhan-literate-telegram

Conversation

@pelikhan

Copy link
Copy Markdown
Collaborator

Summary

  • Reduce optional daily-AIC snapshot discovery to two recent completed producers within the 48-hour observation window and at most two repository-history pages. A miss still requires a complete authoritative accounting scan.
  • On original run attempt 1 only, allow one bounded quota-reset deferral per read-only accounting phase. Honor the later applicable server reset/Retry-After deadline plus one second; restart the entire phase rather than retrying a partial scan.
  • Preserve fresh full-window history, exact run-attempt cache validation, the 100-request reserve, existing credentials, and fail-closed startup. Permission/authentication failures, missing deadlines, waits beyond the ten-minute elapsed-plus-wait budget, late timer wakes, and renewed exhaustion remain errors.
  • Document the contract and add regressions for first-read exhaustion, fresh-window restart, changed attempts, renewed exhaustion, deadlines, and bounded cache misses.

Evidence and scope

Six inspected scheduled dispatcher failures on October 15, 16, and 18–21 stopped on the first snapshot-restore request with core quota remaining 0/15000. The October 17 successful baseline spent 11 requests discovering no usable snapshot, followed by only three authoritative accounting requests. This change reduces that amplification and provides bounded same-activation recovery; it does not attribute the shared quota consumers or guarantee API availability.

No schedule, compiler, generated-lock, workflow configuration, credential, queue, or deployment changes. No dispatches or reruns. Existing noon schedule mitigation remains unchanged.

Validation

  • Four focused Vitest suites: 306 tests passed.
  • make fmt-cjs and make lint-cjs passed.
  • make agent-report-progress passed using installed modern Bash (macOS system Bash lacks mapfile): build, strict TypeScript check, JavaScript lint, schema freshness, impacted tests. No changed Go files; impacted Go tests correctly skipped. Existing lint warnings remain; no newly introduced warning.
  • Read-only security gate passed with no confirmed vulnerabilities.

Residual live confirmation

Observe the next naturally scheduled run after deployment. Recovery is limited to ten minutes per phase, with one minute of timer scheduling tolerance; longer or unknown reset deadlines and renewed exhaustion deliberately still block startup. The activation job has no compiled timeout override; the agent-step timeout is unchanged. No stale-cache or partial-total authorization is introduced.

Restart complete accounting phases once after a bounded server quota reset while retaining attempt-1 identity and fail-closed authorization. Limit optional snapshot discovery without weakening authoritative window accounting.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan marked this pull request as ready for review October 21, 2026 23:42
Copilot AI max review requested due to automatic review settings October 21, 2026 23:42
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · sonnet50 · 14.2 AIC · ⌖ 15.1 AIC · ◇ 6.61 AIC · ⊞ 12.7K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · gpt56 · 7.61 AIC · ⌖ 5.32 AIC · ⊞ 14.8K · ◷

Generated by Ponytail Reviewer for #70968

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · opus55 · 10.2 AIC · ⌖ 36.5 AIC · ◇ 16.5 AIC · ⊞ 1.9K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

No actionable blocking issues found in the changed lines. The quota recovery, bounded snapshot discovery, and cache-attempt validation changes are internally consistent; the advisory findings about budget reuse and snapshot candidate breadth do not establish defects from the inspected implementation.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · 2.51 AIC · ⌖ 0.55 AIC · ⊞ 1.6K
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs and /codebase-design (pr-triage was not invoked directly; I applied the same bug-fix/hardening classification heuristic manually against the pre-fetched diff — noting this per the fallback rules). One minor cleanup suggestion posted inline; otherwise this is solid, well-evidenced, well-tested work.

📋 Key Themes & Highlights

Key Themes

  • Dead code left behind by refactor: MAX_RESTORE_RUNS/MAX_RESTORE_PAGES/SCAN_WINDOW_MS in constants.cjs are now unused in production (only restore_aic_scan_cache.cjs consumed them, and it now defines its own local bounds) but remain exported and asserted in constants.test.cjs. Flagged inline.

Verified Correctness

  • Traced withQuotaResetRecovery (daily_aic_api_budget.cjs) against getGitHubAPIDiagnostic/getGitHubAPIQuota: the one-shot deferral correctly excludes authentication_failed/permission_denied, requires GITHUB_RUN_ATTEMPT === "1", bounds the wait to maxWaitMs (10 min) + 60s timer tolerance, and deliberately does not re-catch a second failure ("renewed exhaustion cannot defer again") — this matches the PR's stated contract exactly.
  • Confirmed the recovery option wiring end-to-end: check_daily_aic_workflow_guardrail.cjs → scanDailyAIC → withQuotaResetRecovery, and separately restore_aic_scan_cache.cjs mainWithPaths → withQuotaResetRecovery(() => restoreWithBudget(...)).
  • Verified the bounded snapshot-discovery change (MAX_SNAPSHOT_PRODUCERS=2, MAX_SNAPSHOT_HISTORY_PAGES=2, 48h CACHE_RETENTION_MS window) is exercised by tests asserting the exact per_page: 2 call and the page-by-page trusted-candidate counting/early-break logic in restoreWithBudget.

Positive Highlights (/diagnosing-bugs, /tdd)

  • ✅ Root-caused against real production evidence (6 failed scheduled runs + 1 successful-but-wasteful baseline cited in the PR body), not just patching a symptom.
  • ✅ Strong regression coverage: descriptive spec-like test names ("restarts the complete phase only after the later server deadline", "does not defer HTTP 401/403 access failures even when quota headers are exhausted") covering unknown/invalid/expired/long resets, rerun vs. original attempt, early/late timer wake, and renewed exhaustion.
  • ✅ Spec doc (ai-credits-specification.md) updated in lockstep with the code, keeping the bounded-discovery and deferral contract documented.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 198.6 AIC · ⌖ 15.1 AIC · ⊞ 10.2K
Comment /matt to run again

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Timing-sensitive quota recovery in the startup guardrail has a confirmed deadline-selection defect and needs maintainer validation.

2 open issues
What changed in this PR

This PR updates gh-aw’s daily AI Credits startup guardrail to bound optional snapshot discovery and introduce limited recovery from GitHub API quota exhaustion.

Changes:

  • Limits snapshot discovery by retention window, producer count, and history pages.
  • Adds bounded, first-attempt quota-reset recovery that restarts the complete accounting phase.
  • Documents the recovery contract and adds regression coverage.
File Description
docs/​src/​content/​docs/​specs/​ai-credits-specification.md Defines snapshot bounds and quota-recovery requirements.
actions/​setup/​js/​restore_aic_scan_cache.cjs Bounds snapshot discovery and adds restore-phase recovery.
actions/​setup/​js/​daily_aic_scan.test.cjs Covers phase restarts, refreshed history, changed attempts, and bounded misses.
actions/​setup/​js/​daily_aic_scan.cjs Wraps authoritative accounting in complete-phase recovery.
actions/​setup/​js/​daily_aic_api_budget.test.cjs Tests recovery deadlines, elapsed-time bounds, and fail-closed conditions.
actions/​setup/​js/​daily_aic_api_budget.cjs Implements a single bounded quota-reset deferral.
actions/​setup/​js/​check_daily_aic_workflow_guardrail.cjs Passes recovery options into authoritative accounting.

Review effort: Max

Comment thread actions/setup/js/daily_aic_api_budget.cjs Outdated
Comment thread actions/setup/js/restore_aic_scan_cache.cjs Outdated
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (actions/setup/js/daily_aic_api_budget.cjs:75): Passing true makes the primary quota reset binding even for secondary limits with healthy primary quota. With 4,000 requests remaining, a 429 carrying Retry-After: 60 and a primary reset one hour away rejects an otherwise eligible 61-second deferral as reset_exceeds_budget. Conversely, a secondary-limit 403 without Retry-After restarts after 11 seconds when the unrelated primary reset is 10 seconds away, despite having no known secondary-limit deadline. Preserve whether the budget stopped at its configured reserve separately from secondary restrictions. Include the primary reset only f... - fix: bound daily AIC quota recovery and snapshot discovery #70968 (comment)
  3. Review (actions/setup/js/restore_aic_scan_cache.cjs:49): per_page limits unfiltered runs, not trusted snapshot producers. If the two newest completed runs are pull_request runs, this request returns no usable producer even when the third run has a trusted snapshot. On a fresh activation, that avoidable miss makes authoritative accounting resolve historical runs individually instead of reusing recorded observations, adding per-run API reads. A larger page still costs one history request, and restoreWithBudget already caps artifact inspections at two trusted producers. Keep those bounds separate: for example, list 10 runs but inspect at most two... - fix: bound daily AIC quota recovery and snapshot discovery #70968 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 98ed4ff
Sous-chef work: 40bb79b9a014ee6ca9b8d7452263e552911ff6fd46167a18825accdfa0e54ce3 6fbb816e18acf552c8f4f05950bbf3de092963eb5b0672326979f1980a85febf
Sous-chef state: fffde721f4eafa0ad65f49d0961a00086d9bef945441ec6593d3345d0cc65197

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 10.3 AIC · ⌖ 20.1 AIC · ⊞ 1.1K · ◷
Comment /souschef to run again

Track configured reserve stops independently of secondary restrictions, and list ten completed runs while limiting trusted snapshot inspections to two. Add deadline and untrusted-run regression coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to #70968 (comment): No code change was requested by this completed skills review. The two actionable inline findings were fixed separately with regression coverage in 75d423d.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to #70968 (comment): No code change is needed for this reported_no_action notice, and I am not treating it as proof of validation. The combined corrections passed 363 focused tests and strict TypeScript checking locally.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to #70968 (comment): This reported_no_action notice contains no actionable finding, so no additional code change is needed. The quota and snapshot corrections are covered by explicit regression tests rather than relying on the no-op outcome.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to #70968 (comment): The completed code-quality review requests no specific change. No additional change is needed beyond the two separately addressed inline findings.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to #70968 (comment): This reported_no_action design-gate notice requests no ADR or design change. The narrow corrections retain the existing fail-closed accounting design, and the specification now distinguishes secondary deadlines and listed runs from trusted producer inspections.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

In response to #70968 (comment): Accepted: updated from main using make merge-main, fixed both inline findings in 75d423d, pushed the batch, and replied to and resolved both threads. Local validation passed (363 focused tests, strict TypeScript, lint, final progress gate, and 346/346 workflow recompilations); no CI dispatch or rerun was requested, and CI for the new head plus human review remain unverified.

…egram

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI requested a review from gh-aw-bot October 22, 2026 02:05
@pelikhan
pelikhan merged commit 2fdca20 into main Oct 22, 2026
2 checks passed
@pelikhan
pelikhan deleted the pelikhan-literate-telegram branch October 22, 2026 02:06
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.93.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants