Skip to content

Restore safe-output handler and tool-description conformance - #71609

Merged
pelikhan merged 12 commits into
mainfrom
copilot/aw-top-10-fix-safe-output-conformance
Oct 24, 2026
Merged

pelikhan merged 12 commits into
mainfrom
copilot/aw-top-10-fix-safe-output-conformance

Conversation

Copilot AI commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Safe-output handlers and tool descriptions had drifted from the conformance specification: required add_comment constraints were undisclosed, several handlers lacked standardized error codes, and two staged-mode guards gave no preview notice.

  • Conformance checks: Updated the comment-limit signature check to match the implementation and added regression assertions for the four affected checks.
  • Tool guidance: Added an IMPORTANT CONSTRAINTS section to both add_comment tool definitions.
  • Handler diagnostics: Applied standard error-code prefixes across the eight flagged handlers.
  • Staged previews: Added visible 🎭 preview notices to the PR-lifecycle and Claim-resource guards.
throw new Error(`${ERR_VALIDATION}: ...`);
logStagedPreviewInfo("Claim resource effects are not performed");

Copilot AI and others added 2 commits October 24, 2026 10:33
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix safe output conformance drift in handlers and docs Restore safe-output handler and tool-description conformance Oct 24, 2026
Copilot AI requested a review from pelikhan October 24, 2026 10:49
@pelikhan
pelikhan marked this pull request as ready for review October 24, 2026 10:54
Copilot AI max review requested due to automatic review settings October 24, 2026 10:54
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts on this branch.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new grader dependency is not staged, causing gh aw graders run to fail before evaluation.

4 open issues
What changed in this PR

This PR addresses #71584 by restoring safe-output conformance checks, tool guidance, and diagnostic consistency across gh-aw’s runtime helpers.

Changes:

  • Aligns the comment-limit signature check and adds assertions for four conformance rules.
  • Labels existing add_comment constraints prominently in both tool definitions.
  • Adds standardized error prefixes, staged-preview notices, and related tests.
File Description
scripts/​check-safe-outputs-conformance.sh Matches the current comment-limit helper signature.
scripts/​check-safe-outputs-conformance_test.sh Adds assertions for four conformance rules.
pkg/​workflow/​js/​safe_outputs_tools.json Emphasizes existing comment constraints.
actions/​setup/​js/​work_queue_delivery.test.cjs Tests the verifier configuration error prefix.
actions/​setup/​js/​work_queue_delivery.cjs Standardizes delivery validation errors.
actions/​setup/​js/​work_queue_claim_scope.test.cjs Tests staged-preview logging and error codes.
actions/​setup/​js/​work_queue_claim_scope.cjs Prefixes scope errors and logs staged guards.
actions/​setup/​js/​start_mcp_gateway.test.cjs Updates gateway diagnostic expectations.
actions/​setup/​js/​start_mcp_gateway.cjs Adds configuration and system error prefixes.
actions/​setup/​js/​safe_outputs_tools.json Mirrors the comment-description update.
actions/​setup/​js/​safe_outputs_tools_loader.test.cjs Tests tool-cloning error codes.
actions/​setup/​js/​safe_outputs_tools_loader.cjs Prefixes cloning and file-write failures.
actions/​setup/​js/​reconcile_work_queue_run.test.cjs Updates the finalizer error expectation.
actions/​setup/​js/​reconcile_work_queue_run.cjs Prefixes queue reconciliation errors.
actions/​setup/​js/​pr_lifecycle.test.cjs Tests staged-preview notices.
actions/​setup/​js/​pr_lifecycle.cjs Logs skipped lifecycle checks in staged mode.
actions/​setup/​js/​model_pin_check.test.cjs Tests missing-configuration error codes.
actions/​setup/​js/​model_pin_check.cjs Prefixes argument and configuration errors.
actions/​setup/​js/​grader_session.cjs Prefixes invalid-session errors.
actions/​setup/​js/​custom_graders.test.cjs Tests invalid-session error codes.
actions/​setup/​js/​check_version_updates.test.cjs Checks the execution-provenance error prefix.
actions/​setup/​js/​check_version_updates.cjs Prefixes compatibility and provenance failures.

Review effort: Max

Comment thread actions/setup/js/grader_session.cjs Outdated
// @ts-check

const { calculateWorkingSetFromEntries } = require("./working_set_metrics.cjs");
const { ERR_VALIDATION } = require("./error_codes.cjs");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added error_codes.cjs to stageGraderRuntime; the staged grader run test passes. Fix: 441b8a8.

Comment thread actions/setup/js/start_mcp_gateway.cjs Outdated
if (resourceExecution.getStore()?.verification) throw scopeError("Claim resource verification is read-only");
if (process.env.GH_AW_SAFE_OUTPUTS_STAGED === "true" || execution?.staged === true) throw scopeError("read-only Claim preview cannot perform resource effects");
if (process.env.GH_AW_SAFE_OUTPUTS_STAGED === "true" || execution?.staged === true) {
logStagedPreviewInfo("Claim resource effects are not performed");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guarded the preview notice on core.info availability and added a no-logger regression test; the coded Claim scope error remains intact. Fix: 441b8a8.

Comment on lines +68 to +70
if grep -E "\\[(HIGH|MEDIUM|LOW)\\] $finding:" "$OUTPUT" >/dev/null; then
echo "FAIL: Expected no $finding safe-output conformance findings"
grep -E "\\[(HIGH|MEDIUM|LOW)\\] $finding:" "$OUTPUT"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The conformance test now strips ANSI escape sequences before matching findings, and the regression test passes. Fix: 441b8a8.

…-safe-output-conformance

# Conflicts:
#	scripts/check-safe-outputs-conformance.sh

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts on this branch.

Merged main and resolved the conformance-check conflict. Merge commit: 71ed330. The safe-output conformance checks and final impacted tests pass.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (actions/setup/js/grader_session.cjs:4): The staged runtime for gh aw graders run omits error_codes.cjs, so its unconditional trace_graders.cjs import now fails with MODULE_NOT_FOUND before evaluating a grader. Add error_codes.cjs to the file list in stageGraderRuntime (pkg/cli/graders_run.go:253) alongside this new dependency. - Restore safe-output handler and tool-description conformance #71609 (comment)
  3. Review (actions/setup/js/start_mcp_gateway.cjs:1549): Errors from codedError already carry a prefix, so this wrapper reports configuration failures as ERR_SYSTEM: ERR_CONFIG: ... and duplicates system prefixes. Pass message directly to setFailed, which already preserves existing codes and supplies the system fallback. Same issue at actions/setup/js/start_mcp_gateway.cjs:1375: pass getErrorMessage(err) directly there. - Restore safe-output handler and tool-description conformance #71609 (comment)
  4. Review (actions/setup/js/work_queue_claim_scope.cjs:195): When this guard runs without a global core logger, logStagedPreviewInfo throws ReferenceError before the intended scope error. This breaks staged Claim callers that rely on the read-only Claim preview message and claim_scope_invalid code. Make the notice conditional on core.info being available so the guard still throws its coded authorization error. - Restore safe-output handler and tool-description conformance #71609 (comment)
  5. Review (scripts/check-safe-outputs-conformance_test.sh:70): The checker emits an ANSI reset sequence between each severity label and the following space, so this pattern never matches the captured findings. As a result, all four new regression assertions pass even when those checks report failures. Strip ANSI escapes first, or allow them between the closing bracket and the finding ID in both grep expressions. - Restore safe-output handler and tool-description conformance #71609 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 71ed330
Sous-chef work: 09a01309d0781a770d0e6bb59aef76fed35b368a2cdf3163b14e9195f96d3070 27c1aab456d06eb9daa60b25d25f7e04a2c18303f894d81c09a19d24d3b771a1 d39a14b9f81d5cfbe4e03922799cee26866d6a87cbc8a82d68efbe2335f33a84 fe00f9f0f328cf52f79fb6e1d3d66eb97d548ea567e6bc8fea0ed90b9255cfb1
Sous-chef state: da58956dbd0fdbc40158cd756743717f8ca694a06a51262afa4faec5e555f324

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · ~9 AIC · ⌖ ~7 AIC · ⊞ 1.1K · ◷
Comment /souschef to run again

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts on this branch.

Copilot AI and others added 2 commits October 24, 2026 15:07
…-safe-output-conformance

# Conflicts:
#	actions/setup/js/pr_lifecycle.cjs
#	actions/setup/js/pr_lifecycle.test.cjs
#	actions/setup/js/safe_outputs_tools.json
#	actions/setup/js/work_queue_claim_scope.cjs
#	actions/setup/js/work_queue_claim_scope.test.cjs
#	pkg/workflow/js/safe_outputs_tools.json

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts on this branch.

Merged the latest main and resolved the conflicts while preserving the PR changes. Merge commit: aa69d8f; follow-up model-pin baseline fix: 2a99f44.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 2a99f44
Sous-chef work:
Sous-chef state: eb77b2afcf5c48f98b134481006ec5953e5ab1ce0078ca108ad07e603011bd58

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · ~5 AIC · ⌖ ~10 AIC · ⊞ 1.1K · ◷
Comment /souschef to run again

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts in this pull request

Copilot AI and others added 2 commits October 24, 2026 16:11
…-safe-output-conformance

# Conflicts:
#	actions/setup/js/check_version_updates.cjs
#	actions/setup/js/grader_session.cjs
#	actions/setup/js/model_pin_check.cjs
#	actions/setup/js/reconcile_work_queue_run.cjs
#	actions/setup/js/safe_outputs_tools_loader.cjs
#	actions/setup/js/start_mcp_gateway.cjs
#	actions/setup/js/work_queue_claim_scope.cjs
#	actions/setup/js/work_queue_delivery.cjs

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Updated from latest main with make merge-main and resolved conflicts in merge commit f795518; the referenced fixes are preserved. Local validation passed.

@copilot resolve the merge conflicts in this pull request

Resolved the latest main merge conflicts in merge commit f795518; the final local validation gate passed.

Comment thread actions/setup/js/start_mcp_gateway.cjs Outdated
core.error("ERROR: MCP_GATEWAY_DOCKER_COMMAND has incorrect syntax");
core.error("Expected: docker run command with image and arguments");
core.setFailed(`ERROR: MCP_GATEWAY_DOCKER_COMMAND has incorrect syntax — got: ${dockerCommand}`);
setFailed(`${ERR_CONFIG}: MCP_GATEWAY_DOCKER_COMMAND has incorrect syntax — got: ${dockerCommand}`);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the supplied Docker command from the syntax error and kept the expected format in the diagnostic. Fix: f386a2d.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (actions/setup/js/start_mcp_gateway.cjs:933): Sensitive information disclosure risk: actions/setup/js/start_mcp_gateway.cjs logs the raw MCP_GATEWAY_DOCKER_COMMAND value in a failure ("... incorrect syntax — got: ${dockerCommand}"). If this command string includes credentials (e.g., registry tokens, -e SECRET=..., or embedded auth headers), they will be exposed in Actions logs. Redact/omit the command contents (or apply redactCliProxyDiagnostics-style redaction) before calling setFailed. - Restore safe-output handler and tool-description conformance #71609 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: a10832a
Sous-chef work: 7afba6bcf1182545a7dfd4594a4d80c036b154d6b931eada9aeefbd334f0c31e
Sous-chef state: a648c4fe7775816645f19967daeb0372a189c1dd062cfe2fcee2515d8987fcea

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · ~6 AIC · ⌖ ~11 AIC · ⊞ 1.1K · ◷
Comment /souschef to run again

Copilot AI and others added 2 commits October 24, 2026 17:42
…-safe-output-conformance

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts on this branch.

…-safe-output-conformance

# Conflicts:
#	pkg/cli/graders_run.go

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts on this branch.

Merged latest main via make merge-main and resolved the grader runtime conflict, preserving error_codes.cjs. Pushed merge commit e5055b3. Local validation, conformance regressions, grader tests, and workflow compilation passed. CI was not triggered; CodeQL timed out.

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Updated from latest main using make merge-main and resolved conflicts in e5055b3. The Docker-command logging fix in f386a2d is preserved. Local validation passed; no CI was triggered. Previously answered review threads still need resolution because the available authenticated tools do not expose that operation.

@pelikhan
pelikhan merged commit ec50a19 into main Oct 24, 2026
48 checks passed
@pelikhan
pelikhan deleted the copilot/aw-top-10-fix-safe-output-conformance branch October 24, 2026 20:05
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.93.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[AW Top 10] 04 Fix safe output conformance drift in handlers and docs

5 participants