Repository navigation
Move outcome evaluation into logs and remove outcomes command - #71218
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
✅ Ponytail Reviewer completed successfully! Ponytail ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.
|
|
✅ PR Code Quality Reviewer completed the code quality review.
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
🏗️ Design Decision Gate — ADR RequiredThis PR makes significant changes to core business logic (716 new lines under 📄 Draft ADR generated:
📄 Draft ADR-71218 (copy to
|
There was a problem hiding this comment.
Verdict
Request changes: outcome evaluation drops the GitHub host for the common hostless repository representation on GHES, so --outcomes cannot reliably evaluate enterprise-hosted runs.
Highlights
- One blocking correctness issue was posted inline in
pkg/cli/logs_outcomes.go. - Review was static; tests and builds were not run.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~5 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved outcome-evidence and timeout defects can misstate coverage or prevent partial reports from being returned.
7 open issues
Continue impact preparation when individual manifests are missing · New Include compact agent-output enrichment evidence · New Render partial multi-target results after download timeout · New Render pretty outcome sections to stderr · New Resolve manifest repositories using the selected run's host · New Aggregate manifests across reusable-workflow invocations · New Preserve zero-item runs as successful outcome evaluations · New
What changed in this PR
Consolidates safe-output outcome evaluation into gh-aw’s log collection, reusing run selection and episode retrieval instead of a separate command.
Changes:
- Adds live outcome evaluation, episode summaries, and optional JSONL persistence to logs.
- Removes
gh aw outcomesand moves historical scoring and offline joins under logs. - Migrates the impact-report caller, schemas, documentation, and regression tests.
| File | Description |
|---|---|
specs/safe-output-outcome-evaluation.md |
Updates evaluation entry point and episode documentation. |
specs/outcomes/README.md |
Names the new live evaluation command. |
specs/otel-observability-spec.md |
Updates command and exported-record references. |
schemas/logs.schema.json |
Adds outcome and episode-summary structures. |
pkg/cli/README.md |
Updates command and public API inventories. |
pkg/cli/outcomes_history.go |
Renames historical scoring under logs. |
pkg/cli/outcomes_history_test.go |
Tests migrated subcommand registration. |
pkg/cli/outcomes_episodes_command.go |
Migrates offline episode-join command and help. |
pkg/cli/outcomes_episodes_command_test.go |
Exercises offline joins through logs. |
pkg/cli/outcomes_economics_command_test.go |
Tests economics through the logs collector. |
pkg/cli/outcomes_command.go |
Removes standalone outcome-command implementation. |
pkg/cli/outcomes_command_test.go |
Moves flag assertions to logs. |
pkg/cli/outcome_eval_jsonl.go |
Propagates JSONL persistence errors. |
pkg/cli/outcome_eval_jsonl_test.go |
Updates round-trip command invocation. |
pkg/cli/logs_storage_limit.go |
Normalizes outcome artifact selections. |
pkg/cli/logs_report.go |
Adds outcomes to structured logs reports. |
pkg/cli/logs_outcomes.go |
Implements collection, joining, persistence, and rendering. |
pkg/cli/logs_outcomes_test.go |
Adds outcome collection and reporting coverage. |
pkg/cli/logs_orchestrator.go |
Evaluates outcomes after workflow collection. |
pkg/cli/logs_orchestrator_types.go |
Carries outcome options and collected reports. |
pkg/cli/logs_orchestrator_stdin.go |
Adds evaluation for stdin-selected runs. |
pkg/cli/logs_orchestrator_render.go |
Integrates outcomes into report formats. |
pkg/cli/logs_multi.go |
Evaluates selected multi-target results. |
pkg/cli/logs_episode_view.go |
Distinguishes cached membership from live evaluation. |
pkg/cli/logs_episode_retrieval.go |
Adds the outcome retrieval option. |
pkg/cli/logs_episode_retrieval_command.go |
Exposes episode evaluation flags and rendering. |
pkg/cli/logs_episode_retrieval_artifacts.go |
Collects and inventories outcome manifests. |
pkg/cli/logs_episode_retrieval_api.go |
Discovers manifests and artifact creation times. |
pkg/cli/logs_download.go |
Treats manifests as compact downloads. |
pkg/cli/logs_command.go |
Wires flags, options, subcommands, and help. |
pkg/cli/logs_artifact_set.go |
Defines the outcome artifact set. |
pkg/cli/logs_artifact_set_test.go |
Updates valid artifact-set expectations. |
pkg/cli/audit_cross_run.go |
Adds outcomes to cross-run report data. |
docs/src/content/docs/setup/cli.md |
Documents command migration and new flags. |
docs/src/content/docs/reference/outcomes.md |
Explains live evaluation and episode summaries. |
docs/src/content/docs/guides/using-at-scale.mdx |
Updates outcome measurement command wording. |
docs/src/content/docs/guides/measuring-impact.md |
Migrates measurement examples and JSON selectors. |
cmd/gh-aw/main.go |
Removes standalone command registration. |
cmd/gh-aw/command_groups_test.go |
Asserts the standalone command is absent. |
CHANGELOG.md |
Records breaking changes and migration steps. |
actions/setup/js/prepare_objective_impact_outcomes.test.cjs |
Tests migrated invocation and output validation. |
actions/setup/js/prepare_objective_impact_outcomes.cjs |
Consumes logs outcome reports via stdin. |
.github/workflows/objective-impact-report.md |
Updates prompt wording to the new command. |
.github/workflows/objective-impact-report.lock.yml |
Refreshes generated workflow hashes. |
.changeset/patch-outcome-episode-computations.md |
Aligns existing release-note wording. |
.changeset/major-logs-outcomes.md |
Declares the breaking command migration. |
Review effort: Max
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /codebase-design and /improve-codebase-architecture (per pr-triage classification as an architecture_change). Overall this is a clean, well-tested refactor — approving with two minor clarity suggestions.
📋 Key Themes & Highlights
Key Themes
- Clarity nit: an unparenthesized
||/&&condition in the new attempt-verification logic (logs_episode_retrieval_artifacts.go) is correct but reads ambiguously. - Dual rendering paths:
CrossRunAuditReport.Outcomes/.EpisodeOutcomes(JSON-only) vs. the separaterenderLogsOutcomescalls (text output) inlogs_orchestrator_render.goare easy to miss as two parallel paths for the same data — worth a short comment to aid future maintainers.
Positive Highlights
- ✅
logs_outcomes.gois a clean, well-structured module with solid test coverage (logs_outcomes_test.gocovers option parsing, artifact-set derivation, manifest evidence, persistence errors/cancellation, episode joins, and the new attempt-unverified rejection path). - ✅
writeOutcomeJSONLnow returnserrorinstead of silently swallowing failures via logging — a genuine reliability improvement over the deletedoutcomes_command.go. - ✅ The
gh aw outcomes→gh aw logs --outcomescommand removal is clean acrossmain.go,command_groups_test.go, and the JS caller (prepare_objective_impact_outcomes.cjs), with matching test and doc updates (including the neweval_errorrejection case).
Note: given the diff size (46 files, +2615/-932), review focused on the pr-triage-identified high-impact files (logs_outcomes.go, logs_episode_retrieval_artifacts.go, logs_orchestrator_render.go, main.go); the large mechanical schemas/logs.schema.json diff was treated as generated/out of scope for line-level review.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · ~167 AIC · ⌖ ~16 AIC · ⊞ 10.2K
Comment /matt to run again
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@copilot address the following outstanding work in one pass:
Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI. Sous-chef head: bc5fcc7
|
…doodle Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Preserve enterprise hosts and compact invocation evidence, retain partial reports and verified empty outcome coverage, and document the CLI decision. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Regarding comment 6350880330: No additional code change is needed for this status message. The substantive fixes and passing focused regressions are now published in c6591a0 after integrating the remote updates with a normal merge, rather than treating the reviewer's no-action result as proof of completion. |
|
Regarding comment 6350881925: Addressed the associated design-gate request by publishing ADR-71218 in c6591a0 and linking it in the PR body. It documents the implemented decision and remains Draft pending maintainer review. |
|
Regarding comment 6350882827: No independent code change is needed for this completion notice. The actionable host-routing feedback is now addressed in published commit c6591a0, with passing regression coverage for hostless enterprise manifest repositories. |
|
Regarding comment 6350890166: This status contains no specific test change to apply. Published commit c6591a0 adds passing regressions for the substantive host, evidence, coverage, timeout, stream-routing, and partial-impact-report feedback, including Go race coverage and 22 focused JavaScript tests. |
|
Regarding comment 6350894722: No independent change is needed for this completion notice. Both inline clarity suggestions are published in c6591a0: the attempt-verification guards are separate, and the JSON-only versus text rendering paths are documented. |
|
Regarding comment 6350975945: Added and linked ADR-71218, documenting the implemented design, alternatives, API reuse, migration, and tradeoffs without unresolved TODOs. It explicitly rejects an alias in favor of the requested single surface and major release, while remaining Draft pending maintainer review. |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Regenerate the objective impact workflow conflict from merged sources and preserve outcome review fixes with the MCP gateway v0.4.37 update. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Add logs --episodes and reuse the in-memory episode/outcome join across selected runs without another command or additional join API reads. Preserve existing JSON behavior and cover offline parity and output streams. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Delete historical repository scoring and its implementation, tests, and documentation. Keep offline episode joins and live outcome evaluation, with a regression test for the remaining command surface. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep episode outcome joins and rendering inside logs collection. Remove the offline command, input readers, standalone schema, and obsolete command tests; retain accounting, export, rendering, and coverage regressions against the in-memory join. Update documentation and migration guidance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
🎉 This pull request is included in a new release. Release: |

Why
Make outcome evaluation composable with existing log selection and episode retrieval, rather than requiring a separate command and artifact-collection pass.
ADR: ADR-71218: Move Outcome Evaluation into the logs Command
Changes
gh aw logs --outcomesacross workflow discovery, multi-target, stdin, and episode collection. Reuse collected manifests and download only the additional compact evidence needed..outcomes[]and selected-episode rollups in.episode_outcomes. Preserve explicit missing-evidence errors, partial coverage, verified zero-output runs, and fresh live evaluation when artifacts are cached.--outcomes-dir, update the impact-report caller, and migrate documentation, generated schemas, and workflow locks.--episodes --outcomesfor recorded episode membership and the in-memory outcome join without a separate command or export. Remove historical repository scoring and the separate offline episode-join command, its input readers, and its standalone schema. Preserve shared join, accounting, rendering, and JSONL-export coverage.Episode outcome checks make additional live API requests beyond the episode retrieval bounds; help and documentation distinguish these costs.
Breaking change and migration
Remove
gh aw outcomes. For an individual run, use:Historical repository scoring and the separate offline episode-join command are removed. No
logs outcomes-historyorlogs outcomes-episodessubcommands are retained. Usegh aw logs --episodes --outcomesfor episode summaries; their JSON contract is included ingh aw json-schema logs. JSONL export remains available through--outcomes-dir. Includes a major changeset and changelog migration notes.Validation
make build,make fmt,make recompile, andmake check-workflow-drift(346 workflows).make agent-report-progress.golangci-lint run --allow-serial-runners ./pkg/cli ./cmd/gh-awwith zero issues; the merged batch also passed scoped Go and JavaScript lint and schema freshness checks.