Skip to content

Expand aliases in engine.model-routing.allowed-models - #70942

Merged
SivaKesava1 merged 10 commits into
mainfrom
copilot/expand-aliases-in-allowed-models
Oct 22, 2026
Merged

SivaKesava1 merged 10 commits into
mainfrom
copilot/expand-aliases-in-allowed-models

Conversation

Copilot AI commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

engine.model-routing.allowed-models: [small, sonnet] compiled without error but was emitted as github-copilot/small and github-copilot/sonnet. Those names match no live model, so AWF exited 78 with no_route before the agent started. This change expands gh-aw aliases at compile time, using the existing shared alias table and expandModelPatterns, so a shortlist can survive model retirements.

engine:
  id: copilot
  model-routing:
    allowed-models: [small, sonnet]
# before: "candidateModels": ["github-copilot/small", "github-copilot/sonnet"]
# after:  "candidateModels": ["github-copilot/*haiku*", "github-copilot/gpt-5*mini*",
#          "github-copilot/gpt-5*nano*", "github-copilot/gemini-*flash*lite*", "github-copilot/*sonnet*"]

Candidate resolution (pkg/workflow/model_routing_candidates.go, new)

  • Each entry is resolved in this order: expression, then known alias, then plausible exact ID, otherwise a compile error. The error names the entry and lists the valid aliases, so smal fails at compile time instead of at run time. Exact IDs are not checked against the live model catalogue.
  • Aliases expand through the new walkModelPatterns helper, which now backs expandModelPatterns. Only copilot/ patterns are kept, rewritten to github-copilot/. This applies to Copilot, Claude, Codex and Pi, since all four routing engines use Copilot inference.
  • Claude/Codex model families still apply. Patterns outside the engine's family are dropped. A leading-* pattern is narrowed to the family prefix only when the alias's own direct entries include that engine's native provider (*sonnet* becomes claude-*sonnet*). An alias left with no compatible patterns is an error, so Codex with [sonnet] fails and cites the Responses API.
  • Exact IDs keep the existing qualification, deduplication and family checks.
  • Behaviour change: names that look like model IDs but are defined as aliases (gpt-5, gpt-5.4, gpt-5.6, gpt-6) now expand, matching how engine.model resolves them. No existing workflow uses them in a shortlist.

Policy intersection with models.allowed / models.blocked

  • Each candidate is checked against glob coverage and overlap rather than treated as a concrete model name:
    • A blocked rule that covers the pattern drops it; blocking always wins.
    • With no allow list, or when an allowed rule covers the pattern, it is kept.
    • Otherwise the pattern is narrowed to the allowed rules it covers, so an exact allowed model inside an alias stays usable.
    • A partial overlap keeps the whole pattern, and AWF filters it at run time.
  • On AWF before v0.28.33 (no candidateModels; candidates replace allowedModels), partial overlaps are dropped so models.allowed is never widened.
  • If the policy removes every candidate, the existing policy error is raised and lists the excluded authored names, for example excluded: small.
  • One helper, modelRoutingAWFCandidates, now feeds both BuildAWFConfigJSON and the run metadata, so the two cannot disagree.

Metadata and audit

  • GH_AW_INFO_MODEL_CONFIG and aw_info.model_config keep routing.allowed_models as authored and gain routing.candidate_models, the exact list emitted to AWF. It is omitted when any candidate is an expression.
  • routing_analytics.cjs keeps the new field when sanitizing and projects it to model.config as routing.candidateModels. The d.ts types and the regenerated session schema are updated, and the unified session spec is bumped to 2.2.0.
  • Eight routed smoke lock files change by one line each (the new candidate_models field): smoke-claude-routed, smoke-copilot-routed, smoke-copilot-routed-claude, smoke-copilot-routed-steering, smoke-copilot-sdk-routed, smoke-copilot-sdk-routed-claude, smoke-pi-routed and smoke-pi-routed-claude.

Docs

  • The model-routing reference now explains aliases in shortlists, provider-scoped expansion, policy narrowing and the Claude/Codex family limits, with new troubleshooting rows. The allowed-models description in the main workflow schema and frontmatter-full.md now mentions aliases.

Tests

  • pkg/workflow/model_routing_aliases_test.go adds the issue's named tests. They compile real workflows and read the AWF JSON from the lock file. RoutingModelConfigRecordsAuthoredAndExpandedCandidates runs the compiled GH_AW_INFO_MODEL_CONFIG through parseCompiledModelConfig and projectModelConfig in node, and checks the result against the emitted AWF candidates.
  • Expectations in routing_analytics.test.cjs are updated because they read the recompiled lock files.
  • RoutingAliasesSandboxEndToEnd is left for the sandbox owners to run on this branch.

Known failures on main (not addressed)

  • golangci-lint modernize embedlit in claude_engine.go, codex_engine.go and comment_memory.go.
  • TestAgyProductionConformancePermissionsAreBoundedAndScoped, TestAgyConformanceEntryPointsShareConfiguration, TestDryRunCompiledCacheMode and TestEngineConformanceCatalogCoverage.

Copilot AI and others added 3 commits October 21, 2026 20:33
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix alias expansion in model routing Expand aliases in engine.model-routing.allowed-models Oct 21, 2026
Copilot AI requested a review from SivaKesava1 October 21, 2026 21:09
@SivaKesava1
SivaKesava1 marked this pull request as ready for review October 21, 2026 21:37
Copilot AI max review requested due to automatic review settings October 21, 2026 21:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved candidate-resolution and alias-traversal regressions can change model selection or prevent workflows from starting.

5 open issues
What changed in this PR

Adds alias-aware routing shortlists to gh-aw’s workflow compiler and records the emitted candidate pool for auditing.

Changes:

  • Expands routing aliases with provider, engine-family, and model-policy constraints.
  • Carries authored and expanded candidates through run metadata and unified-session schemas.
  • Adds regression coverage, updates documentation, and refreshes routed smoke lock files.
File Description
pkg/​workflow/​model_routing.go Refactors alias traversal and routing validation.
pkg/​workflow/​model_routing_subagents_test.go Updates expression-policy test inputs.
pkg/​workflow/​model_routing_candidates.go Adds candidate expansion and policy intersection.
pkg/​workflow/​model_routing_aliases_test.go Adds compilation and metadata regression coverage.
pkg/​workflow/​awf_config_build.go Uses shared candidate resolution for AWF configuration.
pkg/​workflow/​aw_info_model_config.go Records authored and emitted routing candidates.
pkg/​workflow/​aw_info_model_config_test.go Checks expanded-candidate metadata.
pkg/​parser/​schemas/​main_workflow_schema.json Documents alias support in shortlist descriptions.
docs/​src/​content/​docs/​specs/​unified-agent-session-specification.md Adds effective candidates to the session contract.
docs/​src/​content/​docs/​reference/​model-routing.md Explains alias shortlists and policy interactions.
docs/​src/​content/​docs/​reference/​frontmatter-full.md Updates shortlist field comments.
docs/​public/​schemas/​unified-session.schema.json Allows effective candidates in routing metadata.
actions/​setup/​js/​types/​unified_session.d.ts Types effective routing candidates.
actions/​setup/​js/​routing_analytics.test.cjs Updates routing metadata expectations.
actions/​setup/​js/​routing_analytics.cjs Sanitizes and projects effective candidates.
.github/​workflows/​smoke-pi-routed.lock.yml Refreshes Pi routing metadata.
.github/​workflows/​smoke-pi-routed-claude.lock.yml Refreshes Pi/Claude routing metadata.
.github/​workflows/​smoke-copilot-sdk-routed.lock.yml Refreshes Copilot SDK routing metadata.
.github/​workflows/​smoke-copilot-sdk-routed-claude.lock.yml Refreshes SDK/Claude routing metadata.
.github/​workflows/​smoke-copilot-routed.lock.yml Refreshes Copilot routing metadata.
.github/​workflows/​smoke-copilot-routed-steering.lock.yml Refreshes steering-workflow routing metadata.
.github/​workflows/​smoke-copilot-routed-claude.lock.yml Refreshes Copilot/Claude routing metadata.
.github/​workflows/​smoke-claude-routed.lock.yml Refreshes Claude routing metadata.

Review effort: Max

Comment thread pkg/workflow/model_routing_candidates.go
Comment thread pkg/workflow/model_routing.go Outdated
Comment thread pkg/workflow/model_routing_candidates.go Outdated
Comment thread pkg/workflow/model_routing_candidates.go
Comment thread pkg/workflow/model_routing_candidates.go
@SivaKesava1

Copy link
Copy Markdown
Collaborator

Review findings (round 2):

Round 2 for this PR (#70942, issue #70936). Review findings to address:

  1. Exact ids must stay exact. In resolveModelRoutingCandidateGroups (pkg/workflow/model_routing_candidates.go), the alias check runs before the exact-id check, and the github-copilot//copilot/ prefix is stripped before the alias lookup. Some alias names in model_aliases.json are also real model ids (gemini-3.8-flash, gpt-5-mini, gpt-5-codex, gpt-5, gpt-5.5, gpt-6), so allowed-models: [gemini-3.8-flash] becomes the pattern github-copilot/gemini-3.8*flash*, which also matches -lite and -image variants. Writing github-copilot/gemini-3.8-flash gets expanded too, so an author can't pin the exact id. Fix:
    • An entry with a provider prefix (github-copilot/…, copilot/…) is always exact, never alias-expanded.
    • For an unprefixed name that is both an alias key and a model id, follow the same rule engine.model uses for that string, and say in the PR and the model-routing docs which wins and how to force the other (the prefix).
    • Add named tests: RoutingAllowedModelsPrefixedIdStaysExact (github-copilot/gemini-3.8-flash → exact) and RoutingAllowedModelsAliasNamedLikeIdFollowsEngineModelRule.
  2. No invented patterns. In expandModelRoutingAlias, the branch that prefixes a leading-wildcard Copilot pattern with the engine's family prefix produces patterns that match no model: for Codex, grok (copilot/*grok*, openai/*grok*) becomes github-copilot/gpt-*grok*, and computer-use becomes gpt-*computer-use*. They compile and then fail at runtime with no_route. Apply the family rewrite only when the alias's native-provider entry has the same pattern (as in anthropic/*sonnet* next to copilot/*sonnet*); otherwise drop the pattern so the existing "no compatible models" compile error fires. Add a named test RoutingAllowedModelsNoInventedFamilyPatterns covering grok and computer-use for Codex, and sonnet for Claude still working.
  3. Note in the PR (no code change): for Codex, [small] expands to gpt-5*mini*/gpt-5*nano*, which no live Copilot model matches since 19 Oct, so it compiles and fails at runtime with no_route. That's the mini alias gap, a separate decision for the maintainers; it's also why gh-aw#70945 moves two workflows from small to gpt-5.
  4. Don't change anything else.

Keep the PR to this one fix. Before fixing a failing check, compare it with the same check on main; if it also fails on main, leave it and note it in the PR. Run make agent-finish; regenerated-file drift from main is fine to include. Reply on the PR with what changed in each item.

Copilot AI and others added 2 commits October 21, 2026 22:37
…licy like AWF

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot AI commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Review findings (round 2):

Round 2 for this PR (#70942, issue #70936). Review findings to address:
...

Round 2 is done in ba345dd (pushed by a parallel session while I was working) and 2944b8c.

  1. Exact ids stay exact. A github-copilot/ or copilot/ prefix always means an exact id. For an unprefixed name that's also an alias, the compiler applies the engine.model rule: the model id wins. At compile time it checks the name against gh-aw's bundled Copilot catalogue (models.json) in place of the live one.

    • gemini-3.8-flash, gpt-5-mini and gpt-5.4 stay exact.
    • gpt-5, gpt-5-codex and gpt-6 aren't Copilot ids, so they expand as aliases.
    • The prefix forces the exact id, e.g. github-copilot/gpt-5.
    • To get the alias's model family when the id wins, list a broader alias such as gemini-flash, or define a differently named alias.

    This is documented in model-routing.md. Tests: TestRoutingAllowedModelsPrefixedIdStaysExact and TestRoutingAllowedModelsAliasNamedLikeIdFollowsEngineModelRule. The second runs the JS resolveConfiguredModel and checks that the compiler makes the same exact-or-alias choice.

  2. No invented patterns. Checking that the native-provider entry has the same pattern isn't enough here: grok and computer-use also list openai/*grok* and openai/*computer-use*. So the family rewrite is now kept only when the narrowed pattern matches a Copilot model in the bundled catalogue. With Codex, grok and computer-use now fail to compile with "no models compatible"; with Claude, sonnet still gives claude-*sonnet*. Test: TestRoutingAllowedModelsNoInventedFamilyPatterns.

  3. [small] with Codex: noted in the PR description. It expands to gpt-5*mini* and gpt-5*nano*, which compile but fail at runtime with no_route. That's the separate mini alias gap, and the reason gh-aw#70945 moves two workflows to gpt-5.

  4. make agent-finish:

    • make recompile leaves no drift. I didn't include the unrelated workflow-source rewrites from make fix.
    • Golint modernize and TestWorkCommandPolicyFromConfigClosedFlags fail on main too.
    • The JS test, SEC-005, gosec and govulncheck failures are in files this PR doesn't touch (govulncheck had no network).

@SivaKesava1

Copy link
Copy Markdown
Collaborator

Review findings:

Round 4 for this PR (#70942). Two gaps from our verification of rounds 2 and 3:

  1. Default candidates use the old provider matcher. The github//copilot//github-copilot/ provider aliases and case-insensitive matching were applied where allowed-models is set (copilotQualifiedModelPolicyRule), but not when it is omitted: modelPolicyRuleCoversFamily and modelPolicyRuleMayMatchFamily (pkg/workflow/model_routing.go) still use modelPolicyRuleProviderMatchesCopilot, which is case-sensitive and accepts only github-copilot. So Claude or Codex with no allowed-models and models.blocked: ["copilot/claude-*"], ["github/claude-*"] or ["Claude-*"] compiles, then AWF blocks every candidate at runtime, while the same rule with allowed-models set fails to compile. The PR's docs now promise firewall-style matching. Use the same lower-cased provider-alias matcher in both paths, remove modelPolicyRuleProviderMatchesCopilot, and add regression tests for the default-candidates path (the three examples above).
  2. The prefixed-id test doesn't isolate the prefix rule. TestRoutingAllowedModelsPrefixedIdStaysExact only uses gemini-3.8-flash, which is a bundled id and stays exact anyway. Add cases whose bare name is an alias but not a bundled id: github-copilot/gpt-5 → github-copilot/gpt-5 and copilot/gpt-5.6 → github-copilot/gpt-5.6, not alias globs.

Don't change anything else. Keep the PR to this one fix. Before fixing a failing check, compare it with the same check on main; if it also fails on main, leave it and note it in the PR. Regenerated-file drift from main is fine. Reply on the PR with what changed.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot AI commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Review findings:

Round 4 for this PR (#70942). Two gaps from our verification of rounds 2 and 3:...

Round 4 is in dc1d9b6.

  1. Default candidates now use the same provider matcher. When allowed-models is omitted, modelPolicyRuleCoversFamily and modelPolicyRuleMayMatchFamily now check rules with the same matcher as the allowed-models path. That matcher accepts github/, copilot/ and github-copilot/ and ignores case. I removed modelPolicyRuleProviderMatchesCopilot. I also switched the exact models.allowed entry check in the same function from matchesModelPolicy to that matcher, so it has the same provider and case gap fixed.
    • New regression tests: Claude with models.blocked: ["copilot/claude-*"], ["github/claude-*"] or ["Claude-*"] now fails to compile. All three tests fail without the fix.
  2. The prefixed-id test now isolates the prefix rule. TestRoutingAllowedModelsPrefixedIdStaysExact now checks that github-copilot/gpt-5 becomes github-copilot/gpt-5 and copilot/gpt-5.6 becomes github-copilot/gpt-5.6. Each case first asserts that the bare name is an alias but not a bundled model id.

The full pkg/workflow test suite passes.

On checks: the local pre-push gate fails because golangci-lint reports modernize/embedlit in claude_engine.go, codex_engine.go and comment_memory.go. Those lines are the same on main and this PR doesn't touch them, so I left them. The lint-go-golangci CI jobs pass on this PR. The CodeQL scan timed out, so it didn't produce a result for this change.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: dc1d9b6
Sous-chef work:
Sous-chef state: adcfa6c02f801f6566f71b058b2401de5bde3c293820c7d7b164b8fbd33031f7

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 7.32 AIC · ⌖ 13.9 AIC · ⊞ 1.1K · ◷
Comment /souschef to run again

… spec changes to 2.3.0

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
…s-in-allowed-models

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Branch is up to date with main as of 4b82c21.

No review threads were open, so there was nothing to reply to or resolve. What's left is CI and human review.

@SivaKesava1

Copy link
Copy Markdown
Collaborator

Verified at 4b82c21 (fixes gh-aw#70936):

  • End to end with this PR's build (private sandbox githubnext/gh-aw-routing-sandbox, run 39723404462, the setup that failed before with no_route, exit 78). It's a Copilot routed workflow with allowed-models: [small, sonnet].
    • The compiled AWF config has candidateModels: ["github-copilot/*haiku*","github-copilot/gpt-5*mini*","github-copilot/gpt-5*nano*","github-copilot/gemini-*flash*lite*","github-copilot/*sonnet*"].
    • AWF routed to claude-sonnet-5.5:low with no routing failure, and all 4 requests completed with 200. The run succeeded.
    • model.config records both the authored allowedModels: ["small","sonnet"] and the effective candidateModels.
  • Review rounds 2–5 checked:
    • Prefixed entries are always exact, with tests using names that aren't bundled ids (github-copilot/gpt-5, copilot/gpt-5.6).
    • Bundled Copilot ids stay exact before alias expansion.
    • The empty-key default alias works again.
    • Family constraints are intersected rather than prefixed, so no invented gpt-*grok* patterns.
    • One case-insensitive matcher accepts github/, copilot/ and github-copilot/ on both the allowed-models and default-candidates paths, with tests.
    • After merging main, Codex: stage native TOML sub-agents with resolved models and fail closed before startup #70945's unsupported_endpoint stays under spec 2.2.0, and this PR's routing.candidateModels is 2.3.0.
  • Lock files: only the 8 routed smoke locks differ from main, each by adding candidate_models to GH_AW_INFO_MODEL_CONFIG; their routing is unchanged.
  • CI: CGO (impacted Go tests, golangci, govulncheck), CJS (4 shards), typecheck and lint all pass.
  • Known follow-up, not in scope: for Codex, small expands to gpt-5*mini*/gpt-5*nano*, which no live Copilot model matches since 19 Oct. That's the mini alias-table gap.

@SivaKesava1
SivaKesava1 merged commit 273e7e1 into main Oct 22, 2026
49 checks passed
@SivaKesava1
SivaKesava1 deleted the copilot/expand-aliases-in-allowed-models branch October 22, 2026 04:45
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.93.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand aliases in engine.model-routing.allowed-models instead of emitting literal candidate IDs

4 participants