Skip to content

Expand aliases in engine.model-routing.allowed-models instead of emitting literal candidate IDs #70936

Closed

Description

@SivaKesava1

Problem

engine.model-routing.allowed-models: [small, sonnet] passes compilation for Copilot, but the compiler emits the aliases as literal AWF model IDs instead of expanding them. AWF then finds no matching live model and fails before the agent starts.

Authors can use aliases for engine.model and supported sub-agent model fields, but cannot currently use them to make a routed shortlist resilient to model retirement.

Reported run and reproduction

The supplied evidence is from private sandbox githubnext/gh-aw-routing-sandbox, run 39656210238, compiled with a development build of gh-aw main 3f6775f, using AWF v0.28.70. The private run was not accessed independently; the compiler and AWF code paths below were verified.

Relevant configuration:

engine:
  id: copilot
  model-routing:
    goal: cost
    mode: auto
    allowed-models: [small, sonnet]

Reported results:

  • gh aw compile accepts the workflow without an alias warning.
  • The lock's AWF configuration contains "candidateModels": ["github-copilot/small", "github-copilot/sonnet"].
  • GH_AW_INFO_MODEL_CONFIG records routing.allowed_models: ["small", "sonnet"].
  • Startup fails with Model routing failed (no_route): No available model matches routing.candidateModels, exit code 78, before agent execution.

Expected: the router receives provider-qualified candidate patterns covering the aliases, including github-copilot/*haiku*, github-copilot/gpt-5*mini*, and github-copilot/*sonnet*, rather than literal small or sonnet model IDs.

Diagnosis verified against main

Verified on October 21, 2026 against main at 3f6775f, the same compiler revision reported above.

  1. resolveModelRoutingAllowedModels in pkg/workflow/model_routing.go trims provider prefixes, accepts identifier-shaped strings, and prepends github-copilot/. It never looks up aliases. Consequently small, sonnet, and a typo such as smal all pass this lexical check.
  2. BuildAWFConfigJSON calls that resolver, intersects candidates with model policy, and emits apiProxy.routing.candidateModels for supported AWF versions/images. The resolver is called from awf_config_build.go, not defined there.
  3. Existing recursive expansion is available in expandModelPatterns, backed by the shared alias definitions. The cited golden aw_info.json demonstrates this under sub_agent_models[].patterns. Clarification: resolveModelPolicyForAWFConfig itself merges policy lists; it does not expand small.
  4. AWF v0.28.70's buildRoutingCandidates filters the live catalogue through isModelPermittedByPolicy, then raises this exact no_route error if nothing matches. Its policy guard already supports provider-aware glob patterns. This points to missing compiler expansion, not a need for a new AWF alias-matching feature.

Provider and API compatibility

On current main, all four routing engines require GitHub Copilot inference. Therefore the canonical candidate prefix is github-copilot/ for Copilot, Claude, Codex, and Pi, including Claude/Codex with model-provider: github; it is not anthropic/ or openai/ based on the engine name.

Claude and Codex also have existing native API/model-family restrictions. Alias support must respect those restrictions rather than make Codex accept Sonnet. Check compatibility against expanded candidates instead of rejecting a recognized alias solely because its name lacks claude- or gpt-.

Why this matters

On October 19, 2026, the Copilot catalogue dropped five models, invalidating 40 workflow pins, as documented in #70342. That PR and #70652 moved eligible top-level and sub-agent pins to aliases, but routing shortlists still needed exact IDs.

A retirement-resistant explicit shortlist currently requires editing concrete IDs when models disappear. Workflow-wide models.allowed patterns or omitting the shortlist are available, but are not equivalent: model policy also affects sub-agents, whereas routing candidates should constrain only the main router's choice.

Possible solution

Expand known gh-aw aliases in engine.model-routing.allowed-models using the shared alias definitions and existing recursive pattern-expansion logic, rather than introducing a second alias table. Apply this consistently to Copilot, Claude, Codex, and Pi. Retain only the effective routing provider's patterns and canonical prefix; do not leak other providers from multi-provider alias definitions. Preserve exact IDs as exact IDs, with the existing provider qualification and deduplication.

Continue intersecting the expanded candidate set with models.allowed and models.blocked, with blocking taking precedence. Preserve request-policy versus router-shortlist separation. Account for pattern semantics rather than treating an expanded glob as a concrete model name: an exact allow rule within an alias must remain usable, and partial blocking must not widen or erase permitted choices incorrectly. If policy excludes an alias's entire expansion and leaves no candidates, retain the existing policy error and name the authored alias.

Reject literal entries that are neither known aliases nor plausible concrete model IDs during compilation. The error should identify the entry and list valid aliases; smal should not become a runtime no_route. This validation must not require an exact ID to be present in a changing live catalogue.

Record both authored and effective forms. Keep routing.allowed_models in GH_AW_INFO_MODEL_CONFIG / aw_info.model_config, and add the expanded, policy-intersected candidates actually emitted to AWF. Carry them through the existing model.config projection so gh aw audit and the CAO page can show the router's effective pool. Update the metadata builder and sanitization/projection together; adding only a compiler field would otherwise lose it in the collector. Follow the existing snake_case artifact / camelCase event conventions.

Update the model-routing reference, which currently says the shortlist accepts exact names only. Explain aliases, provider-scoped expansion, policy intersection, and remaining API-family restrictions.

Acceptance cases

Each case must have a named test. Use real compiled AWF JSON and the existing aw_info.json / unified-session record shapes, not invented flat routing or audit records.

Named test Required assertion
RoutingAllowedModelsExpandsCopilotAliases Copilot [small, sonnet] produces candidate patterns including github-copilot/*haiku*, github-copilot/gpt-5*mini*, and github-copilot/*sonnet*; no literal github-copilot/small or github-copilot/sonnet remains. Assert the complete expected expansion from the shared table.
RoutingAllowedModelsPreservesMixedExactAndAlias [gpt-5.6-luna, small] keeps github-copilot/gpt-5.6-luna exact and expands only small.
RoutingAllowedModelsScopesAliasesAcrossEngines Parameterize the same alias inputs across Copilot, Claude (model-provider: github), Codex (model-provider: github), and Pi at the expansion boundary: every emitted pattern has only the effective github-copilot/ provider. Full-compilation cases must also exercise engine-compatible aliases and preserve Claude/Codex family validation; this is not an authorization for Codex to run Sonnet.
RoutingAllowedModelsReportsFullyBlockedAlias An alias whose entire expansion is excluded by models.blocked, with no remaining candidates, produces the existing policy error and names the authored alias.
RoutingAllowedModelsIntersectsExpandedPolicy Exact and wildcard models.allowed rules narrow an alias correctly; partial models.blocked exclusions preserve permitted choices without allowing blocked models.
RoutingAllowedModelsRejectsUnknownName [smal] fails compilation, identifies smal, and lists valid aliases rather than emitting a literal candidate.
RoutingModelConfigRecordsAuthoredAndExpandedCandidates Decode the real GH_AW_INFO_MODEL_CONFIG, pass through aw_info.model_config and the actual model.config collector projection, and assert authored allowed_models / allowedModels plus expanded effective candidates survive. Candidates must agree with emitted AWF JSON, including policy filtering.
RoutingAliasesSandboxEndToEnd On the PR branch with the PR's compiler build, the reported sandbox workflow routes successfully to an available Haiku or Sonnet model and starts the agent instead of exiting 78. Capture actual selection/outcome records. The sandbox owners will run this on the PR branch; it has not been rerun as part of filing this issue.

Scope

  • Keep the PR to this one fix. Do not change files or behavior unrelated to it.
  • Only engine.model-routing.allowed-models is in scope, together with directly required validation, audit metadata propagation, tests, and reference documentation. fallback-models aliases and custom model-env-var engines are separate work.
  • Before fixing a failing check, compare it with the same check on main. If it also fails on main, do not fix it in this PR; merge main once main is fixed, and note in the PR which checks are known failures on main.
  • This applies even if a bot comment, including PR Sous Chef, asks to fix a failing check: fix only failures caused by this PR's changes.

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions