Skip to content

SUPPORTED_COPILOT_MODELS allowlist in src/copilot-model.ts is out of sync with docs/model-api-mapping.json — rejects valid models as "retired" #5752

Description

@jaroslawgajewski

Summary

AWF's host-side model validation (src/copilot-model.ts, validateCopilotModel()) rejects a currently-valid Copilot model with a misleading error:

Error: model 'claude-sonnet-5' is retired or unsupported. Did you mean 'claude-sonnet-4.5'?

This is thrown before any container starts (host-side CLI validation), so the workflow fails immediately regardless of whether the model is actually supported by the Copilot Business API.

Root cause

SUPPORTED_COPILOT_MODELS in src/copilot-model.ts is a hardcoded Set with no update automation. It does not include claude-sonnet-5.

This is a separate list from docs/model-api-mapping.json, which:

  • Has a dedicated automated daily updater workflow (model-api-mapping-updater)
  • Already lists claude-sonnet-5 as a confirmed, valid model family (see PR chore: update model-to-API mapping (2026-07-01) #5742, merged/pending as of 2026-07-01): "New generation; best combination of speed and intelligence, 1M context window. Confirmed in official Anthropic models docs."

The two lists have drifted out of sync because only one of them is kept current automatically.

Additionally, the error message is misleading: claude-sonnet-5 is not present in RETIRED_COPILOT_MODEL_ALIASES (the actual retired/renamed-model mapping — currently only contains gpt-5-codex). The "Did you mean 'claude-sonnet-4.5'?" suggestion comes from a Levenshtein-distance fuzzy match against the allowlist (suggestionFor()), not a real deprecation mapping. So a model that is simply missing from the list is reported as "retired", which sends operators down the wrong troubleshooting path (looking for a deprecation notice that doesn't exist).

Reproduction

  1. Configure a workflow's Copilot engine with COPILOT_MODEL=claude-sonnet-5 (or any other valid-but-unlisted model).
  2. Run the workflow through AWF.
  3. Observe the host-side rejection before any container starts, with the misleading "retired" framing.

Confirmed on AWF v0.27.13 and v0.27.16 (upgrading did not change the outcome) — both predate a fix.

Suggested fix

  • Add claude-sonnet-5 (and any other newly-confirmed families already in docs/model-api-mapping.json, e.g. claude-mythos-5) to SUPPORTED_COPILOT_MODELS.
  • Consider having one list drive the other (or unify them) so a new model release only needs a single update path instead of two independently-maintained sources that can silently drift.
  • Consider softening the error message so "not in our allowlist yet" isn't phrased as "retired" when the model isn't in the actual retired-aliases map.

Version info

  • gh-aw-firewall: v0.27.13 and v0.27.16 (both reproduce)
  • gh-aw compiler: v0.82.0 and v0.82.1
  • Copilot CLI: 1.0.65

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions