fix: prevent duplicate "Download container images" step in codex evals job by Copilot · Pull Request #45991 · github/gh-aw

Copilot · 2026-07-16T12:54:25Z

engine: codex workflows with evals: triggered a compiler validation error — duplicate step 'Download container images' found in job 'evals' — because the step was emitted twice: once by buildPullAWFContainersStep (unconditional in buildEvalsJobSteps) and again by generateMCPSetup inside buildEvalsEngineSteps (codex always needs AWF gateway config, even with no MCP tools).

The detection job had the same issue and was already patched in threat_detection_steps.go.

Changes

pkg/workflow/evals_steps.go — skip buildPullAWFContainersStep for codex; generateMCPSetup already emits the download step for that engine:

// For codex, buildEvalsEngineSteps → generateMCPSetup already emits the
// Docker download step, so skip here to avoid a duplicate step name.
if c.getEvalsEngineID(data) != "codex" {
    steps = append(steps, c.buildPullAWFContainersStep(data)...)
}

pkg/workflow/evals_steps_test.go — two new regression tests:
- TestBuildEvalsJobStepsCodexNoDuplicateContainerDownload — exactly one download step for codex
- TestBuildEvalsJobStepsNonCodexIncludesContainerDownload — copilot/claude still get the step
changeset.md, necromancer.md, duplicate-code-detector.md — add evals: sections, which were held back pending this fix; recompile all 257 workflows (0 errors)

…n evals job Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

In buildEvalsJobSteps, buildPullAWFContainersStep was called unconditionally. For engine: codex, buildEvalsEngineSteps → generateMCPSetup also emits the Docker download step, producing a duplicate that trips duplicate-step validation. Fix: skip buildPullAWFContainersStep for codex (same pattern as the detection job fix in threat_detection_steps.go). Add regression tests for both the codex case (no duplicate) and non-codex engines (step still present). Also add evals sections to changeset.md, necromancer.md, and duplicate-code-detector.md, which were excluded pending this fix, and recompile all affected workflows. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-07-16T13:18:13Z

✅ PR Code Quality Reviewer completed the code quality review.

github-actions · 2026-07-16T13:18:16Z

✅ Test Quality Sentinel completed test quality analysis.

github-actions · 2026-07-16T13:18:30Z

✅ Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR #45991 does not have the 'implementation' label and has only 70 new lines of code in business logic directories (threshold: 100).

github-actions · 2026-07-16T13:19:26Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

Copilot

Pull request overview

This PR fixes a gh-aw compiler validation failure for engine: codex workflows with evals: by preventing the evals job from emitting two steps with the same name (Download container images). It also adds eval question blocks to several workflows and includes the resulting regenerated workflow outputs.

Changes:

Avoid generating a duplicate Download container images step in the evals job for the Codex engine by skipping the extra pre-pull step.
Add regression tests to ensure Codex produces exactly one download step and non-Codex engines still include the download step.
Add evals: sections to multiple workflow markdown sources and recompile the corresponding generated workflow YAMLs.

Show a summary per file

File	Description
pkg/workflow/evals_steps.go	Skip `buildPullAWFContainersStep` for Codex to prevent duplicate “Download container images” in the evals job.
pkg/workflow/evals_steps_test.go	Add regression tests covering Codex/no-duplicate and non-Codex/still-present behavior.
.github/workflows/changeset.md	Add `evals:` questions to the workflow source.
.github/workflows/necromancer.md	Add `evals:` questions to the workflow source.
.github/workflows/duplicate-code-detector.md	Add `evals:` questions to the workflow source.
.github/workflows/changeset.lock.yml	Regenerated compiled workflow reflecting `evals` support and related jobs/steps.
.github/workflows/necromancer.lock.yml	Regenerated compiled workflow reflecting `evals` support and related jobs/steps.
.github/workflows/duplicate-code-detector.lock.yml	Regenerated compiled workflow reflecting `evals` support and related jobs/steps.
.github/workflows/skillet.lock.yml	Regenerated compiled workflow output after recompile.
.github/workflows/release.lock.yml	Regenerated compiled workflow output after recompile.
.github/workflows/hourly-ci-cleaner.lock.yml	Regenerated compiled workflow output after recompile.
.github/workflows/cli-consistency-checker.lock.yml	Regenerated compiled workflow output after recompile (includes strict-policy related output changes).
.github/workflows/avenger.lock.yml	Regenerated compiled workflow output after recompile.
.github/workflows/agentic-auto-upgrade.yml	Regenerated auto-upgrade workflow; schedule cron changed in this diff.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 14/14 changed files
Comments generated: 2
Review effort level: Low

Copilot · 2026-07-16T14:07:22Z

+	// For codex, buildEvalsEngineSteps calls generateMCPSetup which already emits
+	// the Docker download step via generateDownloadDockerImagesStep, so skip here
+	// to avoid a duplicate "Download container images" step name.
+	if c.getEvalsEngineID(data) != "codex" {


Fixed in 7ff45da — replaced "codex" with string(constants.CodexEngine).

Copilot · 2026-07-16T14:07:30Z

 on:
  schedule:
-    - cron: "11 4 * * 6"  # Weekly (auto-upgrade)
+    - cron: "21 3 * * 5"  # Weekly (auto-upgrade)
  workflow_dispatch:


Reverted in 9fced06 — the cron change was an inadvertent side-effect of running make recompile (the scatter is supposed to be deterministic but apparently drifted). The file is back to "11 4 * * 6" matching the pre-PR state.

github-actions

The fix is correct and well-reasoned. The root cause (unconditional buildPullAWFContainersStep in buildEvalsJobSteps colliding with the codex MCP setup path that already emits the same step) is clearly identified, and the guard matches the existing pattern used elsewhere (engine.GetID() == codex). Tests cover both the regression (exactly 1 step for codex) and the normal path (step still present for non-codex engines). LGTM.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · 21.6 AIC · ⌖ 4.29 AIC · ⊞ 5K

github-actions · 2026-07-16T13:26:01Z

Hey @copilot-swe-agent 👋 — great catch on the duplicate Download container images step in the codex evals job! The fix is clean, the regression tests cover both the codex and non-codex paths, and the recompile of all 257 workflows (0 errors) gives good confidence the change is safe. This looks ready for review.

Generated by ✅ Contribution Check · 100.5 AIC · ⌖ 8.71 AIC · ⊞ 6.2K · ◷

github-actions · 2026-07-16T13:26:18Z

🧪 Test Quality Sentinel Report

✅ Test Quality Score: 95/100 — Excellent

Analyzed 6 test(s): 6 design, 0 implementation, 0 violation(s).

📊 Metrics (6 tests)

Metric	Value
Analyzed	6 (Go: 6, JS: 0)
✅ Design	6 (100%)
⚠️ Implementation	0 (0%)
Edge/error coverage	5 (83%)
Duplicate clusters	0
Inflation	Yes (12.8:1, justified)
🚨 Violations	0

Test	File	Classification	Coverage
TestBuildEvalsEngineStepsArcDindTopology	evals_steps_test.go:18	Design	Arc-dind topology, staging step, harness path (2 subtests)
TestBuildEvalsJobStepsRenderSummary	evals_steps_test.go:89	Design	Step presence, ordering (redact → render → upload), outcome gating
TestBuildEvalsJobStepsRedactionUsesEvalsSecretReferences	evals_steps_test.go:148	Design	Secret extraction, environment variable bindings
TestBuildParseEvalsResultsStepUsesResolvedExecutionModel	evals_steps_test.go:173	Design	Model resolution (resolved vs. default), github.run_id propagation
TestBuildEvalsJobStepsCodexNoDuplicateContainerDownload	evals_steps_test.go:206	Regression	Direct fix validation: exactly 1 "Download container images" step for codex
TestBuildEvalsJobStepsNonCodexIncludesContainerDownload	evals_steps_test.go:236	Design	Non-codex engines (copilot, claude) receive container download step (table-driven)

Code Quality Highlights

✅ 0% implementation tests (threshold: 30%) — all tests verify behavioral contracts and design invariants, not implementation details.

✅ No violations — proper build tags, no mock libraries (gomock/testify/mock), all assertions have descriptive failure messages.

✅ High-value regression test — TestBuildEvalsJobStepsCodexNoDuplicateContainerDownload directly validates the PR fix (deduplication logic for codex engines).

✅ Edge case coverage — multiple engine types (arc-dind vs. standard, codex vs. non-codex), step ordering constraints, secret resolution.

⚠️ Test inflation (12.8:1 ratio) — 64 test lines vs. 5 net production lines. Justified: tests add 6 comprehensive behavioral contracts, not just the 1-line fix. Appropriate for critical workflow compilation code.

Verdict

✅ Passed. 0% implementation tests (0% < 30% threshold). No coding violations. Tests provide robust coverage of behavioral contracts and regression prevention for the fix.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🧪 Test quality analysis by Test Quality Sentinel · 18.2 AIC · ⌖ 9.64 AIC · ⊞ 6.8K · ◷
_{Comment /review to run again}

github-actions

✅ Test Quality Sentinel: 95/100. 0% implementation tests (threshold: 30%). Tests validate critical behavioral contracts and provide robust regression coverage for the fix.

github-actions

Skills-Based Review 🧠

Applied /diagnosing-bugs — the fix is correct, well-targeted, and properly regression-tested.

📋 Key Themes & Highlights

Positive Highlights

✅ Root cause correctly identified: buildEvalsJobSteps unconditionally emits buildPullAWFContainersStep, while buildEvalsEngineSteps → generateMCPSetup also emits it for codex.
✅ Fix mirrors the established pattern in threat_detection_steps.go — good consistency.
✅ Two regression tests: one asserts exactly-one download step for codex, one asserts non-codex engines still get the step.
✅ Guard c.getEvalsEngineID(data) != "codex" is tightly scoped to the only engine that calls generateMCPSetup (confirmed at evals_steps.go:218).
✅ PR description accurately explains root cause; no documentation gap.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · 30.1 AIC · ⌖ 4.35 AIC · ⊞ 6.7K
_{Comment /matt to run again}

github-actions

Review: non-blocking observations, consider before next similar patch

The immediate bug fix (duplicate step for codex evals) is correct and the regression tests are a good addition. Three non-blocking issues worth addressing before this pattern spreads to more engines:

Findings summary

Medium — fragile negative allowlist (evals_steps.go:47): The guard != "codex" is structurally decoupled from the real condition that causes the duplicate (whether generateMCPSetup is called). A new engine routed through generateMCPSetup would silently re-introduce the same bug. See inline comment.

Medium — asymmetric guard vs threat_detection (evals_steps.go:47): threat_detection_steps.go:41 includes an || usingExternalDetector term; this guard does not. Either document why that case doesn't apply to evals, or mirror the shape. See inline comment.

Low — weak assertion in non-codex test (evals_steps_test.go:1417): strings.Contains won't catch a double-emission. Use strings.Count(...) == 1 to match the codex test. See inline comment.

🔎 Code quality review by PR Code Quality Reviewer · 66.8 AIC · ⌖ 4.71 AIC · ⊞ 5.6K
_{Comment /review to run again}

Comments that could not be inline-anchored

pkg/workflow/evals_steps.go:47

Fragile negative allowlist — any new engine routed through generateMCPSetup will silently re-introduce the duplicate step.

<details>
<summary>💡 Details & suggested fix</summary>

The guard reads:

if c.getEvalsEngineID(data) != &quot;codex&quot; {
    steps = append(steps, c.buildPullAWFContainersStep(data)...)
}

This works today because only codex reaches generateMCPSetup inside buildEvalsEngineSteps (line 218). But the check at line 47 is structurally decoupled from that decision …

pkg/workflow/evals_steps.go:47

Guard in evals_steps.go is asymmetric with the equivalent guard in threat_detection_steps.go — the external-detector case is silently unhandled.

<details>
<summary>💡 Details</summary>

threat_detection_steps.go:41 uses:

if c.getThreatDetectionEngineID(data) != &quot;codex&quot; || usingExternalDetector {
    steps = append(steps, c.buildPullAWFContainersStep(data)...)
}

The evals guard added here uses only != "codex" with no usingExternalDetector equivalent. If an analogous fea…

pkg/workflow/evals_steps_test.go:1417

Non-codex test uses strings.Contains — it won't catch a double-emission for non-codex engines.

<details>
<summary>💡 Suggested fix</summary>

strings.Contains passes even if the step appears twice. Use the same count-based assertion as the codex test:

count := strings.Count(allSteps, &quot;name: Download container images&quot;)
if count != 1 {
    t.Errorf(&quot;expected exactly one &#39;Download container images&#39; step for %s evals job, got %d&quot;, engine, count)
}

This makes the non-codex test …

pelikhan · 2026-07-16T13:42:16Z

@copilot run pr-finisher skill

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

… in evals_steps.go Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-07-16T19:58:06Z

🎉 This pull request is included in a new release.

Release: v0.82.11

Copilot AI and others added 2 commits July 16, 2026 12:28

wip: start compiler bug fix for duplicate Download container images i…

079aaa5

…n evals job Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Jul 16, 2026

Copilot AI changed the title ~~fix: prevent duplicate Download container images step in codex evals job~~ fix: prevent duplicate "Download container images" step in codex evals job Jul 16, 2026

Copilot created this pull request from a session on behalf of pelikhan July 16, 2026 12:54 View session

Copilot AI requested a review from pelikhan July 16, 2026 12:54

Copilot finished work on behalf of pelikhan July 16, 2026 12:54

pelikhan marked this pull request as ready for review July 16, 2026 13:17

Copilot AI review requested due to automatic review settings July 16, 2026 13:17

Copilot started reviewing on behalf of pelikhan July 16, 2026 13:17 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

github-actions Bot approved these changes Jul 16, 2026

View reviewed changes

This was referenced Jul 16, 2026

[Contribution Check Report] Contribution Check — 2026-07-16 #45866

Closed

[aw] No-Op Runs #43499

Open

github-actions Bot approved these changes Jul 16, 2026

View reviewed changes

github-actions Bot reviewed Jul 16, 2026

View reviewed changes

Copilot started work on behalf of pelikhan July 16, 2026 13:42 View session

Copilot AI and others added 2 commits July 16, 2026 13:52

chore: initial state (no changes yet)

9fced06

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

fix: use constants.CodexEngine constant instead of raw string literal…

7ff45da

… in evals_steps.go Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 16, 2026 14:11

pelikhan merged commit f1e2e6b into main Jul 16, 2026
29 checks passed

pelikhan deleted the copilot/investigate-compiler-bug branch July 16, 2026 14:28

This was referenced Jul 16, 2026

Smoke Test: MAI-Code-1-Flash - 29538613340 #46092

Closed

Smoke Test: MAI-Code-1-Flash - 29562782974 #46167

Closed

Uh oh!

Conversation

Copilot AI commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Copilot AI Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

github-actions Bot commented Jul 16, 2026

🧪 Test Quality Sentinel Report

Code Quality Highlights

Verdict

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Positive Highlights

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Review: non-blocking observations, consider before next similar patch

Comments that could not be inline-anchored

Uh oh!

pelikhan commented Jul 16, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading