Skip to content

fix: detect "no model endpoints available" as http_400_response_error#42593

Merged
pelikhan merged 5 commits into
mainfrom
copilot/investigate-400-error
Jul 1, 2026
Merged

fix: detect "no model endpoints available" as http_400_response_error#42593
pelikhan merged 5 commits into
mainfrom
copilot/investigate-400-error

Conversation

Copilot AI commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

The Copilot SDK emits 400 400 400 no model endpoints available given user constraints when no model endpoints match the user's constraints. This string didn't match HTTP_400_RESPONSE_ERROR_PATTERN, so isHTTP400ResponseError returned false, the failure fell through to partial_execution, and the harness wasted retries that could never succeed (observed in run 28484493629).

Changes

  • copilot_harness.cjs / detect_agent_errors.cjs — extend HTTP_400_RESPONSE_ERROR_PATTERN to also match the new error string:

    const HTTP_400_RESPONSE_ERROR_PATTERN =
      /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i;

    The error is now classified as http_400_response_error and treated as non-retryable.

  • copilot_harness.test.cjs — two new test cases for the new alternative: exact SDK error line and embedded-in-larger-output.

Copilot AI and others added 3 commits July 1, 2026 00:41
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…rror

Extend HTTP_400_RESPONSE_ERROR_PATTERN in both copilot_harness.cjs and
detect_agent_errors.cjs to also match the Copilot SDK error message:
  "400 400 400 no model endpoints available given user constraints"

Previously this error fell through to `partial_execution` and triggered
retries that could never succeed. It is now classified as
`http_400_response_error` and treated as non-retryable.

Observed in run 28484493629 (PR Sous Chef) job 84428042868 step 35.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title fix: detect '400 no model endpoints available' as http_400_response_error fix: detect "no model endpoints available" as http_400_response_error Jul 1, 2026
Copilot AI requested a review from pelikhan July 1, 2026 00:47
@pelikhan pelikhan marked this pull request as ready for review July 1, 2026 00:52
Copilot AI review requested due to automatic review settings July 1, 2026 00:52
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Test Quality Sentinel completed test quality analysis.

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

PR Code Quality Reviewer completed the code quality review.

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR #42593 does not have the 'implementation' label and has 0 new lines of code in business logic directories (threshold: 100).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request improves Copilot harness error classification by recognizing the Copilot SDK’s “no model endpoints available given user constraints” output as a non-retryable HTTP 400 response, avoiding wasted retries and misclassification as partial_execution.

Changes:

  • Expanded HTTP_400_RESPONSE_ERROR_PATTERN in both the harness and shared detection script to match the new Copilot SDK error string.
  • Added unit tests to ensure isHTTP400ResponseError() matches the new error shape (including when embedded in larger output).
  • Updated the generated pr-code-quality-reviewer.lock.yml metadata hash.
Show a summary per file
File Description
actions/setup/js/detect_agent_errors.cjs Extends HTTP 400 detection to include the “no model endpoints…” SDK message.
actions/setup/js/copilot_harness.cjs Keeps the harness’ HTTP 400 pattern in sync with the shared detector and ensures the failure class becomes non-retryable.
actions/setup/js/copilot_harness.test.cjs Adds coverage for the new “no model endpoints…” HTTP 400 detection cases.
.github/workflows/pr-code-quality-reviewer.lock.yml Updates generated workflow metadata (body_hash).

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 4/4 changed files
  • Comments generated: 2
  • Review effort level: Low

Comment on lines +67 to +71
// NOTE: keep in sync with HTTP_400_RESPONSE_ERROR_PATTERN in copilot_harness.cjs.
const HTTP_400_RESPONSE_ERROR_PATTERN = /Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?/i;
// Also matches "400 400 400 no model endpoints available given user constraints" which is emitted
// by the Copilot SDK when no model endpoints are available for the user's configured constraints.
const HTTP_400_RESPONSE_ERROR_PATTERN =
/(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the missing test cases to the HTTP_400_RESPONSE_ERROR_PATTERN block in detect_agent_errors.test.cjs.

Comment thread .github/workflows/pr-code-quality-reviewer.lock.yml
@github-actions github-actions Bot mentioned this pull request Jul 1, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: fix: detect "no model endpoints available" as http_400_response_error

The fix is correct and well-targeted. The regex extension is safe (no ReDoS risk), both files are kept in sync, and the new tests in copilot_harness.test.cjs cover the exact error and the embedded-in-output case.

One non-blocking suggestion: detect_agent_errors.test.cjs has its own describe("HTTP_400_RESPONSE_ERROR_PATTERN") suite that should mirror the new test cases (see inline comment).

Everything else looks good — no security issues, no correctness issues, and the fix directly addresses the observed failure in run 28484493629.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · 30.6 AIC · ⌖ 6.52 AIC · ⊞ 4.8K

// Also matches "400 400 400 no model endpoints available given user constraints" which is emitted
// by the Copilot SDK when no model endpoints are available for the user's configured constraints.
const HTTP_400_RESPONSE_ERROR_PATTERN =
/(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

detect_agent_errors.test.cjs has its own describe("HTTP_400_RESPONSE_ERROR_PATTERN", ...) suite (lines 185–198), but no test case for the new no model endpoints available given user constraints alternative was added there. Since the pattern is kept in sync between both files, the test suite should be too.

Suggested additions to detect_agent_errors.test.cjs:

it("matches the 'no model endpoints available given user constraints' SDK error", () => {
  expect(HTTP_400_RESPONSE_ERROR_PATTERN.test("[copilot-sdk-driver] [sdk-driver] error: 400 400 400 no model endpoints available given user constraints")).toBe(true);
});

it("matches the no-model-endpoints error embedded in larger output", () => {
  const output = 'some prior output\n[copilot-sdk-driver] [sdk-driver] error: 400 400 400 no model endpoints available given user constraints\n{"type":"subagent.failed"}';
  expect(HTTP_400_RESPONSE_ERROR_PATTERN.test(output)).toBe(true);
});

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added three test cases to the HTTP_400_RESPONSE_ERROR_PATTERN describe block in detect_agent_errors.test.cjs in commit fix: anchor no-model-endpoints regex...: exact SDK error line, embedded-in-larger-output, and a negative case confirming the phrase without a leading 400 does not match (since I also anchored the regex — see the related thread).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /tdd and /diagnosing-bugs — one observation on test coverage symmetry.

📋 Key Themes & Highlights

Key Themes

  • Test coverage gap: detect_agent_errors.test.cjs directly tests HTTP_400_RESPONSE_ERROR_PATTERN from detect_agent_errors.cjs, but wasn't updated with cases for the new no model endpoints available alternative. Two matching tests exist in copilot_harness.test.cjs (via isHTTP400ResponseError), but the companion file remains unsynced. See inline comment for suggested additions.

Positive Highlights

  • ✅ Root cause correctly identified and fixed — the pattern now classifies this error as non-retryable, stopping wasted retry cycles
  • ✅ Both copilot_harness.cjs and detect_agent_errors.cjs updated atomically, maintaining the "keep in sync" contract
  • ✅ The new regex alternative is specific enough to avoid false positives
  • ✅ Two new tests cover both the exact SDK error string and the embedded-in-larger-output case
  • ✅ PR description links directly to the failing run — excellent traceability

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · 50.5 AIC · ⌖ 7.43 AIC · ⊞ 6.6K
Comment /matt to run again

// Also matches "400 400 400 no model endpoints available given user constraints" which is emitted
// by the Copilot SDK when no model endpoints are available for the user's configured constraints.
const HTTP_400_RESPONSE_ERROR_PATTERN =
/(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] detect_agent_errors.test.cjs was not updated with tests for the new pattern variant — this file directly tests HTTP_400_RESPONSE_ERROR_PATTERN but still only covers the original Response status code... form.

💡 Suggested additions to detect_agent_errors.test.cjs

Add inside the existing describe("HTTP_400_RESPONSE_ERROR_PATTERN", ...) block:

it("matches the Copilot SDK no-model-endpoints error", () => {
  expect(HTTP_400_RESPONSE_ERROR_PATTERN.test(
    "[copilot-sdk-driver] [sdk-driver] error: 400 400 400 no model endpoints available given user constraints"
  )).toBe(true);
});

it("matches the no-model-endpoints error embedded in larger output", () => {
  const log =
    "some output\n[sdk-driver] error: 400 400 400 no model endpoints available given user constraints\n{}";
  expect(HTTP_400_RESPONSE_ERROR_PATTERN.test(log)).toBe(true);
});

Keeping both test files in sync with the pattern they each cover prevents silent regressions if the two files ever diverge.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the missing test cases to detect_agent_errors.test.cjs — exact SDK error line, embedded in larger output, and a negative case for the bare phrase without a 400 prefix.

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

🧪 Test Quality Sentinel Report

Test Quality Score: 90/100 — Excellent

Analyzed 2 test(s): 2 design, 0 implementation, 0 guideline violation(s).

📊 Metrics & Test Classification (2 tests analyzed)
Metric Value
New/modified tests analyzed 2
✅ Design tests (behavioral contracts) 2 (100%)
⚠️ Implementation tests (low value) 0 (0%)
Tests with error/edge cases 2 (100%)
Duplicate test clusters 0
Test inflation detected YES (9 test lines / 4 prod lines = 2.25×)
🚨 Coding-guideline violations 0
Test File Classification Issues Detected
it("matches the 'no model endpoints available given user constraints' SDK error") actions/setup/js/copilot_harness.test.cjs:989 ✅ Design
it("matches the no-model-endpoints error embedded in larger output") actions/setup/js/copilot_harness.test.cjs:993 ✅ Design

JavaScript: 2 (*.test.cjs); Go: 0.

Score breakdown:

  • Design test ratio: (2/2) × 40 = 40
  • Edge-case coverage: (2/2) × 30 = 30
  • Duplicate penalty: 20 − 0 = 20
  • Inflation bonus: 0 (ratio 2.25 > 2.0)
  • Total: 90 / 100
🔍 Individual Test Analysis

it("matches the 'no model endpoints...' SDK error")copilot_harness.test.cjs:989

Verifies the new regex branch correctly classifies the exact Copilot SDK error message 400 400 400 no model endpoints available given user constraints as an HTTP 400 error. This is the primary regression guard for the bug fix: without it, the error silently fell through to partial_execution and triggered non-terminating retries. 1 behavioral assertion. ✅ Design test, high value.

it("matches the no-model-endpoints error embedded in larger output")copilot_harness.test.cjs:993

Tests that the regex matches when the error appears mid-stream in realistic log output (surrounded by other lines and JSON). This mirrors actual isHTTP400ResponseError(output) call-sites where output is a full program stdout buffer. 1 behavioral assertion. ✅ Design test, high value.

Minor observation: detect_agent_errors.cjs received the identical regex change but has no dedicated test file. Both patterns are kept in sync by design (comment notes this), and the copilot_harness.test.cjs tests validate the regex itself — this is acceptable but worth noting for completeness.

Verdict

Check passed. 0% implementation tests (threshold: 30%). Both new tests are high-value behavioral contracts that directly guard the bug regression. The minor inflation flag (2.25×) reflects that the fix touched two production files while tests only exist for one — not a quality concern.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🧪 Test quality analysis by Test Quality Sentinel · 54 AIC · ⌖ 16.8 AIC · ⊞ 7K ·
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Test Quality Sentinel: 90/100. 0% implementation tests (threshold: 30%). Both new tests are high-value behavioral contracts that directly guard the regression introduced by the bug fix.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES — one correctness issue must be fixed before merge

The fix direction is correct: classifying no model endpoints available given user constraints as a non-retryable HTTP-400 error stops wasted retries. The problem is the regex implementation.

🔴 Blocking: unanchored regex causes false positive non-retryable kills

Both detect_agent_errors.cjs and copilot_harness.cjs use:

/(?:...existing pattern...|no model endpoints available given user constraints)/i

This matches the phrase anywhere in output — no 400 prefix required. If a debug line, an error summary, or any contextual message ever contains this phrase without being the actual SDK failure, the run will be killed as non-retryable incorrectly.

The observed error always carries a 400 prefix (400 400 400 no model endpoints...). The regex must require it. See the inline comments for the suggested fix.

🟡 Non-blocking: `detect_agent_errors.test.cjs` has no coverage for the new pattern variant

copilot_harness.test.cjs got two new test cases for the new alternative, but detect_agent_errors.test.cjs was not updated. The HTTP_400_RESPONSE_ERROR_PATTERN in detect_agent_errors.cjs is a separate copy that must be kept in sync — it should have equivalent tests. Suggested additions:

it("matches the 'no model endpoints available given user constraints' SDK error", () => {
  expect(HTTP_400_RESPONSE_ERROR_PATTERN.test(
    "[sdk] error: 400 400 400 no model endpoints available given user constraints"
  )).toBe(true);
});

it("does not match phrase without 400 prefix after fix", () => {
  expect(HTTP_400_RESPONSE_ERROR_PATTERN.test(
    "Warning: no model endpoints available given user constraints"
  )).toBe(false);
});
🔵 Minor: unexplained `body_hash` change in lock file

.github/workflows/pr-code-quality-reviewer.lock.yml has a changed body_hash but the corresponding .md workflow source was not modified in this PR. If this is stale churn from make recompile picking up an unrelated change, it should be reverted. If it represents a deliberate workflow update, the source change should be visible in the diff.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • patchdiff.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "patchdiff.githubusercontent.com"

See Network Configuration for more information.

🔎 Code quality review by PR Code Quality Reviewer · 68.7 AIC · ⌖ 7.63 AIC · ⊞ 1.6K
Comment /review to run again

// Also matches "400 400 400 no model endpoints available given user constraints" which is emitted
// by the Copilot SDK when no model endpoints are available for the user's configured constraints.
const HTTP_400_RESPONSE_ERROR_PATTERN =
/(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regex alternative is unanchored to the 400 status context — false positive non-retryable classification risk.

💡 Explanation and suggested fix

The new no model endpoints available given user constraints branch matches that phrase anywhere in the output with no requirement that a 400 status code precede it. Any diagnostic log line, SDK warning, or even an informational message containing that phrase would be silently classified as http_400_response_error and the run terminated as non-retryable — even if the actual run could still succeed.

The observed SDK error was 400 400 400 no model endpoints available given user constraints, so the 400 prefix is always present in real occurrences. Anchor the new branch to it:

const HTTP_400_RESPONSE_ERROR_PATTERN =
  /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|400[^\n]*no model endpoints available given user constraints)/i;

The same unanchored issue is present in the mirror copy in copilot_harness.cjs and should be fixed there too.

The comment on line 68 says this matches "400 400 400 no model endpoints..." which implies the triple-400 prefix is expected — the regex should reflect that documented intent.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — changed the second alternative to 400[^\n]*no model endpoints available given user constraints in both detect_agent_errors.cjs and copilot_harness.cjs. The pattern now requires a leading 400 on the same line, eliminating the false-positive risk. Also added a negative test case to detect_agent_errors.test.cjs confirming the bare phrase without 400 does not match.

Comment thread actions/setup/js/copilot_harness.cjs Outdated
// Also matches "400 400 400 no model endpoints available given user constraints" which is emitted
// by the Copilot SDK when no model endpoints are available for the user's configured constraints.
const HTTP_400_RESPONSE_ERROR_PATTERN =
/(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same unanchored regex as detect_agent_errors.cjs — requires the same fix here to stay in sync.

💡 Suggested fix

The no model endpoints available given user constraints alternative needs to be anchored to the 400 prefix to prevent false positive non-retryable classification:

const HTTP_400_RESPONSE_ERROR_PATTERN =
  /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|400[^\n]*no model endpoints available given user constraints)/i;

See the detailed explanation in the corresponding comment on detect_agent_errors.cjs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in sync with detect_agent_errors.cjs — same 400[^\n]* anchoring applied here.

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

🤖 PR Triage — §28486872548

Field Value
Category bug
Risk medium
Score 54 / 100
Priority medium
Action 🚀 fast_track

Score breakdown: Impact 25/50 · Urgency 15/30 · Quality 14/20

Rationale: Targeted bug fix (+18/-3) detecting unhandled 400 model-endpoint errors in JS harness. Has tests. Mixed reviews (APPROVED + CHANGES_REQUESTED). Small, well-scoped fix ready for expedited review.

Labels applied: pr-type:bug pr-risk:medium pr-priority:medium pr-action:fast_track pr-agent:copilot-swe-agent

Generated by 🔧 PR Triage Agent · 77.2 AIC · ⌖ 9.82 AIC · ⊞ 1.6K ·

@pelikhan

pelikhan commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

@copilot run pr-finisher skill

…errors tests; revert lock churn

- Fix unanchored regex in detect_agent_errors.cjs and copilot_harness.cjs:
  change `no model endpoints available given user constraints` to
  `400[^\n]*no model endpoints available given user constraints` to
  prevent false positive non-retryable classification
- Add three new test cases to detect_agent_errors.test.cjs for the new
  pattern alternative (exact SDK error, embedded in output, bare phrase
  without 400 prefix should not match)
- Revert accidental body_hash churn in pr-code-quality-reviewer.lock.yml

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

@copilot run pr-finisher skill

Done. All review threads have been addressed:

  • Unanchored regex — updated both detect_agent_errors.cjs and copilot_harness.cjs to use 400[^\n]*no model endpoints available given user constraints, preventing false-positive non-retryable classification.
  • Missing tests — added three test cases to detect_agent_errors.test.cjs (exact SDK error, embedded in larger output, negative case without leading 400).
  • Lock file churn — reverted the accidental body_hash change in pr-code-quality-reviewer.lock.yml.

All local checks (fmt, lint, test-unit, 236 JS tests) pass.

@pelikhan pelikhan merged commit 099bccf into main Jul 1, 2026
14 checks passed
@pelikhan pelikhan deleted the copilot/investigate-400-error branch July 1, 2026 02:23
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.82.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants