fix: detect "no model endpoints available" as http_400_response_error#42593
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…rror Extend HTTP_400_RESPONSE_ERROR_PATTERN in both copilot_harness.cjs and detect_agent_errors.cjs to also match the Copilot SDK error message: "400 400 400 no model endpoints available given user constraints" Previously this error fell through to `partial_execution` and triggered retries that could never succeed. It is now classified as `http_400_response_error` and treated as non-retryable. Observed in run 28484493629 (PR Sous Chef) job 84428042868 step 35. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
✅ Test Quality Sentinel completed test quality analysis. |
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅ |
|
✅ PR Code Quality Reviewer completed the code quality review. |
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. No ADR enforcement needed: PR #42593 does not have the 'implementation' label and has 0 new lines of code in business logic directories (threshold: 100). |
There was a problem hiding this comment.
Pull request overview
This pull request improves Copilot harness error classification by recognizing the Copilot SDK’s “no model endpoints available given user constraints” output as a non-retryable HTTP 400 response, avoiding wasted retries and misclassification as partial_execution.
Changes:
- Expanded
HTTP_400_RESPONSE_ERROR_PATTERNin both the harness and shared detection script to match the new Copilot SDK error string. - Added unit tests to ensure
isHTTP400ResponseError()matches the new error shape (including when embedded in larger output). - Updated the generated
pr-code-quality-reviewer.lock.ymlmetadata hash.
Show a summary per file
| File | Description |
|---|---|
| actions/setup/js/detect_agent_errors.cjs | Extends HTTP 400 detection to include the “no model endpoints…” SDK message. |
| actions/setup/js/copilot_harness.cjs | Keeps the harness’ HTTP 400 pattern in sync with the shared detector and ensures the failure class becomes non-retryable. |
| actions/setup/js/copilot_harness.test.cjs | Adds coverage for the new “no model endpoints…” HTTP 400 detection cases. |
| .github/workflows/pr-code-quality-reviewer.lock.yml | Updates generated workflow metadata (body_hash). |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 4/4 changed files
- Comments generated: 2
- Review effort level: Low
| // NOTE: keep in sync with HTTP_400_RESPONSE_ERROR_PATTERN in copilot_harness.cjs. | ||
| const HTTP_400_RESPONSE_ERROR_PATTERN = /Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?/i; | ||
| // Also matches "400 400 400 no model endpoints available given user constraints" which is emitted | ||
| // by the Copilot SDK when no model endpoints are available for the user's configured constraints. | ||
| const HTTP_400_RESPONSE_ERROR_PATTERN = | ||
| /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i; |
There was a problem hiding this comment.
Added the missing test cases to the HTTP_400_RESPONSE_ERROR_PATTERN block in detect_agent_errors.test.cjs.
There was a problem hiding this comment.
Review: fix: detect "no model endpoints available" as http_400_response_error
The fix is correct and well-targeted. The regex extension is safe (no ReDoS risk), both files are kept in sync, and the new tests in copilot_harness.test.cjs cover the exact error and the embedded-in-output case.
One non-blocking suggestion: detect_agent_errors.test.cjs has its own describe("HTTP_400_RESPONSE_ERROR_PATTERN") suite that should mirror the new test cases (see inline comment).
Everything else looks good — no security issues, no correctness issues, and the fix directly addresses the observed failure in run 28484493629.
🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · 30.6 AIC · ⌖ 6.52 AIC · ⊞ 4.8K
| // Also matches "400 400 400 no model endpoints available given user constraints" which is emitted | ||
| // by the Copilot SDK when no model endpoints are available for the user's configured constraints. | ||
| const HTTP_400_RESPONSE_ERROR_PATTERN = | ||
| /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i; |
There was a problem hiding this comment.
detect_agent_errors.test.cjs has its own describe("HTTP_400_RESPONSE_ERROR_PATTERN", ...) suite (lines 185–198), but no test case for the new no model endpoints available given user constraints alternative was added there. Since the pattern is kept in sync between both files, the test suite should be too.
Suggested additions to detect_agent_errors.test.cjs:
it("matches the 'no model endpoints available given user constraints' SDK error", () => {
expect(HTTP_400_RESPONSE_ERROR_PATTERN.test("[copilot-sdk-driver] [sdk-driver] error: 400 400 400 no model endpoints available given user constraints")).toBe(true);
});
it("matches the no-model-endpoints error embedded in larger output", () => {
const output = 'some prior output\n[copilot-sdk-driver] [sdk-driver] error: 400 400 400 no model endpoints available given user constraints\n{"type":"subagent.failed"}';
expect(HTTP_400_RESPONSE_ERROR_PATTERN.test(output)).toBe(true);
});@copilot please address this.
There was a problem hiding this comment.
Added three test cases to the HTTP_400_RESPONSE_ERROR_PATTERN describe block in detect_agent_errors.test.cjs in commit fix: anchor no-model-endpoints regex...: exact SDK error line, embedded-in-larger-output, and a negative case confirming the phrase without a leading 400 does not match (since I also anchored the regex — see the related thread).
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /tdd and /diagnosing-bugs — one observation on test coverage symmetry.
📋 Key Themes & Highlights
Key Themes
- Test coverage gap:
detect_agent_errors.test.cjsdirectly testsHTTP_400_RESPONSE_ERROR_PATTERNfromdetect_agent_errors.cjs, but wasn't updated with cases for the newno model endpoints availablealternative. Two matching tests exist incopilot_harness.test.cjs(viaisHTTP400ResponseError), but the companion file remains unsynced. See inline comment for suggested additions.
Positive Highlights
- ✅ Root cause correctly identified and fixed — the pattern now classifies this error as non-retryable, stopping wasted retry cycles
- ✅ Both
copilot_harness.cjsanddetect_agent_errors.cjsupdated atomically, maintaining the "keep in sync" contract - ✅ The new regex alternative is specific enough to avoid false positives
- ✅ Two new tests cover both the exact SDK error string and the embedded-in-larger-output case
- ✅ PR description links directly to the failing run — excellent traceability
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · 50.5 AIC · ⌖ 7.43 AIC · ⊞ 6.6K
Comment /matt to run again
| // Also matches "400 400 400 no model endpoints available given user constraints" which is emitted | ||
| // by the Copilot SDK when no model endpoints are available for the user's configured constraints. | ||
| const HTTP_400_RESPONSE_ERROR_PATTERN = | ||
| /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i; |
There was a problem hiding this comment.
[/tdd] detect_agent_errors.test.cjs was not updated with tests for the new pattern variant — this file directly tests HTTP_400_RESPONSE_ERROR_PATTERN but still only covers the original Response status code... form.
💡 Suggested additions to detect_agent_errors.test.cjs
Add inside the existing describe("HTTP_400_RESPONSE_ERROR_PATTERN", ...) block:
it("matches the Copilot SDK no-model-endpoints error", () => {
expect(HTTP_400_RESPONSE_ERROR_PATTERN.test(
"[copilot-sdk-driver] [sdk-driver] error: 400 400 400 no model endpoints available given user constraints"
)).toBe(true);
});
it("matches the no-model-endpoints error embedded in larger output", () => {
const log =
"some output\n[sdk-driver] error: 400 400 400 no model endpoints available given user constraints\n{}";
expect(HTTP_400_RESPONSE_ERROR_PATTERN.test(log)).toBe(true);
});Keeping both test files in sync with the pattern they each cover prevents silent regressions if the two files ever diverge.
@copilot please address this.
There was a problem hiding this comment.
Added the missing test cases to detect_agent_errors.test.cjs — exact SDK error line, embedded in larger output, and a negative case for the bare phrase without a 400 prefix.
🧪 Test Quality Sentinel Report✅ Test Quality Score: 90/100 — Excellent
📊 Metrics & Test Classification (2 tests analyzed)
JavaScript: 2 ( Score breakdown:
🔍 Individual Test Analysis
Verifies the new regex branch correctly classifies the exact Copilot SDK error message
Tests that the regex matches when the error appears mid-stream in realistic log output (surrounded by other lines and JSON). This mirrors actual Minor observation: Verdict
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
|
There was a problem hiding this comment.
REQUEST_CHANGES — one correctness issue must be fixed before merge
The fix direction is correct: classifying no model endpoints available given user constraints as a non-retryable HTTP-400 error stops wasted retries. The problem is the regex implementation.
🔴 Blocking: unanchored regex causes false positive non-retryable kills
Both detect_agent_errors.cjs and copilot_harness.cjs use:
/(?:...existing pattern...|no model endpoints available given user constraints)/iThis matches the phrase anywhere in output — no 400 prefix required. If a debug line, an error summary, or any contextual message ever contains this phrase without being the actual SDK failure, the run will be killed as non-retryable incorrectly.
The observed error always carries a 400 prefix (400 400 400 no model endpoints...). The regex must require it. See the inline comments for the suggested fix.
🟡 Non-blocking: `detect_agent_errors.test.cjs` has no coverage for the new pattern variant
copilot_harness.test.cjs got two new test cases for the new alternative, but detect_agent_errors.test.cjs was not updated. The HTTP_400_RESPONSE_ERROR_PATTERN in detect_agent_errors.cjs is a separate copy that must be kept in sync — it should have equivalent tests. Suggested additions:
it("matches the 'no model endpoints available given user constraints' SDK error", () => {
expect(HTTP_400_RESPONSE_ERROR_PATTERN.test(
"[sdk] error: 400 400 400 no model endpoints available given user constraints"
)).toBe(true);
});
it("does not match phrase without 400 prefix after fix", () => {
expect(HTTP_400_RESPONSE_ERROR_PATTERN.test(
"Warning: no model endpoints available given user constraints"
)).toBe(false);
});🔵 Minor: unexplained `body_hash` change in lock file
.github/workflows/pr-code-quality-reviewer.lock.yml has a changed body_hash but the corresponding .md workflow source was not modified in this PR. If this is stale churn from make recompile picking up an unrelated change, it should be reverted. If it represents a deliberate workflow update, the source change should be visible in the diff.
Warning
Firewall blocked 1 domain
The following domain was blocked by the firewall during workflow execution:
patchdiff.githubusercontent.com
To allow these domains, add them to the
network.allowedlist in your workflow frontmatter:
network:
allowed:
- defaults
- "patchdiff.githubusercontent.com"See Network Configuration for more information.
🔎 Code quality review by PR Code Quality Reviewer · 68.7 AIC · ⌖ 7.63 AIC · ⊞ 1.6K
Comment /review to run again
| // Also matches "400 400 400 no model endpoints available given user constraints" which is emitted | ||
| // by the Copilot SDK when no model endpoints are available for the user's configured constraints. | ||
| const HTTP_400_RESPONSE_ERROR_PATTERN = | ||
| /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i; |
There was a problem hiding this comment.
Regex alternative is unanchored to the 400 status context — false positive non-retryable classification risk.
💡 Explanation and suggested fix
The new no model endpoints available given user constraints branch matches that phrase anywhere in the output with no requirement that a 400 status code precede it. Any diagnostic log line, SDK warning, or even an informational message containing that phrase would be silently classified as http_400_response_error and the run terminated as non-retryable — even if the actual run could still succeed.
The observed SDK error was 400 400 400 no model endpoints available given user constraints, so the 400 prefix is always present in real occurrences. Anchor the new branch to it:
const HTTP_400_RESPONSE_ERROR_PATTERN =
/(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|400[^\n]*no model endpoints available given user constraints)/i;The same unanchored issue is present in the mirror copy in copilot_harness.cjs and should be fixed there too.
The comment on line 68 says this matches "400 400 400 no model endpoints..." which implies the triple-400 prefix is expected — the regex should reflect that documented intent.
There was a problem hiding this comment.
Fixed — changed the second alternative to 400[^\n]*no model endpoints available given user constraints in both detect_agent_errors.cjs and copilot_harness.cjs. The pattern now requires a leading 400 on the same line, eliminating the false-positive risk. Also added a negative test case to detect_agent_errors.test.cjs confirming the bare phrase without 400 does not match.
| // Also matches "400 400 400 no model endpoints available given user constraints" which is emitted | ||
| // by the Copilot SDK when no model endpoints are available for the user's configured constraints. | ||
| const HTTP_400_RESPONSE_ERROR_PATTERN = | ||
| /(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|no model endpoints available given user constraints)/i; |
There was a problem hiding this comment.
Same unanchored regex as detect_agent_errors.cjs — requires the same fix here to stay in sync.
💡 Suggested fix
The no model endpoints available given user constraints alternative needs to be anchored to the 400 prefix to prevent false positive non-retryable classification:
const HTTP_400_RESPONSE_ERROR_PATTERN =
/(?:Response status code does not indicate success:\s*400(?:\s*\(Bad Request\))?|400[^\n]*no model endpoints available given user constraints)/i;See the detailed explanation in the corresponding comment on detect_agent_errors.cjs.
There was a problem hiding this comment.
Fixed in sync with detect_agent_errors.cjs — same 400[^\n]* anchoring applied here.
🤖 PR Triage — §28486872548
Score breakdown: Impact 25/50 · Urgency 15/30 · Quality 14/20 Rationale: Targeted bug fix (+18/-3) detecting unhandled 400 model-endpoint errors in JS harness. Has tests. Mixed reviews (APPROVED + CHANGES_REQUESTED). Small, well-scoped fix ready for expedited review. Labels applied:
|
|
@copilot run pr-finisher skill |
…errors tests; revert lock churn - Fix unanchored regex in detect_agent_errors.cjs and copilot_harness.cjs: change `no model endpoints available given user constraints` to `400[^\n]*no model endpoints available given user constraints` to prevent false positive non-retryable classification - Add three new test cases to detect_agent_errors.test.cjs for the new pattern alternative (exact SDK error, embedded in output, bare phrase without 400 prefix should not match) - Revert accidental body_hash churn in pr-code-quality-reviewer.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Done. All review threads have been addressed:
All local checks (fmt, lint, test-unit, 236 JS tests) pass. |
|
🎉 This pull request is included in a new release. Release: |
The Copilot SDK emits
400 400 400 no model endpoints available given user constraintswhen no model endpoints match the user's constraints. This string didn't matchHTTP_400_RESPONSE_ERROR_PATTERN, soisHTTP400ResponseErrorreturnedfalse, the failure fell through topartial_execution, and the harness wasted retries that could never succeed (observed in run 28484493629).Changes
copilot_harness.cjs/detect_agent_errors.cjs— extendHTTP_400_RESPONSE_ERROR_PATTERNto also match the new error string:The error is now classified as
http_400_response_errorand treated as non-retryable.copilot_harness.test.cjs— two new test cases for the new alternative: exact SDK error line and embedded-in-larger-output.