Add max-cache-misses guardrail for API proxy token budget enforcement#5202
Conversation
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
There was a problem hiding this comment.
Pull request overview
This PR introduces a new API-proxy guardrail (max-cache-misses) to detect cache effectiveness regressions by tracking consecutive “usage-bearing” responses where input_tokens > 0 and cache_read_tokens === 0, and blocking subsequent requests once a configured threshold is reached.
Changes:
- Added a
max-cache-missesguard in the api-proxy guard pipeline (HTTP + WebSocket), returning HTTP 429 witherror.type = max_cache_misses_exceededonce the streak limit is hit. - Wired the guard into normalized usage processing and exposed its state in
/reflectascache_misses. - Plumbed configuration end-to-end (CLI flag, config file mapping, validation, container env projection, and both schema copies) with accompanying tests.
Show a summary per file
| File | Description |
|---|---|
| src/types/rate-limit-options.ts | Adds maxCacheMisses to the typed guard/limit options surface. |
| src/services/api-proxy-service-rate-limit.test.ts | Verifies AWF_MAX_CACHE_MISSES env projection into the api-proxy service. |
| src/services/api-proxy-service-config.ts | Projects maxCacheMisses into AWF_MAX_CACHE_MISSES for the api-proxy container. |
| src/schema.test.ts | Extends schema coverage to validate apiProxy.maxCacheMisses. |
| src/config-file.ts | Adds apiProxy.maxCacheMisses to config-file type + CLI mapping. |
| src/config-file-mapping.test.ts | Tests mapping behavior for apiProxy.maxCacheMisses. |
| src/commands/validators/log-and-limits.ts | Validates maxCacheMisses as a positive integer and returns it in validator output. |
| src/commands/validators/log-and-limits.test.ts | Adds success/failure coverage for maxCacheMisses validation. |
| src/commands/validators/config-assembly.ts | Threads validated maxCacheMisses into final assembled config. |
| src/commands/validators/config-assembly.test.ts | Updates minimal config fixture to include maxCacheMisses. |
| src/commands/validate-options.test.ts | Adds validateOptions coverage for invalid maxCacheMisses. |
| src/commands/build-config.ts | Includes maxCacheMisses in the assembled WrapperConfig. |
| src/commands/build-config.test.ts | Updates minimal build-config inputs to include maxCacheMisses. |
| src/cli-options.ts | Adds --max-cache-misses CLI flag help text. |
| src/awf-config-schema.json | Updates the source JSON schema with apiProxy.maxCacheMisses. |
| docs/awf-config.schema.json | Updates the published docs schema with apiProxy.maxCacheMisses. |
| containers/api-proxy/websocket-proxy.js | Wires max-cache-misses guard deps into the WebSocket proxy guard checks. |
| containers/api-proxy/token-budget-log.js | Updates guard streak state based on normalized usage via applyMaxCacheMissesUsage. |
| containers/api-proxy/server.websocket.test.js | Adds WebSocket guard coverage for max-cache-misses blocking behavior. |
| containers/api-proxy/server.token-guards.test.js | Adds HTTP guard coverage: block after consecutive misses; reset on cache hits. |
| containers/api-proxy/server.network.test.js | Ensures /reflect output includes the new cache_misses section. |
| containers/api-proxy/server.js | Exposes max-cache-misses reflect state through management handlers. |
| containers/api-proxy/proxy-request.js | Integrates max-cache-misses into shared guard enforcement + exports reflect/test reset. |
| containers/api-proxy/management.js | Adds cache_misses to reflect endpoint payload. |
| containers/api-proxy/guards/max-cache-misses-guard.test.js | Unit tests for streak tracking, reset semantics, reflect state, and error payload. |
| containers/api-proxy/guards/max-cache-misses-guard.js | Implements the new guard state machine + reflect + error builder. |
| containers/api-proxy/guards/common-guard-checks.js | Adds the new guard descriptor into the common guard check list. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 27/27 changed files
- Comments generated: 1
|
@copilot address review feedback |
Done — changed |
|
✅ Copilot review passed with no inline comments. @copilot Add the |
|
❌ Smoke Claude failed No actionable task was provided in the prompt — the workflow template appears to contain an unrendered |
|
✅ Contribution Check completed successfully! |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓 |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded. |
|
✅ Build Test Suite completed successfully! |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
|
❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed... |
|
🔌 Smoke Services — All services reachable! ✅ |
|
✅ Smoke Gemini completed. All facets verified. 💎 |
|
Smoke Test: Copilot BYOK (Direct) Mode ✅ PASS
Status: Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com
|
🔬 Smoke Test Results — PASSPR: Add max-cache-misses guardrail for API proxy token budget enforcement
Overall: PASS ✅
|
|
Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra Overall status: PASS
|
🔥 Smoke Test: Copilot PAT Auth — PR #5202
Overall: FAIL Auth mode: PAT (COPILOT_GITHUB_TOKEN) | Author:
|
|
Smoke test: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "registry.npmjs.org"See Network Configuration for more information.
|
🔍 Smoke Test: API Proxy OpenTelemetry Tracing
All 5 scenarios pass. OTEL tracing integration is fully functional on this branch.
|
Chroot Version Comparison Results
Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot environments.
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
|
Smoke Test Results: Gemini Engine Validation
Overall Status: FAIL Note: GitHub MCP tools were not found in the environment, and connectivity to github.com failed with SSL errors. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
Smoke Test Results
Overall: FAIL
|
PR #5202 added the maxCacheMisses guardrail with full implementation: - JSON schemas (src + docs) - TypeScript type (RateLimitOptions) - config-file.ts mapping - CLI option --max-cache-misses - AWF_MAX_CACHE_MISSES env var wiring But docs/awf-config-spec.md was not updated with: 1. Section 5 CLI Mapping entry for apiProxy.maxCacheMisses → --max-cache-misses 2. Behavioral spec section (§11b) describing counting rules, enforcement, introspection format, and configuration Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(spec): add maxCacheMisses to Section 5 CLI mapping and add §11b spec PR #5202 added the maxCacheMisses guardrail with full implementation: - JSON schemas (src + docs) - TypeScript type (RateLimitOptions) - config-file.ts mapping - CLI option --max-cache-misses - AWF_MAX_CACHE_MISSES env var wiring But docs/awf-config-spec.md was not updated with: 1. Section 5 CLI Mapping entry for apiProxy.maxCacheMisses → --max-cache-misses 2. Behavioral spec section (§11b) describing counting rules, enforcement, introspection format, and configuration Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(spec): align cache-miss guard wording and reflect key --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
This PR adds a new API proxy guardrail to stop runs after too many consecutive cache misses. The guard enforces a configurable threshold for consecutive responses with
input_tokens > 0andcache_read_tokens == 0, which closes the gap where cache effectiveness regressions were not being caught.Guardrail behavior (
max-cache-misses)max-cache-missesenforcement in the API proxy guard pipeline.input_tokens > 0 && cache_read_tokens == 0cache_read_tokens > 0429anderror.type = max_cache_misses_exceededonce threshold is reached.Proxy/runtime integration
/reflectascache_misses.Configuration surface (CLI + config file + schema + env wiring)
--max-cache-misses <number>.apiProxy.maxCacheMisses.log-and-limits/build-configpaths.AWF_MAX_CACHE_MISSES.Representative behavior
{ "apiProxy": { "maxCacheMisses": 3 } }After 3 consecutive successful responses with non-zero
input_tokensand zerocache_read_tokens, the next request is rejected with:{ "error": { "type": "max_cache_misses_exceeded", "consecutive_cache_misses": 3, "max_cache_misses": 3 } }