Skip to content

Add max-cache-misses guardrail for API proxy token budget enforcement#5202

Merged
lpcox merged 3 commits into
mainfrom
copilot/max-cache-misses-fix
Jun 18, 2026
Merged

Add max-cache-misses guardrail for API proxy token budget enforcement#5202
lpcox merged 3 commits into
mainfrom
copilot/max-cache-misses-fix

Conversation

Copilot AI commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

This PR adds a new API proxy guardrail to stop runs after too many consecutive cache misses. The guard enforces a configurable threshold for consecutive responses with input_tokens > 0 and cache_read_tokens == 0, which closes the gap where cache effectiveness regressions were not being caught.

  • Guardrail behavior (max-cache-misses)

    • Added max-cache-misses enforcement in the API proxy guard pipeline.
    • Tracks a consecutive miss streak only on successful usage-bearing responses:
      • increments when input_tokens > 0 && cache_read_tokens == 0
      • resets when cache_read_tokens > 0
      • ignores zero-input responses
    • Blocks subsequent requests with 429 and error.type = max_cache_misses_exceeded once threshold is reached.
  • Proxy/runtime integration

    • Wired into shared guard checks used by both HTTP and WebSocket paths.
    • Wired into token budget usage flow so streak state is updated from normalized usage extraction.
    • Exposed current state in /reflect as cache_misses.
  • Configuration surface (CLI + config file + schema + env wiring)

    • Added CLI option: --max-cache-misses <number>.
    • Added config key: apiProxy.maxCacheMisses.
    • Added validation and config assembly plumbing through existing log-and-limits/build-config paths.
    • Added container env projection to API proxy: AWF_MAX_CACHE_MISSES.
    • Updated both source and published AWF config schemas to include the new field.
  • Representative behavior

    {
      "apiProxy": {
        "maxCacheMisses": 3
      }
    }

    After 3 consecutive successful responses with non-zero input_tokens and zero cache_read_tokens, the next request is rejected with:

    {
      "error": {
        "type": "max_cache_misses_exceeded",
        "consecutive_cache_misses": 3,
        "max_cache_misses": 3
      }
    }

Copilot AI changed the title [WIP] Fix max cache misses guardrail for cached tokens Add max-cache-misses guardrail for API proxy token budget enforcement Jun 17, 2026
Copilot finished work on behalf of lpcox June 17, 2026 23:14
Copilot AI requested a review from lpcox June 17, 2026 23:14
@lpcox
lpcox marked this pull request as ready for review June 17, 2026 23:37
Copilot AI review requested due to automatic review settings June 17, 2026 23:37
@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation build failed for this PR. View logs.

Built from commit 50f41d0

@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 97.57% 97.61% 📈 +0.04%
Statements 97.49% 97.53% 📈 +0.04%
Functions 98.84% 98.84% ➡️ +0.00%
Branches 92.93% 92.98% 📈 +0.05%
📁 Per-file Coverage Changes (1 files)
File Lines (Before → After) Statements (Before → After)
src/workdir-setup.ts 92.7% → 94.5% (+1.82%) 92.7% → 94.5% (+1.82%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new API-proxy guardrail (max-cache-misses) to detect cache effectiveness regressions by tracking consecutive “usage-bearing” responses where input_tokens > 0 and cache_read_tokens === 0, and blocking subsequent requests once a configured threshold is reached.

Changes:

  • Added a max-cache-misses guard in the api-proxy guard pipeline (HTTP + WebSocket), returning HTTP 429 with error.type = max_cache_misses_exceeded once the streak limit is hit.
  • Wired the guard into normalized usage processing and exposed its state in /reflect as cache_misses.
  • Plumbed configuration end-to-end (CLI flag, config file mapping, validation, container env projection, and both schema copies) with accompanying tests.
Show a summary per file
File Description
src/types/rate-limit-options.ts Adds maxCacheMisses to the typed guard/limit options surface.
src/services/api-proxy-service-rate-limit.test.ts Verifies AWF_MAX_CACHE_MISSES env projection into the api-proxy service.
src/services/api-proxy-service-config.ts Projects maxCacheMisses into AWF_MAX_CACHE_MISSES for the api-proxy container.
src/schema.test.ts Extends schema coverage to validate apiProxy.maxCacheMisses.
src/config-file.ts Adds apiProxy.maxCacheMisses to config-file type + CLI mapping.
src/config-file-mapping.test.ts Tests mapping behavior for apiProxy.maxCacheMisses.
src/commands/validators/log-and-limits.ts Validates maxCacheMisses as a positive integer and returns it in validator output.
src/commands/validators/log-and-limits.test.ts Adds success/failure coverage for maxCacheMisses validation.
src/commands/validators/config-assembly.ts Threads validated maxCacheMisses into final assembled config.
src/commands/validators/config-assembly.test.ts Updates minimal config fixture to include maxCacheMisses.
src/commands/validate-options.test.ts Adds validateOptions coverage for invalid maxCacheMisses.
src/commands/build-config.ts Includes maxCacheMisses in the assembled WrapperConfig.
src/commands/build-config.test.ts Updates minimal build-config inputs to include maxCacheMisses.
src/cli-options.ts Adds --max-cache-misses CLI flag help text.
src/awf-config-schema.json Updates the source JSON schema with apiProxy.maxCacheMisses.
docs/awf-config.schema.json Updates the published docs schema with apiProxy.maxCacheMisses.
containers/api-proxy/websocket-proxy.js Wires max-cache-misses guard deps into the WebSocket proxy guard checks.
containers/api-proxy/token-budget-log.js Updates guard streak state based on normalized usage via applyMaxCacheMissesUsage.
containers/api-proxy/server.websocket.test.js Adds WebSocket guard coverage for max-cache-misses blocking behavior.
containers/api-proxy/server.token-guards.test.js Adds HTTP guard coverage: block after consecutive misses; reset on cache hits.
containers/api-proxy/server.network.test.js Ensures /reflect output includes the new cache_misses section.
containers/api-proxy/server.js Exposes max-cache-misses reflect state through management handlers.
containers/api-proxy/proxy-request.js Integrates max-cache-misses into shared guard enforcement + exports reflect/test reset.
containers/api-proxy/management.js Adds cache_misses to reflect endpoint payload.
containers/api-proxy/guards/max-cache-misses-guard.test.js Unit tests for streak tracking, reset semantics, reflect state, and error payload.
containers/api-proxy/guards/max-cache-misses-guard.js Implements the new guard state machine + reflect + error builder.
containers/api-proxy/guards/common-guard-checks.js Adds the new guard descriptor into the common guard check list.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 27/27 changed files
  • Comments generated: 1

Comment thread src/cli-options.ts
@lpcox

lpcox commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

@copilot address review feedback

Copilot AI commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address review feedback

Done — changed cache_read_tokens = 0 to cache_read_tokens === 0 in the --max-cache-misses help text (commit Fix help text: use === instead of = for cache_read_tokens comparison).

Copilot finished work on behalf of lpcox June 18, 2026 00:29
@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Smoke Claude failed

No actionable task was provided in the prompt — the workflow template appears to contain an unrendered {{#runtime-import .github/workflows/smoke-claude.md}} directive with no concrete instructions. No GitHub action taken.

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Contribution Check completed successfully!

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Build Test Suite completed successfully!

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Smoke Gemini completed. All facets verified. 💎

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) ModePASS

  • ✅ GitHub MCP connectivity verified
  • ✅ github.com HTTP 200
  • ✅ File write/read test passed
  • ✅ BYOK inference working (direct BYOK mode via COPILOT_PROVIDER_API_KEY)

Status: Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

Copy link
Copy Markdown
Contributor

🔬 Smoke Test Results — PASS

PR: Add max-cache-misses guardrail for API proxy token budget enforcement
Author: @Copilot | Assignees: @lpcox, @Copilot

Test Result
GitHub MCP connectivity
GitHub.com HTTP connectivity (200)
File write/read test ⚠️ pre-step vars unexpanded

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox

  • GitHub MCP PR data: ✅
  • GitHub.com connectivity: ✅
  • File write/read: ✅
  • BYOK inference: ✅

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Overall status: PASS

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@github-actions

Copy link
Copy Markdown
Contributor

🔥 Smoke Test: Copilot PAT Auth — PR #5202

Test Result
GitHub MCP connectivity ✅ Pass
GitHub.com HTTP ✅ 200 OK
File write/read ❌ Fail — pre-step template vars unresolved

Overall: FAIL ⚠️ Pre-computed step outputs (SMOKE_HTTP_CODE, SMOKE_FILE_PATH, etc.) were not substituted — workflow expression evaluation issue.

Auth mode: PAT (COPILOT_GITHUB_TOKEN) | Author: @Copilot | Assignees: @lpcox @Copilot

🔑 PAT report filed by Smoke Copilot PAT

@github-actions

Copy link
Copy Markdown
Contributor

Smoke test: PASS
Add max-cache-misses guardrail for API proxy token budget enforcement
Reduce Pelis Advisor AI credit burn with fixed low-cost model + turn cap
fix(ci-cd-gaps-assessment): resolve AI credits rate limit with DataOps refactor
✅ GitHub PR queries
✅ GitHub title check
✅ File write/read
✅ Discussion comment
✅ npm ci && npm run build

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions

Copy link
Copy Markdown
Contributor

🔍 Smoke Test: API Proxy OpenTelemetry Tracing

Scenario Result Details
S1: Module Loading otel.js loads; exports 7 public functions (startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled) + test helpers
S2: Test Suite 39/39 tests passed in otel.test.js — spans, token attrs, budget attrs, serialization, exporters, shutdown
S3: Env Var Forwarding api-proxy-service-config.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID, OTEL_SERVICE_NAME
S4: Token Tracker Integration onUsage callback exists in proxy-request.js (line 324); wired to setTokenAttributes + setBudgetAttributes + startRequestSpan / endSpan / endSpanError
S5: OTEL Diagnostics / Graceful Degradation Falls back to FileSpanExporter (/var/log/api-proxy/otel.jsonl) when no OTLP endpoint configured; no errors

All 5 scenarios pass. OTEL tracing integration is fully functional on this branch.

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.13 Python 3.12.3
Node.js v24.16.0 v22.22.3
Go go1.22.12 go1.22.12

Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot environments.

Tested by Smoke Chroot

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color passed ✅ PASS
Go env passed ✅ PASS
Go uuid passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx passed ✅ PASS
Node.js execa passed ✅ PASS
Node.js p-limit passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #5202 ·

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Results: Gemini Engine Validation

  • GitHub MCP Testing: ❌ (Tools missing)
  • GitHub.com Connectivity: ❌ (HTTP 000/SSL error)
  • File Writing Testing: ✅
  • Bash Tool Testing: ✅

Overall Status: FAIL

Note: GitHub MCP tools were not found in the environment, and connectivity to github.com failed with SSL errors.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test Results

  • Redis PING: ❌ (no response on host.docker.internal:6379)
  • PostgreSQL pg_isready: ❌ (no response on host.docker.internal:5432)
  • PostgreSQL SELECT 1: ❌ (connection failed)

host.docker.internal resolves to 172.17.0.1 but both ports are unreachable — service containers are not accessible from this runner.

Overall: FAIL

🔌 Service connectivity validated by Smoke Services

@lpcox
lpcox merged commit fa7e87a into main Jun 18, 2026
81 of 86 checks passed
@lpcox
lpcox deleted the copilot/max-cache-misses-fix branch June 18, 2026 01:12
github-actions Bot added a commit that referenced this pull request Jun 19, 2026
PR #5202 added the maxCacheMisses guardrail with full implementation:
- JSON schemas (src + docs)
- TypeScript type (RateLimitOptions)
- config-file.ts mapping
- CLI option --max-cache-misses
- AWF_MAX_CACHE_MISSES env var wiring

But docs/awf-config-spec.md was not updated with:
1. Section 5 CLI Mapping entry for apiProxy.maxCacheMisses → --max-cache-misses
2. Behavioral spec section (§11b) describing counting rules, enforcement,
   introspection format, and configuration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lpcox pushed a commit that referenced this pull request Jun 19, 2026
* fix(spec): add maxCacheMisses to Section 5 CLI mapping and add §11b spec

PR #5202 added the maxCacheMisses guardrail with full implementation:
- JSON schemas (src + docs)
- TypeScript type (RateLimitOptions)
- config-file.ts mapping
- CLI option --max-cache-misses
- AWF_MAX_CACHE_MISSES env var wiring

But docs/awf-config-spec.md was not updated with:
1. Section 5 CLI Mapping entry for apiProxy.maxCacheMisses → --max-cache-misses
2. Behavioral spec section (§11b) describing counting rules, enforcement,
   introspection format, and configuration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(spec): align cache-miss guard wording and reflect key

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants