Bundle models.dev catalog for AI credits model resolution#4589
Conversation
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
There was a problem hiding this comment.
Pull request overview
This PR adds a bundled models.dev catalog to the api-proxy and uses it as a fallback source for AI-credits model pricing resolution (after the curated in-repo pricing table), improving recognition of more models including zero-cost entries.
Changes:
- Added a bundled
models.devcatalog JSON plus a resolver to normalize catalog pricing into AWF’s$ / 1M tokenspricing shape. - Updated the AI credits guard to consult the bundled catalog when the curated pricing table can’t resolve a model.
- Updated docs and tests to cover catalog-priced models, zero-cost models, and adjusted token budget tests for newly-recognized models.
Show a summary per file
| File | Description |
|---|---|
| docs/awf-config-spec.md | Documents the new catalog-backed fallback behavior for default pricing and unknown-model rejection. |
| containers/api-proxy/ai-credits-pricing.js | Clarifies that the in-repo pricing table is curated and takes precedence over the bundled catalog. |
| containers/api-proxy/models.dev.catalog.json | Adds the bundled models.dev catalog data used for fallback resolution. |
| containers/api-proxy/models-dev-catalog.js | Implements catalog indexing and pricing normalization/resolution. |
| containers/api-proxy/models-dev-catalog.test.js | Adds unit tests for catalog pricing and zero-cost models. |
| containers/api-proxy/guards/ai-credits-guard.js | Integrates catalog-backed fallback into AI credits pricing resolution. |
| containers/api-proxy/guards/ai-credits-guard.test.js | Adds coverage for catalog-priced and zero-cost models in the AI credits guard. |
| containers/api-proxy/token-budget-log.test.js | Adjusts tests to use a truly-unknown model now that common models may be recognized via the catalog. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 7/8 changed files
- Comments generated: 4
| it('recognizes zero-cost catalog models', () => { | ||
| expect(resolveCatalogModel('google/gemma-4-31b-it:free')).toEqual({ | ||
| exists: true, | ||
| pricing: { | ||
| input: 0, | ||
| cachedInput: 0, | ||
| cacheWrite: null, | ||
| output: 0, | ||
| }, | ||
| zeroCost: true, | ||
| }); | ||
| }); | ||
| }); |
There was a problem hiding this comment.
Added regression test in commit c200251 — openrouter/pareto-code (which carries prompt/completion: "-1" in the bundled catalog) asserts pricing: null, zeroCost: false.
| When configured, any model not found in the curated built-in pricing table or | ||
| the bundled models.dev catalog uses these rates as a fallback for AI credits | ||
| calculation. |
There was a problem hiding this comment.
Updated in commit c200251 — section 10.7.1 now describes the curated-table → bundled-catalog two-step lookup, the defaultAiCreditsPricing fallback, and zero-cost model handling.
| let prefixMatch = null; | ||
| for (const [knownModel, pricing] of pricingByModel.entries()) { | ||
| if (canonical.startsWith(`${knownModel}-`)) { | ||
| if (!prefixMatch || knownModel.length > prefixMatch.key.length) { | ||
| prefixMatch = { key: knownModel, pricing }; | ||
| } | ||
| } | ||
| } | ||
|
|
||
| if (prefixMatch) { | ||
| return { exists: true, pricing: prefixMatch.pricing, zeroCost: isZeroCostPricing(prefixMatch.pricing) }; | ||
| } | ||
|
|
||
| return { exists: knownModels.has(canonical), pricing: null, zeroCost: false }; | ||
| } |
There was a problem hiding this comment.
Replaced in commit c200251 with iterative '-' segment stripping that checks the Map at each step — O(k) where k is the number of dash segments, no full scan.
|
@copilot address review feedback and merge main |
…g-json-from-models-dev
- Reject negative sentinel pricing values (e.g. "-1") in parseDollarsPerToken - Replace O(N) prefix scan with O(k) iterative '-' segment stripping - Add regression test for negative sentinel pricing entries - Update docs section 10.7.1 to describe curated-table → bundled-catalog fallback"
Done — all four review items addressed and main merged in commit
|
|
@copilot run pr-finisher skill |
Done. Here's the summary:
Hand-off: CI must be re-triggered by a maintainer (close/reopen, |
The new module is required by ai-credits-pricing.js but was missing from the container image, causing MODULE_NOT_FOUND at runtime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The JSON catalog file is required by models-dev-catalog.js at runtime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Smoke Test: Copilot BYOK (Direct) Mode ✅Test Results:
Status: PASS — Agent has access via api-proxy; real credentials held by sidecar only. cc/
|
|
Smoke Test: Copilot BYOK (Direct) Mode — Azure OpenAI (Foundry, Entra / GitHub OIDC) • GitHub MCP Connectivity: ✅ Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "api.openai.com"See Network Configuration for more information.
|
|
GitHub API: ✅ PASS Total: PASS
|
🔬 Smoke Test ResultsPR: Bundle models.dev catalog for AI credits model resolution
Overall: PASS (2/2 verified tests passed)
|
🧪 Chroot Runtime Version Comparison
Result: FAILED — Python and Node.js versions differ between host and chroot environments.
|
🔬 Smoke Test: API Proxy OpenTelemetry Tracing
All scenarios pass. OTEL tracing integration is functional.
|
|
Add hard cap on AI credits (10000) and restore agentic-workflows integration ID ✅
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
|
Smoke Test: GitHub Actions Services Connectivity
Overall: FAIL
|
AWF should be able to resolve model pricing and existence from a bundled models.dev catalog, instead of relying only on the curated pricing table or user-supplied fallback pricing. This adds a catalog-backed fallback so AI credits accounting can recognize more models, including zero-cost/free entries, before treating them as unknown.
Catalog-backed model resolution
models.devcatalog JSON to the api-proxy.AI credits fallback behavior
defaultAiCreditsPricingonly when neither the curated table nor the bundled catalog can resolve the model.Known free / zero-cost models
maxAiCreditsis enabled.Guard behavior updates
Docs and tests
Example of the new fallback path: