docs: document and stabilize token/cost field semantics - #330
Conversation
Add a canonical Token and Cost Field Semantics section to the provider response codecs page: a Usage/CostEstimate field reference, the per-provider token normalization table, granularity (per-call values vs the per-trajectory final_metrics aggregate), an exporter field-mapping table (ATOF/ATIF/OpenInference/OpenTelemetry), and a stability contract. Add brief field pointers and back-links on the OpenTelemetry, OpenInference, and ATIF exporter pages. Lock the contract with characterization tests: the OpenTelemetry LLM span emits cost only (no token attributes), Usage ignores unmodeled provider subfields, and OpenAIChatCodec drops completion_tokens_details. No runtime behavior change. Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
WalkthroughAdds codec and observability tests, and expands documentation with canonical token/cost semantics plus per-exporter field mappings. ChangesToken and Cost Field Semantics
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
willkill07
left a comment
There was a problem hiding this comment.
Approving from dev perspective
lvojtku
left a comment
There was a problem hiding this comment.
Left questions and comments.
- Style: "See" -> "Refer to"; title-case the Usage Fields, Cost Fields, and Exporter Field Mapping headings; capitalize the Granularity bullet leads. - Exporter mapping: drop the dangling "(see note)" on the total-tokens row (the following paragraph already explains it); change the currency clause to ", while". - Stability: name NeMo Relay as the actor for the additive-compatibility promise, move the exhaustive-Rust-structs caveat into a Note, and convert the intentional-behavior run-on into a bulleted list. Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/integrate-into-frameworks/provider-response-codecs.mdx (1)
406-425: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMinor: inconsistent ATIF version string formatting.
Line 409 uses
ATIF-v1.7(hyphenated), whiledocs/observability-plugin/atif.mdxrefers to the same schema asATIF v1.7(space-separated). Align the formatting for consistency across docs.📝 Proposed fix
-mappings above, are stable as of ATOF `0.1` (ATIF schema `ATIF-v1.7`, pricing +mappings above, are stable as of ATOF `0.1` (ATIF schema `ATIF v1.7`, pricingAs per path instructions, "Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior" and "Review documentation for technical accuracy against the current API... and consistency across language bindings."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/integrate-into-frameworks/provider-response-codecs.mdx` around lines 406 - 425, The stability note in the documentation uses an inconsistent ATIF schema version format; update the wording in this section to match the same “ATIF v1.7” style used elsewhere in the docs. Keep the existing meaning intact, and only normalize the version string in the Stability paragraph so the `Usage`/`CostEstimate` stability statement is consistent across documentation.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docs/integrate-into-frameworks/provider-response-codecs.mdx`:
- Around line 406-425: The stability note in the documentation uses an
inconsistent ATIF schema version format; update the wording in this section to
match the same “ATIF v1.7” style used elsewhere in the docs. Keep the existing
meaning intact, and only normalize the version string in the Stability paragraph
so the `Usage`/`CostEstimate` stability statement is consistent across
documentation.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: e1ed09b0-3050-4e05-a7f6-4fb131f25805
📒 Files selected for processing (4)
docs/integrate-into-frameworks/provider-response-codecs.mdxdocs/observability-plugin/atif.mdxdocs/observability-plugin/openinference.mdxdocs/observability-plugin/opentelemetry.mdx
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (8)
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
**/*.mdx: In MDX files, top-of-file comments must use JSX comment delimiters:{/*to open and*/}to close; do not use HTML comments for MDX SPDX headers.
New or regenerated MDX files must use{/* ... */}for top-of-file SPDX comments.
Files:
docs/observability-plugin/opentelemetry.mdxdocs/observability-plugin/openinference.mdxdocs/observability-plugin/atif.mdxdocs/integrate-into-frameworks/provider-response-codecs.mdx
**/*
📄 CodeRabbit inference engine (AGENTS.md)
Keep SPDX headers on source, docs, scripts, and configuration files.
Files:
docs/observability-plugin/opentelemetry.mdxdocs/observability-plugin/openinference.mdxdocs/observability-plugin/atif.mdxdocs/integrate-into-frameworks/provider-response-codecs.mdx
docs/**/*.{md,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
docs/**/*.{md,mdx}: Keep documentation underdocs/valid for Fern tooling and external links, including Markdown/MDX link checks where applicable.
Update relevant reference docs and embedded examples when public behavior, bindings, examples, or workspace structure change.
Files:
docs/observability-plugin/opentelemetry.mdxdocs/observability-plugin/openinference.mdxdocs/observability-plugin/atif.mdxdocs/integrate-into-frameworks/provider-response-codecs.mdx
**/*.{rs,go,js,ts,py,html,md,mdx,toml,yml,yaml,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Include SPDX license headers in all source files using the correct comment syntax for the file type.
Files:
docs/observability-plugin/opentelemetry.mdxdocs/observability-plugin/openinference.mdxdocs/observability-plugin/atif.mdxdocs/integrate-into-frameworks/provider-response-codecs.mdx
**/*.{md,mdx}
📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md)
**/*.{md,mdx}: Prefer the documented public API rather than internal shortcuts.
Keep package names, repository references, and build commands current.
Update entry-point docs when examples or reading paths change.
Keep release-process and release-notes guidance in repository-maintainer docs such asRELEASING.md, not in user-facing docs pages orCHANGELOG.md.
Keep stable user-facing wrappers at thescripts/root in docs and examples; only point at namespaced helper paths when documenting internal maintenance work.
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pages.
Dynamic plugin manifests in docs and examples should usecompat.relay = ">=0.5,<1.0"unless deliberately narrower.
Update relevant getting-started or reference docs when the associated behavior or examples change.
Ensure example commands still match current package names and paths.
Dynamic plugin entry pages should link to native, worker, Rust example, Python example, and protocol pages when those pages exist.
When the docs site changes, runjust docs;./scripts/build-docs.sh htmlremains the compatibility wrapper.
**/*.{md,mdx}: Document native and worker plugins as trusted extensions: native plugins are in-process and unsandboxed, while worker plugins provide process isolation but not a security sandbox.
When detailed dynamic plugin guides exist, keep Rust native, Python worker, andgrpc-v1protocol details on separate pages.
Files:
docs/observability-plugin/opentelemetry.mdxdocs/observability-plugin/openinference.mdxdocs/observability-plugin/atif.mdxdocs/integrate-into-frameworks/provider-response-codecs.mdx
docs/**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/contribute-integration/SKILL.md)
Update documentation if activation or usage changes
Files:
docs/observability-plugin/opentelemetry.mdxdocs/observability-plugin/openinference.mdxdocs/observability-plugin/atif.mdxdocs/integrate-into-frameworks/provider-response-codecs.mdx
**
⚙️ CodeRabbit configuration file
**:AGENTS.md
This file provides guidance to agents, including Claude Code and OpenAI Codex, when working in this repository.
Project Overview
NeMo Relay is a multi-language agent runtime framework for execution scopes, lifecycle events, middleware, plugins, and observability around tool and LLM calls. The core runtime is Rust. Primary supported bindings are Rust, Python, and Node.js. Go, WebAssembly, and the raw C FFI are experimental and source-first.
The shared runtime model is:
- Scope stacks decide where work belongs and which scope-local behavior is visible.
- Middleware registries decide what guardrails and intercepts run around managed calls.
- Plugins install reusable runtime behavior from configuration.
- Events record runtime behavior in ATOF form.
- Subscribers and exporters consume events in-process or export them to ATIF, OpenTelemetry, OpenInference, or other backends.
Repository Structure
The repository layout separates the Rust runtime, language bindings, documentation,
integration patches, and agent-facing skills.crates/ core/ # Rust core runtime crate, published as nemo-relay adaptive/ # Adaptive runtime primitives and plugin components python/ # PyO3 native extension for the Python package ffi/ # Raw C ABI layer used by downstream bindings such as Go node/ # NAPI Node.js binding and JavaScript/TypeScript entry points wasm/ # wasm-bindgen WebAssembly binding and JS wrappers python/ nemo_relay/ # Python wrapper package: scopes, tools, LLM, middleware, typed helpers, plugins, adaptive helpers tests/ # Python tests go/ nemo_relay/ # Experimental Go CGo binding and tests fern/ # Fern documentation site scripts/ # Stable wrappers and helper scripts; build/test/docs entry points live in justfile third_party/ # P...
Files:
docs/observability-plugin/opentelemetry.mdxdocs/observability-plugin/openinference.mdxdocs/observability-plugin/atif.mdxdocs/integrate-into-frameworks/provider-response-codecs.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/observability-plugin/opentelemetry.mdxdocs/observability-plugin/openinference.mdxdocs/observability-plugin/atif.mdxdocs/integrate-into-frameworks/provider-response-codecs.mdx
🔇 Additional comments (4)
docs/integrate-into-frameworks/provider-response-codecs.mdx (1)
311-405: LGTM!docs/observability-plugin/atif.mdx (1)
202-208: LGTM!docs/observability-plugin/openinference.mdx (1)
90-95: LGTM!docs/observability-plugin/opentelemetry.mdx (1)
76-81: LGTM!
#### Overview Backfill regression and parity coverage for the shared extraction seams established by #291, #300, #301, #304, and #318, so agent-payload extraction, provider-schema normalization, and exporter projection cannot drift silently. Test-only: the only `src` edits are `#[cfg(test)]` module wiring for two new suites; no production behavior changes. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details **Agent payload extraction** (`crates/cli/tests/coverage/adapters_tests.rs`, 15 tests): - Per-host partial-sparse payloads for Claude Code, Codex, and Hermes (real identifiers kept, absent fields synthesized or null at the adapter boundary). - Path-precedence fallback-chain walks for session IDs, event names, subagent IDs, tool-call IDs, and tool results/status. - Hermes tool result and status extraction (previously uncovered), including explicit-status vs event-name-derived status interaction. - Claude Code LLM-hint extraction and hint field precedence chains. - JSON-path primitive edge cases: empty-string filtering, array intermediates, deep nesting, all-paths-miss. **Provider request extraction** (`crates/cli/tests/coverage/alignment_tests.rs`, 6 tests): - First coverage for `request_affinity_key` (route-gated, length-bounded) and `gateway_turn_input` (Claude Code + Anthropic Messages only). - Header-only session-id routing on the Models and Count-Tokens routes, contrasted with the body fallbacks on the Responses/Chat routes. - Gateway route name round-trips. **Codec parity** (`crates/core/tests/unit/codec/parity_tests.rs`, new, 14 tests): Each test builds the same logical scenario in all three provider schemas (OpenAI Chat Completions, Anthropic Messages, OpenAI Responses) and asserts the normalized output agrees: model name, finish reasons, tool calls (full-struct equality), usage incl. cache-read tokens, provider-reported and catalog-estimated cost, hint hardening (`normalize_request_with_hint`), and request normalization. Schema-inherent divergences are asserted explicitly as part of the parity contract, e.g. Responses normalizes a tool-call turn to `Complete` (no tool-use terminal status), cache-write tokens are Anthropic-only, reasoning tokens are Responses-only. **Exporter parity** (`crates/core/tests/unit/observability/exporter_parity_tests.rs`, new, 9 tests): A shared harness feeds one event stream to the ATIF exporter, `OpenTelemetrySubscriber`, and `OpenInferenceSubscriber` (in-memory span exporters) and asserts the projected facts agree: cost totals, usage, model names, tool-call projection, replay-payload preservation, and the consolidated manual-fallback path — run against both OpenAI-Chat-shaped and Anthropic-shaped payloads. Intentional projection divergences are pinned with explicit assertions so drift fails loudly, notably: OTel emits no token-count attributes, reasoning facts are ATIF-only, ATIF sums cache read+write while OpenInference splits them, and OpenInference omits raw LLM request JSON. Cost-currency policy boundaries stay covered by the existing per-exporter tests from #304 and are not duplicated here. **Validation** - `just test-rust` — full workspace green - `cargo clippy --workspace --all-targets -- -D warnings` — clean - `cargo fmt --all --check` — clean - `uv run pre-commit run --files <changed files>` — all hooks green (SPDX, fmt, clippy, check, linkcheck) - Rebased onto current `main` and re-ran `cargo test -p nemo-relay` and `cargo test -p nemo-relay-cli` — green, including the parity suites against the latest observability changes - Binding matrix not run: the change is test-only; the `#[cfg(test)]` modules are not compiled into the library targets, so binding-facing behavior is unchanged **Breaking changes** None. #### Where should the reviewer start? `crates/core/tests/unit/observability/exporter_parity_tests.rs` — the `export_through_all_exporters` harness and the explicitly pinned divergences. These assertions encode current behavior. The OTel cost-only projection and the ATIF cache-sum vs OpenInference cache-split asymmetry match the token/cost field-semantics contract documented in #330; the remaining pinned divergences (reasoning projection being ATIF-only, OTel not flattening LLM tool calls, OpenInference omitting raw LLM request JSON) encode current behavior without a recorded decision — if any is ruled a defect, the fix should land together with flipping the pinning assertion. Then `crates/core/tests/unit/codec/parity_tests.rs` for the cross-schema normalization contract, and the adapter/alignment additions in `crates/cli/tests/coverage/`. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to: none (test backfill for the extraction-strategy refactors #291, #300, #301, #304, #318) ## Summary by CodeRabbit * **Tests** * Added coverage for Hermes and Claude tool adapter/extractor precedence, partial payload handling, and hint/session/model/request-id resolution (including null/empty-string and fallback-chain behaviors). * Expanded gateway alignment tests for affinity-key/session-id gating and route/provider-specific prompt building. * Introduced core codec and observability cross-provider/exporter parity tests validating normalized fields, tool-call mapping, and consistent usage/cost/model/replay projections (with expected divergence checks). Authors: - Yuchen Zhang (https://github.com/yczhang-nv) Approvers: - Will Killian (https://github.com/willkill07) - Zhongxuan (Daniel) Wang (https://github.com/zhongxuanwang-nv) - Maryam Najafian (https://github.com/mnajafian-nv) URL: #355
c75bdfb to
cde9770
Compare
Overview
Document and stabilize NeMo Relay's LLM token and cost field semantics. This freezes the current behavior as a documented contract and locks it with characterization tests. There is no runtime behavior change. The branch is merged with current
main, so the docs and tests reflect #304 (prefer normalized exporter fallbacks) and #305 (sharednemo_relay_typesDTO crate).Details
Adds a canonical Token and Cost Field Semantics section to
docs/integrate-into-frameworks/provider-response-codecs.mdx:UsageandCostEstimatefield reference (names, units, optionality; optional fields can also be codec-computed or pricing-synthesized).Usage).userstart step (no metrics) and anagentend step that carries metrics;final_metricsis the only aggregate (a per-trajectory sum of present step metrics, excluding embedded subagent trajectories).0.1/ATIF-v1.7/ pricing catalogversion: 1). The additive-compatibility promise is scoped to the serialized JSON/ATOF shapes; the RustUsage/CostEstimatestructs andCostSourceenum are exhaustive, so adding a field/variant is source-breaking for Rust consumers.Short field pointers + back-links were added to the OpenTelemetry, OpenInference, and ATIF exporter pages. Cost policy is stated once on the canonical page, per the runtime-contract docs convention.
Two characterization tests lock the freeze:
OtelEventProcessor, asserting the finished span carriesnemo_relay.llm.cost.{total,currency}and no token-count attribute in any namespace (token-containing orgen_ai.*keys fail).OpenAIChatCodecdropscompletion_tokens_details, andUsageignores unmodeled provider subfields (forward-compat: no serde catch-all).Existing tests already cover the remaining projections, per-provider mapping, reasoning-tokens-in-
api_specific, and the USD-only/currency-aware cost behavior.Testing: targeted
cargo teston the merged tree (the new/changed tests pass; perturbing the OTel exporter with anemo_relay.prompt_tokensattribute makes the span test fail as intended, then reverted),just docs-linkcheck(0 errors), andpre-commit(SPDX, markdown linkcheck, cargo fmt/clippy/check) all pass.Where should the reviewer start?
docs/integrate-into-frameworks/provider-response-codecs.mdx— the Token and Cost Field Semantics section (the exporter field-mapping table and the Stability subsection are the core contract). Thencrates/core/tests/unit/observability/otel_tests.rs::llm_end_emits_cost_only_no_token_or_gen_ai_attributes.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
usagetoken totals while omitting provider-only completion token detail fields.Usageserialization now ignores unknown/unmodeled provider subfields.gen_ai*attributes).usagefields, and cost-only observability attributes.