fix: resolve bare OTLP HTTP trace endpoints - #652
Conversation
Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com>
WalkthroughThe change normalizes bare HTTP OTLP endpoints to ChangesOTLP HTTP normalization and diagnostics
Estimated code review effort: 2 (Simple) | ~15 minutes Sequence Diagram(s)sequenceDiagram
participant Diagnostics
participant EndpointResolver
participant OTLPHTTPEndpoint
Diagnostics->>EndpointResolver: resolve configured HTTP endpoint
EndpointResolver->>Diagnostics: return endpoint with /v1/traces
Diagnostics->>OTLPHTTPEndpoint: send OTLP HTTP probe
OTLPHTTPEndpoint->>Diagnostics: return HTTP status
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
bbednarski9
left a comment
There was a problem hiding this comment.
One compatibility concern.
Signed-off-by: Eric Evans <194135482+ericevans-nv@users.noreply.github.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/cli/tests/coverage/shared/doctor_tests.rs (1)
1007-1009: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAssert the actual path for the bare endpoint.
The mock server discards the request and returns
405for every path. Thechecks[0].detailsassertion only checks the URL passed to the probe. It does not prove that the diagnostic sentGET /v1/traces. Returnread_headersfrom this thread and assert the request line, as the explicit-root case does.Proposed test assertion
- let _ = read_headers(&mut stream); + let request = read_headers(&mut stream); stream .write_all(b"HTTP/1.1 405 Method Not Allowed\r\nContent-Length: 0\r\n\r\n") .unwrap(); + request - accept.join().unwrap(); + let request = accept.join().unwrap(); + assert!(request.starts_with("GET /v1/traces HTTP/1.1"));As per path instructions, tests should cover the behavior promised by the changed API surface.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/cli/tests/coverage/shared/doctor_tests.rs` around lines 1007 - 1009, Update the mock server thread in the bare-endpoint test to capture and return the result of read_headers, then assert the captured request line is GET /v1/traces, matching the explicit-root test. Keep the existing 405 response and checks[0].details assertion.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@crates/cli/tests/coverage/shared/doctor_tests.rs`:
- Around line 1007-1009: Update the mock server thread in the bare-endpoint test
to capture and return the result of read_headers, then assert the captured
request line is GET /v1/traces, matching the explicit-root test. Keep the
existing 405 response and checks[0].details assertion.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: f46acb72-d669-4234-b39a-4e7a2142778e
📒 Files selected for processing (4)
crates/cli/tests/coverage/shared/doctor_tests.rscrates/core/src/observability/otel.rscrates/core/tests/unit/observability/otel_tests.rsdocs/configure-plugins/observability/opentelemetry.mdx
📜 Review details
🧰 Additional context used
📓 Path-based instructions (24)
**/*.rs
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
**/*.rs: Any Rust change must runjust test-rust
Any Rust change must runcargo fmt --all
Any Rust change must runcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allfor all FFI work since it is Rust work
Runjust test-rustto validate FFI changes
Runcargo clippy --workspace --all-targets -- -D warningsto enforce strict linting on FFI workWhen Rust files changed as part of Go work, also run
cargo fmt --all,just test-rust, andcargo clippy --workspace --all-targets -- -D warnings
**/*.rs: Runcargo fmt --allwhen Rust files are changed as part of Node work
Runcargo clippy --workspace --all-targets -- -D warningswhen Rust files are changed as part of Node work
Runjust test-rustwhen Rust files are changed as part of Node workWhen changing the core Rust runtime or Rust-facing API surface, format Rust code with
cargo fmt(rustfmt defaults), keepcargo clippy -- -D warningsclean, and satisfycargo deny checkperdeny.toml.
**/*.rs: If any Rust code changed, always runjust test-rust.
If any Rust code changed, also runcargo fmt --all.
If any Rust code changed, also runcargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, runcargo fmt --allandcargo clippy --workspace --all-targets -- -D warningseven if relying on pre-commit.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rscrates/cli/tests/coverage/shared/doctor_tests.rs
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rs
crates/core/**/*.rs
📄 CodeRabbit inference engine (.agents/skills/test-go-binding/SKILL.md)
If the change touched
crates/coreor shared runtime semantics, also usevalidate-changefor broader validation
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rs
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rscrates/cli/tests/coverage/shared/doctor_tests.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rscrates/cli/tests/coverage/shared/doctor_tests.rs
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rscrates/cli/tests/coverage/shared/doctor_tests.rs
**/*.{rs,go,js,ts}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding
//comment form.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rscrates/cli/tests/coverage/shared/doctor_tests.rs
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
crates/core/tests/unit/observability/otel_tests.rsdocs/configure-plugins/observability/opentelemetry.mdxcrates/core/src/observability/otel.rscrates/cli/tests/coverage/shared/doctor_tests.rs
crates/{core,adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full validation matrix across Rust, Python, Go, and Node.js.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rs
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
**/*.{rs,py,go,js,ts}: When observability configuration or lifecycle is exposed, keep FFI and Python, Go, and Node.js binding-native config objects and subscriber/exporter methods aligned in logical knobs and semantics.
Require every OpenTelemetry endpoint to have a type and nonblank destination; resolveheader_envvalues at activation and reject missing, blank, or duplicate headers.
Concatenate layered ATOF sink, ATIF storage, and OpenTelemetry endpoint lists with higher-precedence entries first.
Preserve correct handling of mark events, start/end events, orphan cases, and span or trajectory fields derived from intended event data.
Run affected Rust tests andjust test-rustwhen event fields change; runjust test-python,just test-go, andjust test-nodewhen binding-native configuration or lifecycle changes.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rscrates/cli/tests/coverage/shared/doctor_tests.rs
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Add tests covering registration and duplicate names, deregistration and missing names, priority ordering, callback failure policy, scope-local inheritance and cleanup, event payload semantics, immutable mark and scope fields, and parity across affected bindings.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rscrates/cli/tests/coverage/shared/doctor_tests.rs
crates/{core,adaptive}/**/*.rs
⚙️ CodeRabbit configuration file
crates/{core,adaptive}/**/*.rs: Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.
Pay close attention to task-local/thread-local scope propagation, callback lifetimes, stream finalization, and root_uuid isolation.
Public API changes should preserve existing behavior unless tests and docs show the intended migration path.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/core/src/observability/otel.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
crates/core/tests/unit/observability/otel_tests.rscrates/cli/tests/coverage/shared/doctor_tests.rs
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
In MDX files, top-of-file comments must use JSX comment delimiters (
{/*to open and*/}to close); do not use HTML comments for MDX SPDX headers
Files:
docs/configure-plugins/observability/opentelemetry.mdx
**/*.{md,mdx}
📄 CodeRabbit inference engine (AGENTS.md)
Update
README.md,fern/, package READMEs, and binding-support notes when public behavior, package names, examples, or supported bindings change.
**/*.{md,mdx}: Prefer the documented public API, not internal shortcuts
Keep package names, repo references, and build commands current
Keep release-process and release-notes guidance in repo-maintainer docs such asRELEASING.md, not as user-facing docs pages orCHANGELOG.md
Keep stable user-facing wrappers atscripts/root in docs and examples; only point at namespaced helper paths when documenting internal maintenance work
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pagesIf links in documentation change, run
just docs-linkcheck.
Files:
docs/configure-plugins/observability/opentelemetry.mdx
**/*.{md,markdown,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Markdown/MDX documentation files using the HTML comment block form.
Files:
docs/configure-plugins/observability/opentelemetry.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/configure-plugins/observability/opentelemetry.mdx
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/configure-plugins/observability/opentelemetry.mdx
**/*.{md,mdx,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Examples and documentation must use each exporter's documented flush/deregister order before shutdown.
Files:
docs/configure-plugins/observability/opentelemetry.mdx
docs/{about-nemo-relay/concepts/subscribers.mdx,configure-plugins/observability/**/*.mdx}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Update observability documentation and examples alongside implementation changes, including configuration version 3 with one
opentelemetrysection containing typed endpoints and no standalone public OpenInference surface.
Files:
docs/configure-plugins/observability/opentelemetry.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/configure-plugins/observability/opentelemetry.mdx
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
crates/core/src/observability/otel.rs
**/crates/core/src/observability/{atif,otel,openinference}.rs
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Keep ATIF, OpenTelemetry, and OpenInference observability projections synchronized with the core event model and emitted fields.
Files:
crates/core/src/observability/otel.rs
**/crates/core/src/observability/otel.rs
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
**/crates/core/src/observability/otel.rs: Thegen_aiOpenTelemetry projection must omitnemo_relay.*and content-capture attributes while preserving descendants through omitted scopes.
Derive compliant trace and span IDs consistently across typed OpenTelemetry endpoints while preserving lifecycle parentage.
Files:
crates/core/src/observability/otel.rs
🧠 Learnings (2)
📚 Learning: 2026-07-14T02:53:59.997Z
Learnt from: willkill07
Repo: NVIDIA/NeMo-Relay PR: 415
File: docs/configure-plugins/observability/opentelemetry.mdx:98-113
Timestamp: 2026-07-14T02:53:59.997Z
Learning: In NeMo-Relay’s OpenTelemetry/OpenInference observability projection docs under docs/configure-plugins/observability/, document the projected-attribute contract as follows: (1) emit scalar top-level `data`/`metadata` fields as typed dotted OTLP attributes (for example, `nemo_relay.start.metadata.tenant`); (2) keep nested objects/arrays as JSON strings at their top-level OTLP attribute (rather than expanding them into nested OTLP attributes); and (3) do not reference the legacy `*_json` payload attributes (e.g., `data_json`, `metadata_json`, `input_json`) because they were intentionally removed as a breaking change.
Applied to files:
docs/configure-plugins/observability/opentelemetry.mdx
📚 Learning: 2026-07-28T20:07:29.880Z
Learnt from: willkill07
Repo: NVIDIA/NeMo-Relay PR: 571
File: crates/core/src/api/runtime/state.rs:996-1020
Timestamp: 2026-07-28T20:07:29.880Z
Learning: In NeMo Relay (RELAY-509), sanitizer callback failures must be treated as intentional fail-open behavior. When an event/tool (request/response) or LLM (request/response) sanitizer callback fails, the sanitizer chain should retain and publish the last valid event/payload snapshot (rather than dropping/invalidating the data) and log the failure including callback context (e.g., which sanitizer/callback failed and relevant identifiers). Apply this consistently across all sanitizer chains mentioned in the RELAY-509 documentation/migration guide.
Applied to files:
crates/core/src/observability/otel.rs
🔇 Additional comments (5)
crates/core/src/observability/otel.rs (2)
17-17: LGTM!Also applies to: 159-171
706-707: 📐 Maintainability & Code QualityConfirm the required validation before merge.
This change modifies shared Rust observability behavior in
crates/core. Confirm that these commands pass:
just test-rustcargo fmt --allcargo clippy --workspace --all-targets -- -D warningsvalidate-changejust test-pythonjust test-gojust test-nodeuv run pre-commit run --all-filesAs per coding guidelines, Rust changes require formatting, tests, linting, and the final pre-commit pass. As per path instructions, changes under
crates/corealso requirevalidate-changeand the full language matrix.Sources: Coding guidelines, Path instructions
crates/core/tests/unit/observability/otel_tests.rs (1)
699-707: LGTM!Also applies to: 1764-1764
docs/configure-plugins/observability/opentelemetry.mdx (1)
80-80: LGTM!crates/cli/tests/coverage/shared/doctor_tests.rs (1)
983-983: LGTM!Also applies to: 1024-1045, 1046-1063, 1666-1666, 1678-1685, 1696-1714
|
/merge |
Overview
Complete bare OTLP/HTTP endpoints with the standard
/v1/tracessignal path and makedoctorreport the effective trace destination consistently with the exporter. Explicit HTTP paths and all gRPC endpoints remain unchanged.Details
httpandhttpsOTLP endpoints to/v1/traceswhile preserving query parameters and explicit paths.doctordiagnostics.Validation:
doctortests passed.cargo clippy --workspace --all-targets -- -D warningspassed.just docspassed with the existing FDR 403 redirect-check warning.just test-rustreached 1,066 passing tests and 86 failures. The first failure is an untouched pricing test using unsupported observability config version 2; the remaining failures are primarily poisoned-state cascades. No focused test for this change failed.Where should the reviewer start?
Start with
resolve_http_trace_endpointincrates/core/src/observability/otel.rs, then review the matchingdoctorbehavior incrates/cli/src/diagnostics/mod.rsand its regression tests.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
Bug Fixes
/v1/traces.Documentation