Skip to content

[codex] Record exec-server lifecycle metrics - #27467

Merged
richardopenai merged 18 commits into
mainfrom
codex/exec-server-connection-metrics
Jun 25, 2026
Merged

[codex] Record exec-server lifecycle metrics#27467
richardopenai merged 18 commits into
mainfrom
codex/exec-server-connection-metrics

Conversation

@richardopenai

@richardopenai richardopenai commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Record bounded connection, request, and process lifecycle metrics.
  • Report active gauges from callbacks on every collection, including delta exports.
  • Serialize active-count updates so concurrent starts and finishes cannot publish stale values.
  • Serialize process exit, explicit termination, and shutdown through the process registry so exactly one completion result wins.
  • Keep the implementation small with single-owner RAII guards and one real OTLP/HTTP integration test using the existing wiremock dependency.

Root cause

Process exit and session shutdown previously used cloned completion state. That avoided duplicate emission, but it duplicated lifecycle ownership and made the ordering harder to reason about. The process registry mutex already defines the lifecycle ordering, so the final implementation stores the metric guard and termination flag directly on the process entry. Whichever path claims the entry first owns the completion result.

Production metric export uses delta temporality. Event-only synchronous gauge recordings disappear after the next collection when no count changes, so active counts now use observable callbacks that report current state on every collection.

The cleanup also removes the constant result="accepted" connection tag, redundant route and response assertions, a custom HTTP collector, and fallback initialization machinery that did not add behavior.

Stack

Review and land this stack in order:

  1. [codex] Trace exec-server JSON-RPC requests #27466 — trace exec-server JSON-RPC requests
  2. [codex] Record exec-server lifecycle metrics #27467 — record bounded connection, request, and process lifecycle metrics (this PR)
  3. [codex] Observe remote exec-server lifecycle #27470 — observe remote registration and Noise rendezvous lifecycle

Validation

  • just test -p codex-exec-server --lib (158 passed)
  • just test -p codex-cli --test exec_server (3 passed)
  • just test -p codex-otel observable_gauge_is_collected_on_every_delta_snapshot (1 passed)
  • CARGO_BUILD_JOBS=1 just fix -p codex-otel -p codex-exec-server
  • just fmt
  • git diff --check

@richardopenai
richardopenai marked this pull request as ready for review June 10, 2026 21:12
@chatgpt-codex-connector

Copy link
Copy Markdown
Contributor

Review source: Codex Cloud Agents (CCA)

Codex Cloud Agents (CCA) couldn't complete this review. The original Codex Review is unaffected.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d6b943d92c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codex-rs/exec-server/src/telemetry.rs Outdated
@richardopenai
richardopenai force-pushed the codex/exec-server-connection-metrics branch from d6b943d to 7a28c0a Compare June 16, 2026 19:21
@richardopenai
richardopenai force-pushed the codex/exec-server-rpc-tracing branch from e06c4e5 to f3958b2 Compare June 16, 2026 19:21
@richardopenai richardopenai changed the title [codex] Record exec-server connection metrics [codex] Record exec-server lifecycle metrics Jun 16, 2026
@richardopenai
richardopenai force-pushed the codex/exec-server-rpc-tracing branch from f3958b2 to 1ef0691 Compare June 18, 2026 18:09
@richardopenai
richardopenai force-pushed the codex/exec-server-connection-metrics branch 2 times, most recently from 5a94cf1 to 1128e27 Compare June 18, 2026 22:39
@richardopenai
richardopenai force-pushed the codex/exec-server-rpc-tracing branch from 1ef0691 to eb4572e Compare June 19, 2026 00:15
@richardopenai
richardopenai force-pushed the codex/exec-server-connection-metrics branch from 1128e27 to 2d7885f Compare June 19, 2026 00:15
Comment thread codex-rs/exec-server/src/local_process.rs
Comment thread codex-rs/exec-server/src/telemetry.rs Outdated
Comment thread codex-rs/exec-server/src/telemetry_tests.rs Outdated
@richardopenai
richardopenai force-pushed the codex/exec-server-rpc-tracing branch from eb0a36c to a37f8f3 Compare June 24, 2026 04:40
richardopenai added a commit that referenced this pull request Jun 24, 2026
## Why

Exec-server JSON-RPC calls can cross local and remote transports, but
trace context stopped at the RPC boundary. That made client and server
work difficult to correlate when diagnosing latency or failures.

## What changed

- Propagate the current W3C trace context on outbound JSON-RPC requests.
- Parent inbound request spans from received trace context.
- Record the received JSON-RPC method on server spans and keep each span
open through response enqueue.
- Add only the OTEL dependencies required by the exec-server crate.

## Stack

Review and land this stack in order:

1. #27466 — trace exec-server JSON-RPC requests **(this PR)**
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle

## Validation

- `just test -p codex-exec-server --lib` (153 passed)
- `just bazel-lock-check`
- `just fix -p codex-exec-server`
Base automatically changed from codex/exec-server-rpc-tracing to main June 24, 2026 19:50
@richardopenai
richardopenai merged commit 2dec46e into main Jun 25, 2026
31 checks passed
@richardopenai
richardopenai deleted the codex/exec-server-connection-metrics branch June 25, 2026 18:02
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 25, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants