bench: add e2e benchmark entrypoints - #31428
Merged
Merged
Conversation
This was referenced Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 17:33
38360b6 to
726589b
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
from
July 7, 2026 17:33
40d035e to
f8fd71a
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 17:35
726589b to
e6f4113
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
from
July 7, 2026 17:36
f8fd71a to
f9458bb
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 18:18
e6f4113 to
ecf0c9c
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
from
July 7, 2026 18:19
f9458bb to
7777895
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
## Why Remote-executor integration tests need one host-agnostic exec-server fixture target instead of a Windows-only wrapper. ## What - rename the testing binary target to exec-server - make the fixture source and target host-agnostic - update Windows remote-executor test wiring to use the shared target ## Validation - bazel build //codex-rs/exec-server/testing:exec-server - bazel cquery --config=ci-windows-cross 'set(//codex-rs/exec-server/testing:exec-server //codex-rs/core/tests/remote_env_windows:smoke-test)' ## Stack 1. [#31422 test: generalize exec-server fixture](#31422) 2. [#31425 test: add TestAppServer builder](#31425) 3. [#31427 test: add delayed exec-server transport](#31427) 4. [#31295 bench: add cold skill load macrobenchmark](#31295) 5. [#31428 bench: add e2e benchmark entrypoints](#31428) 6. [#31429 ci: smoke Bazel e2e benchmarks](#31429)
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 18:38
ecf0c9c to
8ac3c16
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
from
July 7, 2026 18:39
7777895 to
67e8bf0
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 19:05
8ac3c16 to
0079a34
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
from
July 7, 2026 19:05
67e8bf0 to
b860399
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
## Why Test callers need one composable way to create app-server fixtures instead of a growing family of overlapping constructor implementations. ## What - add a feature-complete TestAppServer::builder() - make the default builder own a temporary CODEX_HOME and select the automatic test environment - expose builder knobs for no automatic environment, explicit CODEX_HOME, program, arguments, plugin startup tasks, environment overrides, managed config, and JSON logging - keep the existing public constructor surface, but route every constructor through the builder so the new path is exercised immediately - remove the redundant private constructor ladders; caller migration and public constructor removal live in the optional cleanup stack ## Validation - just test -p codex-app-server (940/941 before updating the expected builder error wording) - just test -p codex-app-server auto_env_rejects_explicit_environment_config - just fix -p codex-app-server - just fmt ## Follow-up stacks Cleanup, optional for the benchmark work: 1. [#31451 test: migrate TestAppServer callers to builder](#31451) 2. [#31452 test: remove TestAppServer constructors](#31452) Benchmark infrastructure: 1. [#31427 test: add delayed exec-server transport](#31427) 2. [#31295 bench: add cold skill load macrobenchmark](#31295) 3. [#31428 bench: add e2e benchmark entrypoints](#31428) 4. [#31429 ci: smoke Bazel e2e benchmarks](#31429)
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 22:55
0079a34 to
ce11104
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
2 times, most recently
from
July 8, 2026 00:21
29179d5 to
e5b20f0
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 9, 2026
## Why Macrobenchmarks benefit from having a way to exercise remote-executor latency without depending on Docker. This is a very minimal first cut, if we find that simulating network conditions is useful we can always expand this scope or switch to a more robust network shaping approach. ## What - add a package-local exec-server binary for Cargo and Bazel test fixtures - add a host-local WebSocket exec-server fixture and fixed-delay interposer - let TestAppServer route its auto environment through that delayed WebSocket transport - cover the delayed thread/start path through the public app-server API ## Stack 1. [#31425 test: add TestAppServer builder](#31425) 2. [#31427 test: add delayed exec-server transport](#31427) 3. [#31295 bench: add cold skill load macrobenchmark](#31295) 4. [#31428 bench: add e2e benchmark entrypoints](#31428) 5. [#31429 ci: smoke Bazel e2e benchmarks](#31429)
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 9, 2026 00:19
ce11104 to
310f235
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
2 times, most recently
from
July 9, 2026 00:27
0a40f91 to
3b35b0b
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 9, 2026 02:21
7011da4 to
37b2d6d
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
from
July 9, 2026 02:22
3b35b0b to
abd411b
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 9, 2026 02:46
37b2d6d to
95a2d1f
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
from
July 9, 2026 02:47
abd411b to
4342c70
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 9, 2026 04:00
95a2d1f to
b24cbd9
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
from
July 9, 2026 04:00
4342c70 to
8f4d31c
Compare
anp-oai
marked this pull request as ready for review
July 9, 2026 04:31
Contributor
Author
|
@codex review |
Contributor
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8f4d31ced6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
2 times, most recently
from
July 9, 2026 04:52
cf5b1b0 to
572f841
Compare
cconger
approved these changes
Jul 9, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
2 times, most recently
from
July 10, 2026 00:08
8510d0c to
2b339d6
Compare
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
from
July 10, 2026 00:08
572f841 to
f97c74f
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 10, 2026
## Why Bazel-backed end-to-end macrobenchmark plumbing needs a small, deterministic first consumer that does not couple the shared infrastructure to the remote-skill scenario. ## What - add `codex_e2e_benchmark`, a small macro for Bazel-only Divan benchmarks and runtime binary runfiles - keep benchmark sources under `e2e_benches/` so Cargo does not auto-discover them - add a CLI example that resolves the real `codex` binary and measures `codex --help` - assert the spawned command succeeds ## Validation - `bazel test --compilation_mode=fastbuild --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=no --@rules_rust//rust/settings:extra_exec_rustc_flag=-Cdebug-assertions=no --cache_test_results=no --test_output=errors --test_arg=--test //codex-rs/cli:codex-help-bench` ## Stack 1. [#31295 bench: add codex help e2e macrobenchmark](#31295) 2. [#31428 bench: add e2e benchmark entrypoints](#31428) 3. [#31429 ci: smoke Bazel e2e benchmarks](#31429)
anp-oai
force-pushed
the
codex/e2e-benchmark-entrypoints
branch
from
July 10, 2026 00:24
f97c74f to
eae9cc3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Bazel-backed macrobenchmarks need stable local entrypoints that can grow without changing every caller.
What
e2e-benchmarksBazel suite, initially containing thecodex --helpbenchmarkjust bench-e2efor direct optimized Bazel measurementsjust bench-e2e-smokefor one direct fastbuild iteration through release-only Rust cfg pathsValidation
just --dry-run bench-e2ejust --dry-run bench-e2e-smokejust bench-e2e-smokeStack