bench: add codex help e2e macrobenchmark - #31295
Merged
Merged
Conversation
anp-oai
added a commit
that referenced
this pull request
Jul 6, 2026
anp-oai
added a commit
that referenced
this pull request
Jul 6, 2026
anp-oai
added a commit
that referenced
this pull request
Jul 6, 2026
anp-oai
added a commit
that referenced
this pull request
Jul 6, 2026
anp-oai
added a commit
that referenced
this pull request
Jul 6, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 6, 2026 23:23
e357ff1 to
557b7c9
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 6, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 6, 2026 23:56
557b7c9 to
8875647
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 00:41
8875647 to
aef0c66
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 01:02
aef0c66 to
790039d
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 02:05
790039d to
9cca043
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 02:38
9cca043 to
3e6ae1a
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 03:10
3e6ae1a to
1e6a517
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 03:19
1e6a517 to
a2119d2
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 03:39
a2119d2 to
47f0e8c
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 04:09
47f0e8c to
633e020
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 04:41
633e020 to
65775a0
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 16:20
9a9b320 to
38360b6
Compare
This was referenced Jul 7, 2026
anp-oai
force-pushed
the
codex/test-app-server-rpc-delay
branch
from
July 7, 2026 17:31
a9701fb to
97c4357
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
2 times, most recently
from
July 7, 2026 17:35
726589b to
e6f4113
Compare
anp-oai
force-pushed
the
codex/test-app-server-rpc-delay
branch
from
July 7, 2026 18:17
97c4357 to
ede6cf5
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 18:18
e6f4113 to
ecf0c9c
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
## Why Remote-executor integration tests need one host-agnostic exec-server fixture target instead of a Windows-only wrapper. ## What - rename the testing binary target to exec-server - make the fixture source and target host-agnostic - update Windows remote-executor test wiring to use the shared target ## Validation - bazel build //codex-rs/exec-server/testing:exec-server - bazel cquery --config=ci-windows-cross 'set(//codex-rs/exec-server/testing:exec-server //codex-rs/core/tests/remote_env_windows:smoke-test)' ## Stack 1. [#31422 test: generalize exec-server fixture](#31422) 2. [#31425 test: add TestAppServer builder](#31425) 3. [#31427 test: add delayed exec-server transport](#31427) 4. [#31295 bench: add cold skill load macrobenchmark](#31295) 5. [#31428 bench: add e2e benchmark entrypoints](#31428) 6. [#31429 ci: smoke Bazel e2e benchmarks](#31429)
anp-oai
force-pushed
the
codex/test-app-server-rpc-delay
branch
from
July 7, 2026 18:37
ede6cf5 to
36ec068
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 18:38
ecf0c9c to
8ac3c16
Compare
anp-oai
force-pushed
the
codex/test-app-server-rpc-delay
branch
from
July 7, 2026 19:05
36ec068 to
ec6b04b
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 19:05
8ac3c16 to
0079a34
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
## Why Test callers need one composable way to create app-server fixtures instead of a growing family of overlapping constructor implementations. ## What - add a feature-complete TestAppServer::builder() - make the default builder own a temporary CODEX_HOME and select the automatic test environment - expose builder knobs for no automatic environment, explicit CODEX_HOME, program, arguments, plugin startup tasks, environment overrides, managed config, and JSON logging - keep the existing public constructor surface, but route every constructor through the builder so the new path is exercised immediately - remove the redundant private constructor ladders; caller migration and public constructor removal live in the optional cleanup stack ## Validation - just test -p codex-app-server (940/941 before updating the expected builder error wording) - just test -p codex-app-server auto_env_rejects_explicit_environment_config - just fix -p codex-app-server - just fmt ## Follow-up stacks Cleanup, optional for the benchmark work: 1. [#31451 test: migrate TestAppServer callers to builder](#31451) 2. [#31452 test: remove TestAppServer constructors](#31452) Benchmark infrastructure: 1. [#31427 test: add delayed exec-server transport](#31427) 2. [#31295 bench: add cold skill load macrobenchmark](#31295) 3. [#31428 bench: add e2e benchmark entrypoints](#31428) 4. [#31429 ci: smoke Bazel e2e benchmarks](#31429)
anp-oai
force-pushed
the
codex/test-app-server-rpc-delay
branch
from
July 7, 2026 22:55
ec6b04b to
0db9962
Compare
anp-oai
force-pushed
the
codex/durable-executor-skill-benchmark
branch
from
July 7, 2026 22:55
0079a34 to
ce11104
Compare
anp-oai
added a commit
that referenced
this pull request
Jul 7, 2026
## Why Loading skills from a remote executor can add a lot to thread start time when there are many skills. Previous changes added some concurrency for the file reads themselves, but we're still bottlenecked on the initial root path discovery. Using the benchmark from [#31295](#31295), this change reduces the measured mean of loading 66 skills about 71%. Behavioral coverage lands separately in [#31369](#31369). ## What - resolve the scanned root's inherited namespace once - resolve discovered nested plugin roots once, retaining nearest-valid-ancestor behavior - pass an explicit resolved Plain / Plugin namespace into skill parsing instead of probing per skill - preserve explicitly provided plugin namespaces as the highest-priority source - reuse the same resolver for environment skills, deleting its duplicate root-probe and ancestor-selection path ## Validation - just test -p codex-core-skills namespace: 12 passed on both the parent and optimized branches
This was referenced Jul 8, 2026
Contributor
Author
|
@codex review |
Contributor
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Contributor
Author
|
@codex review |
Contributor
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Bazel-backed end-to-end macrobenchmark plumbing needs a small, deterministic first consumer that does not couple the shared infrastructure to the remote-skill scenario.
What
codex_e2e_benchmark, a small macro for Bazel-only Divan benchmarks and runtime binary runfilese2e_benches/so Cargo does not auto-discover themcodexbinary and measurescodex --helpValidation
bazel test --compilation_mode=fastbuild --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=no --@rules_rust//rust/settings:extra_exec_rustc_flag=-Cdebug-assertions=no --cache_test_results=no --test_output=errors --test_arg=--test //codex-rs/cli:codex-help-benchStack