ci: smoke Bazel e2e benchmarks by anp-oai · Pull Request #31429 · openai/codex

anp-oai · 2026-07-07T16:23:43Z

Why

The Bazel-backed macrobenchmark path should stay runnable in CI as the suite grows.

What

run the explicit CI-only run-bazel-ci.sh -- test ... smoke invocation in the existing Linux GNU Bazel job
keep the smoke run before the broad Bazel test invocation

Validation

just bench-e2e-smoke

Stack

## Why Remote-executor integration tests need one host-agnostic exec-server fixture target instead of a Windows-only wrapper. ## What - rename the testing binary target to exec-server - make the fixture source and target host-agnostic - update Windows remote-executor test wiring to use the shared target ## Validation - bazel build //codex-rs/exec-server/testing:exec-server - bazel cquery --config=ci-windows-cross 'set(//codex-rs/exec-server/testing:exec-server //codex-rs/core/tests/remote_env_windows:smoke-test)' ## Stack 1. [#31422 test: generalize exec-server fixture](#31422) 2. [#31425 test: add TestAppServer builder](#31425) 3. [#31427 test: add delayed exec-server transport](#31427) 4. [#31295 bench: add cold skill load macrobenchmark](#31295) 5. [#31428 bench: add e2e benchmark entrypoints](#31428) 6. [#31429 ci: smoke Bazel e2e benchmarks](#31429)

## Why Test callers need one composable way to create app-server fixtures instead of a growing family of overlapping constructor implementations. ## What - add a feature-complete TestAppServer::builder() - make the default builder own a temporary CODEX_HOME and select the automatic test environment - expose builder knobs for no automatic environment, explicit CODEX_HOME, program, arguments, plugin startup tasks, environment overrides, managed config, and JSON logging - keep the existing public constructor surface, but route every constructor through the builder so the new path is exercised immediately - remove the redundant private constructor ladders; caller migration and public constructor removal live in the optional cleanup stack ## Validation - just test -p codex-app-server (940/941 before updating the expected builder error wording) - just test -p codex-app-server auto_env_rejects_explicit_environment_config - just fix -p codex-app-server - just fmt ## Follow-up stacks Cleanup, optional for the benchmark work: 1. [#31451 test: migrate TestAppServer callers to builder](#31451) 2. [#31452 test: remove TestAppServer constructors](#31452) Benchmark infrastructure: 1. [#31427 test: add delayed exec-server transport](#31427) 2. [#31295 bench: add cold skill load macrobenchmark](#31295) 3. [#31428 bench: add e2e benchmark entrypoints](#31428) 4. [#31429 ci: smoke Bazel e2e benchmarks](#31429)

## Why Macrobenchmarks benefit from having a way to exercise remote-executor latency without depending on Docker. This is a very minimal first cut, if we find that simulating network conditions is useful we can always expand this scope or switch to a more robust network shaping approach. ## What - add a package-local exec-server binary for Cargo and Bazel test fixtures - add a host-local WebSocket exec-server fixture and fixed-delay interposer - let TestAppServer route its auto environment through that delayed WebSocket transport - cover the delayed thread/start path through the public app-server API ## Stack 1. [#31425 test: add TestAppServer builder](#31425) 2. [#31427 test: add delayed exec-server transport](#31427) 3. [#31295 bench: add cold skill load macrobenchmark](#31295) 4. [#31428 bench: add e2e benchmark entrypoints](#31428) 5. [#31429 ci: smoke Bazel e2e benchmarks](#31429)

## Why Bazel-backed end-to-end macrobenchmark plumbing needs a small, deterministic first consumer that does not couple the shared infrastructure to the remote-skill scenario. ## What - add `codex_e2e_benchmark`, a small macro for Bazel-only Divan benchmarks and runtime binary runfiles - keep benchmark sources under `e2e_benches/` so Cargo does not auto-discover them - add a CLI example that resolves the real `codex` binary and measures `codex --help` - assert the spawned command succeeds ## Validation - `bazel test --compilation_mode=fastbuild --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=no --@rules_rust//rust/settings:extra_exec_rustc_flag=-Cdebug-assertions=no --cache_test_results=no --test_output=errors --test_arg=--test //codex-rs/cli:codex-help-bench` ## Stack 1. [#31295 bench: add codex help e2e macrobenchmark](#31295) 2. [#31428 bench: add e2e benchmark entrypoints](#31428) 3. [#31429 ci: smoke Bazel e2e benchmarks](#31429)

## Why Bazel-backed macrobenchmarks need stable local entrypoints that can grow without changing every caller. ## What - add a top-level `e2e-benchmarks` Bazel suite, initially containing the `codex --help` benchmark - add cross-platform `just bench-e2e` for direct optimized Bazel measurements - add cross-platform `just bench-e2e-smoke` for one direct fastbuild iteration through release-only Rust cfg paths ## Validation - `just --dry-run bench-e2e` - `just --dry-run bench-e2e-smoke` - `just bench-e2e-smoke` ## Stack 1. [#31295 bench: add codex help e2e macrobenchmark](#31295) 2. [#31428 bench: add e2e benchmark entrypoints](#31428) 3. [#31429 ci: smoke Bazel e2e benchmarks](#31429)

This was referenced Jul 7, 2026

bench: add codex help e2e macrobenchmark #31295

Merged

test: generalize exec-server fixture #31422

Merged

test: add TestAppServer builder #31425

Merged

test: add delayed exec-server transport #31427

Merged

bench: add e2e benchmark entrypoints #31428

Merged

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from 40d035e to f8fd71a Compare July 7, 2026 17:33

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from bdfc04d to d6b6627 Compare July 7, 2026 17:34

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from f8fd71a to f9458bb Compare July 7, 2026 17:36

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from d6b6627 to 50be93a Compare July 7, 2026 17:36

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from f9458bb to 7777895 Compare July 7, 2026 18:19

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from 50be93a to 185932b Compare July 7, 2026 18:19

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from 7777895 to 67e8bf0 Compare July 7, 2026 18:39

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from 185932b to b9d8644 Compare July 7, 2026 18:39

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from 67e8bf0 to b860399 Compare July 7, 2026 19:05

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from b9d8644 to ee923c4 Compare July 7, 2026 19:06

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from b860399 to 29179d5 Compare July 7, 2026 22:55

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from ee923c4 to df7a8a1 Compare July 7, 2026 22:55

anp-oai added a commit that referenced this pull request Jul 7, 2026

codex: fix CI failure on PR #31429

3b5cb27

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from 29179d5 to e5b20f0 Compare July 8, 2026 00:21

anp-oai added a commit that referenced this pull request Jul 8, 2026

codex: fix CI failure on PR #31429

95e1f8b

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from 3b5cb27 to 95e1f8b Compare July 8, 2026 00:21

github-actions Bot mentioned this pull request Jul 8, 2026

📊 AI CLI 工具社区动态日报 2026-07-08 loxehate/loxehate.github.io#18

Closed

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from e5b20f0 to 0a40f91 Compare July 9, 2026 00:19

anp-oai added a commit that referenced this pull request Jul 9, 2026

codex: fix CI failure on PR #31429

e8b9ed2

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from 95e1f8b to e8b9ed2 Compare July 9, 2026 00:20

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from 0a40f91 to 3b35b0b Compare July 9, 2026 00:27

anp-oai added a commit that referenced this pull request Jul 9, 2026

codex: fix CI failure on PR #31429

490d325

anp-oai added a commit that referenced this pull request Jul 9, 2026

codex: fix CI failure on PR #31429

c917b2c

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from 490d325 to c917b2c Compare July 9, 2026 02:23

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from abd411b to 4342c70 Compare July 9, 2026 02:47

anp-oai added a commit that referenced this pull request Jul 9, 2026

codex: fix CI failure on PR #31429

65fd933

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from c917b2c to 65fd933 Compare July 9, 2026 02:47

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from 4342c70 to 8f4d31c Compare July 9, 2026 04:00

anp-oai added a commit that referenced this pull request Jul 9, 2026

codex: fix CI failure on PR #31429

f419b29

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from 65fd933 to f419b29 Compare July 9, 2026 04:00

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from 8f4d31c to cf5b1b0 Compare July 9, 2026 04:46

anp-oai added a commit that referenced this pull request Jul 9, 2026

codex: fix CI failure on PR #31429

44d95fe

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from f419b29 to 44d95fe Compare July 9, 2026 04:46

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from cf5b1b0 to 572f841 Compare July 9, 2026 04:52

anp-oai added a commit that referenced this pull request Jul 9, 2026

codex: fix CI failure on PR #31429

c2afae3

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from 44d95fe to c2afae3 Compare July 9, 2026 04:54

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from 572f841 to f97c74f Compare July 10, 2026 00:08

anp-oai added a commit that referenced this pull request Jul 10, 2026

codex: fix CI failure on PR #31429

96316ac

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from c2afae3 to 96316ac Compare July 10, 2026 00:08

anp-oai force-pushed the codex/e2e-benchmark-entrypoints branch from f97c74f to eae9cc3 Compare July 10, 2026 00:24

anp-oai added a commit that referenced this pull request Jul 10, 2026

codex: fix CI failure on PR #31429

1f14285

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from 96316ac to 1f14285 Compare July 10, 2026 00:25

anp-oai commented Jul 10, 2026

View reviewed changes

Comment thread .github/workflows/bazel.yml Outdated

Base automatically changed from codex/e2e-benchmark-entrypoints to main July 10, 2026 00:31

anp-oai added a commit that referenced this pull request Jul 10, 2026

codex: fix CI failure on PR #31429

8156b2a

anp-oai force-pushed the codex/e2e-benchmark-ci-smoke branch from 1f14285 to 8156b2a Compare July 10, 2026 00:32

anp-oai added a commit that referenced this pull request Jul 10, 2026

codex: address PR review feedback (#31429)

cde2df3

anp-oai added 3 commits July 9, 2026 18:33

ci: smoke Bazel e2e benchmarks

1adb6b0

codex: fix CI failure on PR #31429

fb1ac4f

codex: address PR review feedback (#31429)

9f9ce73

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: smoke Bazel e2e benchmarks - #31429

ci: smoke Bazel e2e benchmarks#31429
anp-oai wants to merge 3 commits into
mainfrom
codex/e2e-benchmark-ci-smoke

anp-oai commented Jul 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

anp-oai commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Validation

Stack

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

anp-oai commented Jul 7, 2026 •

edited

Loading