codex-batch-bench

Controlled A/B experiment testing whether an explicit tool-call batching instruction changes token consumption of gpt-5.6-sol in Codex CLI on a fixed, read-heavy investigation task (the mechanism alleged in the #32503 / #35050 discussion: Sol serializing known-independent Code Mode reads into separate exec cells, multiplying outer model cycles).

Hypothesis

H1: With no batching guidance, Sol splits independent reads across many exec cells → more model cycles → more (mostly cached-input) tokens per identical task. An explicit AGENTS.md batching instruction reduces cycles and total tokens; quality stays equal.

Design

• Task (identical in both arms): frozen synthetic Python repo, 12 pipeline modules + config/defaults.toml + MANIFEST.txt. The prompt asks for a per-module table (public functions, config keys read, imports) plus unused config keys. After reading MANIFEST.txt, all 14 file reads are known-independent — the ideal batchable workload. Ground truth is generated, so answers are gradeable (quality parity check between arms).
• Intervention: AGENTS.md differs between arms and in nothing else.
  • arm a = placebo: length-matched, style-only instruction (controls for "any instruction present" and prompt-length effects).
  • arm b = batching: the Promise.allSettled batching instruction.
  • Arms live at anonymized paths work/repo_a / work/repo_b so the model cannot infer the experiment from the path. The word "batch" appears nowhere in the repo content or the prompt.
• Pinned per trial: model gpt-5.6-sol, fixed reasoning effort, sandbox read-only, mcp_servers={}, plugins={}, developer_instructions="", features.multi_agent=false, features.memories=false (prevents cross-trial memory contamination), fresh codex exec session per trial.
• Ordering: arms alternate (a,b,b,a,a,b) so both arms have the same number of same-arm predecessors — prompt-cache self-priming across trials affects both arms equally.
• Metrics (parsed from ~/.codex/sessions rollout JSONL, copied into the trial dir):
  • cycles — number of token_count events ≈ model API responses ("outer model cycles").
  • token totals from final total_token_usage: input, cached input, uncached (= input − cached), output, reasoning, total.
  • credits — weighted usage at the published subscription rate card (125 / 12.5 / 750 credits per 1M uncached-in / cached-in / out). Treat as approximate; the internal Pro allowance formula is not public.
  • batching behavior: exec cells (custom_tool_call name="exec"), tools.*( calls per cell, cells with ≥2 calls, cells containing Promise.all/allSettled, plain function_calls.
  • live weekly quota used_percent first/last (coarse cross-check).
  • grade: modules named in final answer (n/12) and true unused keys found (n/6).

Run it

python3 bench.py setup                      # build frozen repos (small + large presets, arm a/b)
python3 bench.py ping                       # cheap plumbing check
python3 bench.py run --preset large --effort xhigh --trials-per-arm 3
python3 bench.py run-real --repo ~/my/repo --globs "'src/*.py' 'src/**/*.py'"
python3 bench.py analyze runs/<stamp>       # re-run analysis any time

run-real benchmarks a real repository read-only: the arm instruction is injected per-trial via -c developer_instructions=... (which fully replaces the value in ~/.codex/config.toml — verified empirically), so the repo and your global config are never modified. Measured results: see RESULTS.md.

Note: the harness always sets features.memories=false; without it, headless codex exec runs were observed to hang on exit (>6 min for a trivial prompt on CLI 0.145.0).

Validating a tool-description PR (binary vs binary)

To test a patched codex build (e.g. an exec-description change teaching the nested-concurrency path) against stock, run the same benchmark once per binary and compare the placebo arms across the two runs — the placebo arm carries no batching guidance, so any improvement isolates the built-in description change. The batching arm serves as a positive-control ceiling (what an explicit instruction achieves).

python3 bench.py run --preset large --effort xhigh --trials-per-arm 3
python3 bench.py run --preset large --effort xhigh --trials-per-arm 3 \
    --codex-bin ~/codex/target/release/codex

Because the pathology is effort-gated, validate at xhigh (and ideally max/ultra) — a description change validated only at high or below will not exercise the failure mode. Check that high-and-below placebo numbers do not regress (the model already self-batches there).

Outputs land in runs/<stamp>/: per-trial rollout copies, events.jsonl, last_message.txt, metrics.csv, report.md.

Caveats

• n=3 per arm is small; treat ratios as effect-size estimates, not precise multipliers. Add trials for confidence.
• Effort defaults to medium to conserve quota (the config default is xhigh); rerun with --effort xhigh to match production workloads.
• The task is deliberately maximally batchable; real workloads are a mix of batchable and dependent stages, so real-world savings will be smaller than this benchmark's ceiling.
• Server-side prompt/model changes can shift behavior over time; results are a snapshot.