fix: configure Sandbox0 benchmark resources and lifetime

[laotoutou]

What changed

• Size Sandbox0 sequential, staggered, and burst TTI sandboxes at 128 MiB.
• Keep Sandbox0 DAX sandboxes at the standardized 16 GiB profile through the existing DAX resource override.
• Set a 15-minute hard TTL on every Sandbox0 benchmark sandbox while retaining the existing best-effort destroy path.

Why

The TTI workloads only create a sandbox and run the first node -v readiness command. They do not need the 16 GiB footprint used by the DAX build workload, especially when staggered and burst runs create up to 100 sandboxes.

A cancelled GitHub Actions job can also terminate the Node.js process before its finally block calls destroy(). Without a hard TTL, that can leave a claimed Sandbox0 sandbox running indefinitely.

DAX allows up to 10 minutes per sandbox, so a 15-minute hard TTL leaves cleanup headroom. The DAX option merge overrides the 128 MiB base memory with 16 GiB while preserving the hard TTL.

Sandbox0 resource profile

Mode	Memory	Hard TTL
Sequential TTI	128 MiB	15 minutes
Staggered TTI	128 MiB	15 minutes
Burst TTI	128 MiB	15 minutes
DAX	16 GiB	15 minutes

Normal runs still call destroy() immediately. The hard TTL is only a server-side fallback for interrupted or failed cleanup.

Impact

This only changes Sandbox0 benchmark sandboxes. Other providers are unaffected.

Validation

• pnpm typecheck
• Built @benchsdk/client, @benchsdk/runner, and create-bench
• @benchsdk/client tests: 118 passed, 1 skipped
• @benchsdk/runner tests: 56 passed
• create-bench tests: 1 passed
• No live provider benchmark was run and no provider credentials were used for this change

[open-cla]

Contributor License Agreement

All contributors are covered by a CLA.

[devin-ai-integration]

Devin Review found 1 potential issue.

View 1 additional finding in Devin Review.

[devin-ai-integration]

🟡 One provider's startup-time results become non-comparable after its benchmark machines are shrunk

The startup-time benchmark for Sandbox0 now requests a tiny 128 MiB machine (sandboxOptions: { memory: 128, hardTtl: 900 } at benchmarks/sandbox/providers.ts:190) while other providers in the same comparison still request large machines, so its startup numbers are measured on a different footprint than everyone else's.
Impact: Published sequential/staggered/burst startup rankings can favor this one provider for reasons unrelated to provider speed, and a 128 MiB machine may also be too small for the readiness check to run reliably.

Shared provider list feeds both the TTI and DAX benchmarks

benchmarks/sandbox/providers.ts is the single participant list used by benchmarks/sandbox/tti.bench.ts (which passes participant.sandboxOptions straight into compute.sandbox.create() at benchmarks/sandbox/tti.bench.ts:79) and by benchmarks/sandbox/dax.bench.ts:217. Other participants in the same TTI comparison explicitly request the standardized large profile there — e.g. e2b uses templateId: 'base-8cpu-16gb' (benchmarks/sandbox/providers.ts:103) — or use provider defaults, none of which are 128 MiB. Smaller machines typically provision faster, so the reported time-to-interactive for sandbox0 is no longer measured under comparable conditions. CONTRIBUTING.md also asks that methodology changes affecting historical comparability be discussed in an issue first, since past sandbox0 TTI results in results/sequential_tti/*.json were produced with the provider default size.

Prompt for agents

benchmarks/sandbox/providers.ts now sets sandbox0's shared sandboxOptions to { memory: 128, hardTtl: 900 }. That option object is consumed by both benchmarks/sandbox/tti.bench.ts (time-to-interactive) and benchmarks/sandbox/dax.bench.ts. The DAX path overrides memory back to 16384 via DAX_RESOURCE_OPTIONS, but the TTI path now creates 128 MiB sandboxes for sandbox0 while other participants (e.g. e2b's base-8cpu-16gb template) still run on the standardized 8 vCPU / 16 GiB profile, and historical sandbox0 TTI results in results/*_tti were collected with the provider default size. Consider whether TTI sizing should be standardized across providers rather than special-cased for one provider (and whether 128 MiB is sufficient to run the `node -v` readiness probe), or keep only the hardTtl safety net here and apply memory sizing per-benchmark. CONTRIBUTING.md also asks for methodology changes affecting comparability to be discussed in an issue first.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[laotoutou]

Thanks — I tested this against the exact PR head instead of inferring the resource profile from the shared participant list. The premise that Sandbox TTI is standardized at 8 vCPU / 16 GiB across providers is not correct.

The repository has two different resource policies:

• TTI passes each participant's ordinary sandboxOptions directly to create() (tti.bench.ts).
• DAX alone applies the standardized large profile through DAX_RESOURCE_OPTIONS; Sandbox0 is overridden to 16 GiB there while hardTtl is preserved (dax.bench.ts). I also invoked the PR's exported DAX task with a create-option probe: it produced { memory: 16384, hardTtl: 900 }.

Concrete counterexamples to “the other TTI participants use the large profile”:

• Modal has no TTI resource override in the participant list (code); Modal documents a default request of 128 MiB for Functions and Sandboxes (docs).
• Lightning TTI explicitly defaults to cpu-1, while only the DAX workflow sets cpu-8 (participant, DAX workflow). Lightning currently documents cpu-1 as 1 GB (docs).
• Tensorlake also has no TTI override (participant); its documented default is 1 CPU / 1024 MB (docs).
• CreateOS has no TTI override, and its adapter fallback is s-1vcpu-1gb (source).

E2B and Namespace are explicit 16 GiB exceptions; they do not establish a TTI-wide standard.

The separation matters operationally. If 16 GiB is put in the shared provider options, it feeds all four modes (sequential, staggered, burst, and DAX). At 100 iterations per mode that is 16 GiB × 4 × 100 = 6,400 GiB = 6.25 TiB of aggregate memory requested across 400 claims; burst alone can require 100 × 16 GiB = 1.5625 TiB concurrently. The scheduled workflow does not make all 400 concurrent (the three TTI modes use 100 iterations, while DAX defaults to 1), but this is exactly why the 16 GiB setting belongs in the DAX-only map rather than the shared TTI options (TTI workflow, DAX default).

Validation evidence:

• On the exact PR SHA, the live public coding-agent template currently reported 4 GiB, not 16 GiB. An explicit 128 MiB sandbox was then created successfully; the service reported 128Mi, node -v returned v22.23.2, and the observed hard TTL was 898 seconds. Sequential TTI completed 100/100 successfully. The same run's staggered phase was interrupted by a data-plane incident: 65 claims failed before command execution with the identical pod IP is not assigned after 1m30s error; cleanup was verified (current validation run).
• A clean dedicated-team run of the same 128 MiB TTI profile completed 100/100 sequential, 100/100 staggered, and 100/100 burst, with zero errors (full validation run). This directly verifies that 128 MiB is sufficient for the node -v readiness workload.

Historical results collected with a different Sandbox0 default should be treated as a previous baseline when results are regenerated; that comparability concern does not make TTI resource sizing uniform across providers. The PR correctly scopes 128 MiB to the three TTI modes and preserves 16 GiB for DAX, so I don't think this is a bug in the change.