Helios Benchmark Harness

Work Status — Phase 3: spec, tests, and trace layer in progress; not yet a passing release. Last updated 2026-06-18. Live status: PR #177 and repo CI.

A comprehensive benchmarking tool for CLI tools, inspired by Terminal-Bench.

Features

• System Resource Monitoring: CPU, memory (RSS/VMS), file descriptors, threads
• Leak Detection: Detect memory and FD leaks over multiple runs
• Terminal-Bench Style Tasks: Code completion, review, refactoring, debugging, testing
• Comparison Mode: Compare two binaries side-by-side
• JSON Output: Machine-readable results for CI/CD integration

Installation

git clone https://github.com/KooshaPari/heliosBench.git && cd heliosBench
pip install -e .

Usage

List available tasks

helios-bench tasks

Run a benchmark

helios-bench run --binary /path/to/codex --task palindrome --runs 5

Compare two binaries

helios-bench compare --binary-a custom-codex --binary-b homebrew-codex

Detect leaks

helios-bench leak --binary /path/to/codex --runs 20

Tasks

Task ID	Category	Difficulty	Description
palindrome	code_completion	easy	Write a palindrome checker function
fibonacci	code_completion	medium	Implement Fibonacci sequence generator
buggy_add	code_review	easy	Review code and identify bugs
refactor_loop	refactoring	easy	Refactor loop to list comprehension
debug_division	debugging	easy	Find and fix a division bug
write_tests	test_generation	medium	Generate pytest unit tests
bayesian_sampler	scientific_computing	hard	Implement adaptive rejection sampler
log_parser	system_admin	medium	Parse log file and extract errors

Output Metrics

• Latency: Mean, median, stdev (seconds)
• Memory: RSS mean/max (MB)
• CPU: Mean/max percentage
• File Descriptors: Mean/max
• Threads: Mean/max

Terminal-Bench Integration

This harness is compatible with Terminal-Bench style tasks.

References

• Terminal-Bench
• Harbor