Terminal-Bench Eval Harness - Phase 2: Real Harbor Framework Integration

[jeremyeder]

Overview

STATUS: ✅ COMPLETE - Technical implementation finished. Research study split to #201.

Implement real Harbor framework integration for the Terminal-Bench eval harness.

Completed: Harbor framework integration fully functional with real Terminal-Bench evaluations
Follow-on: Assessor refinement research study → #201

---

What Was Implemented ✅

1. Real Harbor Framework Integration

• ✅ Implemented _real_tbench_result() with Harbor subprocess API
• ✅ Harbor 2.0 result.json parsing
• ✅ Environment variable handling (ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN)
• ✅ MiniMax API override fix (Harbor hardcoded issue)
• ✅ Trajectory file path capture
• ✅ Security fixes (path validation, timeout handling, input validation)

2. CLI Implementation

• ✅ agentready benchmark command
• ✅ Terminal-Bench smoketest and full benchmark support
• ✅ Model selection (claude-haiku-4-5, claude-sonnet-4-5)
• ✅ Timeout configuration
• ✅ Verbose output mode
• ✅ --skip-preflight option for advanced users

3. Preflight Checks

• ✅ Automatic Harbor CLI detection
• ✅ Interactive installation prompts (uv/pip fallback)
• ✅ Terminal-Bench dataset management
• ✅ Graceful error handling

4. Tests

• ✅ TbenchResult model validation
• ✅ Harbor configuration tests
• ✅ Harbor services integration tests
• ✅ Preflight check tests (100% coverage)

5. Documentation

• ✅ Updated CLAUDE.md with Harbor integration details
• ✅ Preflight check documentation
• ✅ Usage examples in benchmark.py docstrings

---

What Was Split Out 📋

Assessor Refinement Research Study → Issue #201

The empirical research component (running 10-20 diverse repositories through Terminal-Bench to measure assessor impact) has been split into a separate issue. This was done because:

1. Technical work complete: All infrastructure needed for the research study is now in place
2. Different workflow: Research study is data collection + analysis, not implementation
3. Separate timeline: Can be executed independently after Terminal-Bench Eval Harness - Phase 2: Real Harbor Framework Integration #190 merges

See #201 for the complete research study plan.

---

Success Criteria (Original vs Actual)

Criteria	Status	Notes
Real benchmark runs work end-to-end	✅ YES	Harbor integration fully functional
Tests pass	✅ YES	All tests passing
Harbor framework integration	✅ YES	Complete with security fixes
CLI with subset options	✅ YES	smoketest/full support
Documentation updated	✅ YES	CLAUDE.md updated
10-20 benchmark runs on diverse repos	📋 #201	Split to follow-on research issue
Assessor refinement results	📋 #201	Split to follow-on research issue
docs/tbench/assessor-refinement-results.md	📋 #201	Split to follow-on research issue

---

Key Implementation Details

Files Changed

Core Implementation:

• src/agentready/services/eval_harness/tbench_runner.py - Harbor subprocess integration
• src/agentready/services/eval_harness/harbor_config.py - Configuration model
• src/agentready/cli/benchmark.py - CLI command
• src/agentready/utils/preflight.py - Dependency checking

Tests:

• tests/unit/test_harbor_*.py - Harbor integration tests
• tests/unit/utils/test_preflight.py - Preflight check tests (100% coverage)

Documentation:

• CLAUDE.md - Updated with Harbor integration, preflight checks
• docs/tbench/methodology.md - A/B testing methodology

Usage Examples

# Quick smoketest (1-2 tasks, ~2-5 min)
export ANTHROPIC_API_KEY=your-key-here
agentready benchmark --subset smoketest

# Full Terminal-Bench with Sonnet (~30-40 min)
agentready benchmark --subset full --model claude-sonnet-4-5

# Skip preflight checks (advanced)
agentready benchmark --subset smoketest --skip-preflight

---

Evidence of Completion

Harbor Integration Working:

• Real Harbor result.json files in jobs/ directory
• Successful Terminal-Bench task configurations
• Trajectory file capture working
• All tests passing

Commits (17 total):

• 9e9cc32 - feat: display Harbor command with copy/paste format
• 389958f - fix: override Harbor's hardcoded MiniMax API configuration
• d6f583e - feat: display trajectory file path in benchmark summary
• f9f6cfb - fix: set ANTHROPIC_AUTH_TOKEN for Harbor's Claude Code agent
• 97b848f - feat: add automatic Harbor CLI preflight checks
• b91d11b - fix: correct Harbor results parsing (Harbor 2.0 structure)
• bdbabb2 - fix(security): implement critical security fixes
• 4d5649a - feat: add Harbor framework integration (initial)
• ...and 9 more commits

---

Follow-On Work

Immediate:

• Terminal-Bench Assessor Refinement Research Study (Phase 2b) #201 - Assessor refinement research study (10-20 repos, statistical analysis)

Future:

• Dashboard integration for Terminal-Bench results
• Historical tracking of benchmark performance
• Leaderboard integration (if desired)

---

Resources

Implementation:

• Harbor Framework: https://harborframework.com/docs
• Terminal-Bench: https://tbench.ai
• Branch: 002-harbor-real-integration

Related Issues:

• Terminal-Bench Assessor Refinement Research Study (Phase 2b) #201 - Assessor Refinement Research Study (follow-on)
• feat: Terminal-Bench eval harness (MVP Phase 1) #178 - Terminal-Bench Eval Harness MVP (Phase 1, completed)

---

Labels: enhancement, terminal-bench, harbor-integration, completed

[jeremyeder]

🤖 Automated Code Review Results

Two specialized agents reviewed the Phase 2 implementation plan and identified critical issues that need to be addressed:

---

🔴 Critical Security Issues (Must Fix Before Implementation)

1. API Key Exposure Risk (Confidence: 95%)

Issue: API keys passed through environment variables to subprocess without sanitization.

Vulnerable Pattern:

# DON'T: Exposes all environment variables
subprocess.run(cmd, env=os.environ.copy())

Fix:

# DO: Pass only required variables
clean_env = {
    "ANTHROPIC_API_KEY": os.environ.get("ANTHROPIC_API_KEY"),
    "PATH": os.environ.get("PATH"),
    "HOME": os.environ.get("HOME")
}
subprocess.run(cmd, env=clean_env)

2. Command Injection Vulnerability (Confidence: 92%)

Issue: Model and agent parameters not validated before passing to Harbor CLI.

Fix: Add allowlist validation:

ALLOWED_MODELS = {"anthropic/claude-haiku-4-5", "anthropic/claude-sonnet-4-5"}
ALLOWED_AGENTS = {"claude-code"}

if model not in ALLOWED_MODELS:
    raise ValueError(f"Invalid model: {model}")

3. Missing Resource Limits (Confidence: 88%)

Issue: Parallel execution (8 repos × 35 runs) could exhaust system resources.

Fix: Implement worker pool with limits:

from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor(max_workers=4) as executor:
    futures = [executor.submit(run_benchmark, repo) for repo in repos]
    for future in as_completed(futures, timeout=86400):
        result = future.result(timeout=3600)

---

🟡 Over-Engineering (Simplify to 76% Less Code)

The current plan proposes 507 new lines of code. This can be reduced to ~120 lines (76% reduction) by:

1. Simplify Harbor Integration (186 → 35 lines)

• Remove 7 custom exception classes → use RuntimeError
• Remove 3 pre-flight check methods → let Harbor handle validation
• Trust Harbor's Docker checks instead of reimplementing

2. Inline Aggregation Logic (171 → 30 lines)

• Replace CrossRepoAggregator service with pandas DataFrame operations
• Add directly to existing summarize CLI command

import pandas as pd

df = pd.DataFrame(results)
summary = df.groupby("assessor_id").agg({
    "delta_score": ["mean", "median", "std"],
    "is_significant": "mean"
}).round(2)

3. Reduce Testing Overhead (150 → 40 lines)

• Keep 1 integration test with subprocess mock
• Remove tests for Harbor's internal behavior (Docker checks, etc.)
• Focus on testing our integration logic only

4. Use Environment Variable Instead of CLI Flag

• Keep existing mock behavior as default
• Add TBENCH_USE_REAL=1 environment variable for real benchmarks
• No need to modify CLI commands with --mock/--no-mock flags

---

✅ Recommended Simplified Approach

Implementation (2-3 days instead of 1 week)

1. Harbor Integration (~50 lines total)

   • Replace _real_tbench_result() NotImplementedError
   • Basic subprocess call with proper environment sanitization
   • Simple JSON parsing with path validation
   • Input validation for model/agent parameters

2. Aggregation (~30 lines)

   • Add pandas-based aggregation to existing summarize command
   • No new service module needed

3. Testing (~40 lines)

   • One solid integration test with subprocess mocking
   • Verify command construction and result parsing

4. Documentation

   • Update existing docs/tbench.md (no new files needed)
   • Add Harbor setup section
   • Document results inline

---

📋 Action Items Before Implementation

• Fix API key exposure vulnerability
• Add input validation for command injection
• Implement resource limits for parallel execution
• Simplify Harbor integration (trust Harbor's validation)
• Use pandas for aggregation instead of custom service
• Reduce testing to essentials
• Update plan with security fixes

Estimated Implementation Time: 2-3 days (simplified) vs 1 week (original plan)
Estimated Cost: Still ~$101 USD (8 repos × 35 runs × $0.36)

---

Generated by @agent-pr-review-toolkit:code-reviewer and @agent-pr-review-toolkit:code-simplifier

[jeremyeder]

📝 Specification Complete - Phase 2 Harbor Real Integration

Branch: 001-harbor-real-integration
Specification: specs/001-harbor-real-integration/spec.md
Status: ✅ Ready for Planning (/speckit.plan)

---

📊 Specification Quality Validation

All checklist items passed ✅:

• ✅ No implementation details (technology-agnostic)
• ✅ Security concerns elevated to P1 priority
• ✅ Simplified approach (76% code reduction: 507 → ~120 lines)
• ✅ 4 user stories with independent testability
• ✅ 14 functional requirements with clear acceptance criteria
• ✅ 10 measurable success criteria
• ✅ 6 comprehensive edge cases identified

---

🎯 Key Specification Highlights

User Stories (Prioritized)

1. P1: Run Real Terminal-Bench Evaluations (core functionality)
2. P2: Aggregate Multi-Repository Results (empirical assessor refinement)
3. P1: Secure API Integration (addresses automated review security findings)
4. P2: Resource-Limited Parallel Execution (4 workers, 1-hour timeout)

Security Requirements (Addresses Automated Review Findings)

• FR-002, FR-003: Allowlist validation for model/agent parameters (prevents command injection)
• FR-004: Sanitize environment variables - pass only API key, PATH, HOME (prevents API key exposure)
• FR-008, FR-009: Resource limits (4 concurrent workers, 1-hour timeout per job)

Simplified Approach (76% Code Reduction)

• ❌ Out of Scope: 7 custom exception classes → use RuntimeError
• ❌ Out of Scope: 3 pre-flight check methods → trust Harbor's validation
• ❌ Out of Scope: Separate CrossRepoAggregator service → inline with pandas
• ✅ In Scope: ~120 lines of focused implementation

---

🤖 doubleagent.md Impact Analysis

Impact Rating: 9/10 (High Impact)

The .claude/agents/doubleagent.md agent documentation had critical influence on this specification:

Architecture & Design (doubleagent.md:28-68)

• ✅ Library-First: Environment variable toggle (TBENCH_USE_REAL=1) instead of global state
• ✅ Strategy Pattern: Assessors treated independently in aggregation
• ✅ Dependency Injection: HarborConfig entity with injectable credentials
• ✅ Fail Gracefully: FR-012 handles Harbor framework errors with clear guidance

Security & Anti-Patterns (doubleagent.md:232-238, 502-523)

• ✅ Prevented API key exposure (FR-004) - doubleagent.md warned against this
• ✅ Prevented command injection (FR-002, FR-003) - security principle applied
• ✅ Avoided over-engineering - 76% code reduction aligned with anti-patterns
• ✅ Maintained backward compatibility (FR-014) - explicit anti-pattern guidance

Testing & Quality (doubleagent.md:171-217)

• ✅ 100% independently testable user stories
• ✅ 6 edge cases covering auth, network, timeout, size limits, parsing, partial failures
• ✅ Target: >80% coverage for new code (implied by success criteria)

Proportional Scoring Pattern (doubleagent.md:120-133)

• ✅ FR-010: Aggregation uses mean/median/std for proportional assessor effectiveness
• ✅ FR-011: Statistical significance indicators (confidence intervals, p-values)

Full Impact Report: specs/001-harbor-real-integration/DOUBLEAGENT_IMPACT.md

---

💾 Knowledge Stored to Memory MCP Server

Stored 3 entities with 3 relationships for future context:

Entities Created:

1. AgentReady Terminal-Bench Eval Harness (project-component)

   • Phase 2 goals: real Harbor integration, empirical assessor validation
   • Security vulnerabilities identified: API key exposure, command injection
   • Simplified approach: 507 → ~120 lines (76% reduction)

2. Harbor Framework Integration Pattern (architecture-pattern)

   • Security: Sanitized environment variables, allowlist validation
   • Resource management: ProcessPoolExecutor with 4 workers
   • Graceful degradation: TBENCH_USE_REAL=1 toggle

3. doubleagent.md Agent Documentation Impact (development-guidance)

   • Architectural guardrails: library-first, strategy pattern, dependency injection
   • Security principles: prevented API key exposure and command injection
   • Anti-patterns drove 76% code reduction
   • Testing philosophy shaped user story structure

Relationships Created:

• AgentReady Terminal-Bench Eval Harness implements Harbor Framework Integration Pattern
• doubleagent.md Agent Documentation Impact informed-design-of Harbor Framework Integration Pattern
• doubleagent.md Agent Documentation Impact provided-architectural-guidance-for AgentReady Terminal-Bench Eval Harness

---

📋 Next Steps

1. ✅ Specification complete and validated
2. ⏭️ Ready for /speckit.plan - Design implementation approach
3. ⏭️ After planning: /speckit.tasks - Generate dependency-ordered tasks
4. ⏭️ After tasks: /speckit.implement - Execute implementation

---

Specification Document: specs/001-harbor-real-integration/spec.md
Quality Checklist: specs/001-harbor-real-integration/checklists/requirements.md
Impact Report: specs/001-harbor-real-integration/DOUBLEAGENT_IMPACT.md

[jeremyeder]

📋 Implementation Plan Complete - Phase 2 Harbor Real Integration

Branch: 002-harbor-real-integration (renamed from 001 to avoid conflict)
Plan Document: specs/002-harbor-real-integration/plan.md
Status: ✅ Ready for Task Generation (/speckit.tasks)

---

📦 Planning Deliverables

Phase 0: Research (Complete ✅)

• ✅ research.md: All 7 technical unknowns resolved
  • Harbor package: uv pip install harbor
  • Authentication: ANTHROPIC_API_KEY environment variable
  • CLI syntax: harbor run --dataset terminal-bench@2.0 --agent claude-code --model <model> --jobs-dir <path>
  • Output: JSON at <jobs-dir>/results.json
  • Execution: 5-10 min average, 1-hour timeout (6x buffer)
  • Validation: Allowlists for models (haiku-4-5, sonnet-4-5) and agents (claude-code)
  • Docker: Required for local execution (trust Harbor's validation)

Phase 1: Design & Contracts (Complete ✅)

• ✅ data-model.md: Entity definitions
  • TbenchResult: Extended with 4 new fields (resolved_trials, unresolved_trials, pass@1, pass@3)
  • HarborConfig: New dataclass with validation (model/agent allowlists, API key requirement)
  • AggregatedResult: Conceptual (pandas DataFrame, no formal model)
• ✅ contracts/harbor-results-schema.json: Harbor output validation schema
• ✅ contracts/aggregation-output-schema.json: AgentReady aggregation export schema
• ✅ quickstart.md: 10-minute setup guide with troubleshooting

---

🏗️ Architecture Decisions

Simplified Implementation (76% Code Reduction)

• ~120 lines of new code (vs original 507-line plan)
• ❌ No custom exception classes (use RuntimeError)
• ❌ No pre-flight checks (trust Harbor validation)
• ❌ No separate aggregator service (inline pandas operations)

Security Hardening (Addresses Automated Review)

1. API Key Sanitization (FR-004): Pass only ANTHROPIC_API_KEY, PATH, HOME to subprocess
2. Command Injection Prevention (FR-002, FR-003): Allowlist validation for model/agent parameters
3. Path Traversal Protection (FR-005): Validate results path is relative to jobs_dir

File Structure

src/agentready/services/eval_harness/
├── harbor_config.py (NEW ~40 lines)
├── tbench_runner.py (MODIFY +50 lines)

tests/unit/
├── test_harbor_config.py (NEW ~40 lines)
├── test_eval_harness_services.py (MODIFY +40 lines)

docs/tbench/
├── assessor-refinement-results.md (NEW template)

---

✅ Constitution Compliance

Pre-Research Check: ✅ Pass (all 7 principles)
Post-Design Check: ✅ Pass (all 7 principles)

Key Validations:

• ✅ Evidence-Based: 6 authoritative sources cited (Harbor docs, GitHub, Snorkel AI, VentureBeat)
• ✅ Measurable Quality: Quantifiable success criteria (100% accuracy blocking invalid params, 95% clear error messages)
• ✅ Tool-First: Harbor subprocess interface (text-based I/O), self-contained HarborConfig
• ✅ Structured Output: JSON schemas defined, pandas supports JSON export and markdown tables
• ✅ Incremental Delivery: P1 MVP = User Story 1 (real benchmarks) + User Story 3 (security)
• ✅ Documentation: Quickstart <10 min, research captures all decisions, contracts define formats

Complexity Limits:

• ✅ File Size: <300 lines per file
• ✅ Function Length: <50 lines (subprocess call estimated ~50, parsing ~30)
• ✅ Cyclomatic Complexity: <10 (simple conditionals only)
• ✅ Dependencies: Harbor is only new external dependency

---

🔐 Security Enhancements

Critical Findings Addressed

Finding	Severity	Solution	Verification
API key exposure	Critical	Sanitize env vars (only ANTHROPIC_API_KEY, PATH, HOME)	Unit test checks env dict excludes non-required vars
Command injection	Critical	Allowlist validation (ALLOWED_MODELS, ALLOWED_AGENTS)	Unit test attempts malicious input (e.g., $(rm -rf /))
Path traversal	Medium	Validate results_path.is_relative_to(jobs_dir)	Unit test attempts ../../etc/passwd

---

📚 Documentation Artifacts

1. research.md (Complete): Technical decisions with 6+ sources
2. data-model.md (Complete): Entity definitions and validation rules
3. quickstart.md (Complete): 10-minute setup with troubleshooting
4. plan.md (Complete): Implementation blueprint with file-level details
5. contracts/ (Complete): JSON schemas for Harbor output and aggregation

---

🎯 Expected Implementation Tasks (Preview)

Priority 1 (MVP):

1. Write tests for HarborConfig validation (TDD: red)
2. Implement HarborConfig with allowlist validation
3. Write tests for _real_tbench_result() subprocess (TDD: red, mock subprocess)
4. Implement _real_tbench_result() with sanitized env vars
5. Write tests for JSON parsing with path validation (TDD: red)
6. Implement parse_harbor_results()
7. Write integration test for full workflow (TDD: red)
8. Verify all tests pass (TDD: green)

Priority 2 (Aggregation):
9. Write tests for pandas aggregation (TDD: red)
10. Implement aggregation in summarize command
11. Add parallel execution limits (ProcessPoolExecutor max_workers=4)
12. Add timeout enforcement (3600s per benchmark)

Priority 3 (Documentation):
13. Update README.md with Harbor setup
14. Create docs/tbench/assessor-refinement-results.md template
15. Update docs/tbench/methodology.md
16. Run linters and verify >80% coverage

Full task list will be generated by /speckit.tasks command

---

📈 Performance Goals

• Individual Benchmark: 5-10 minutes (1-hour timeout provides 6x buffer)
• Batch Execution: <24 hours for 280 runs (8 repos × 35 assessors) with 4-worker parallelism
• Resource Limits: <2GB memory, <1024 file handles (enforced by worker limit)

---

💰 Cost Estimate

Per Repository:

• Model: Claude Haiku 4.5
• Duration: ~10 minutes
• Estimated Cost: $0.30 - $0.50 USD

Batch (10 repos × 35 assessors = 350 runs):

• Total Cost: ~$105 - $175 USD
• Time: ~24 hours with 4-worker parallelism

---

🔄 Next Steps

1. ✅ Planning complete (Phases 0 & 1)
2. ✅ Agent context updated (CLAUDE.md)
3. ⏭️ Generate tasks: /speckit.tasks
4. ⏭️ Execute implementation: /speckit.implement
5. ⏭️ Run empirical benchmarks on 10-20 repositories
6. ⏭️ Document assessor refinement findings

---

Planning Artifacts:

• Specification
• Implementation Plan
• Research
• Data Model
• Quickstart
• Contracts

[github-actions]

🎉 This issue has been resolved in version 2.17.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀