Add pass@k and pass^k metrics for Guidance

[Jordak]

Current behavior

As of this issue being written, Guidance reports a single-run behavioral pass/fail summary for a harness run. This works for deterministic failures, but recent Codex Guidance diagnostics showed stochastic fixture behavior: targeted reruns can pass while a full default run may fail a different fixture, and a later full low-effort run can pass all fixtures.

Why this should change

A single run makes it hard to distinguish durable guidance regressions from flaky or stochastic harness behavior. Maintainers need a way to report both whether a fixture can pass at least once across repeated attempts and whether it passes consistently across repeated attempts.

Desired behavior

Add repeat-run metric support for Guidance, without changing fixture expected behavior, judge semantics, or default scoring unless explicitly designed in the implementation issue/PR.

Define and report at least:

• pass@k: a fixture or suite is counted as pass@k when at least one of k repeated independent runs passes.
• pass^k: a fixture or suite is counted as pass^k when all k repeated independent runs pass.

The implementation should make clear whether metrics are computed per fixture, per suite, per harness, or all three. It should also preserve the existing single-run summary for ordinary diagnostic and status-compatible workflows unless the status policy is deliberately updated.

Suggested scope

• Add CLI support for configuring repeated Guidance attempts, for example a repeat count k.
• Record enough structured metadata to compute pass@k and pass^k without copying raw HUT answers into Core.
• Summarize stochastic outcomes in saved reports using public-safe counts and fixture identifiers.
• Keep raw repeated-run evidence under the Personal Overlay report directory.
• Decide whether pass@k/pass^k are diagnostic-only initially or eligible for benchmark status refresh after a separate status-policy update.

Validation

• Guidance self-tests cover pass@k/pass^k aggregation behavior with synthetic results.
• Existing single-run Guidance behavior remains compatible when k is not configured.
• scripts/run-validator passes.
• Saved reports do not copy raw/private evidence into Core.

[Jordak]

This was generated by AI during triage.

Deferred Design Decision

We discussed the repeat-run metrics proposal and decided not to proceed with implementation yet.

Tentative design direction if revisited

• pass@k should mean at least one of k repeated independent Guidance attempts passes.
• pass^k should mean all k repeated independent Guidance attempts pass.
• fixture_stale remains a fixture-maintenance signal rather than a behavioral pass or fail.
• If any required attempt returns needs_user_judgment, repeat aggregation should be withheld for that fixture/harness group and headline suite-level repeat metrics should also be withheld until the judgment is resolved.
• Metrics should be diagnostic-only initially. Existing single-run status eligibility should remain unchanged unless a later status-policy issue deliberately changes that.
• CLI shape can be a repeat count such as --repeat N, defaulting to 1 so existing behavior remains compatible.
• Repeated runs should run all attempts rather than aborting early, so stochastic behavior is observable.
• Public/Core-safe report summaries may include counts, fixture IDs, attempt indices, verdict categories, and metadata. Raw HUT answers and judge rationales from repeated attempts should remain in the Personal Overlay report evidence.
• Acceptance coverage should include synthetic aggregation tests, default k=1 compatibility, CLI validation, and report privacy boundaries.

Why this is deferred

The design is plausible, but repeat metrics would materially increase Guidance runtime cost and reporting complexity. It would turn Guidance from a single expensive diagnostic/status run into a small experiment harness. That may be worthwhile later, but the current value/cost tradeoff is not settled.

Revisit this when recurring stochastic Guidance failures create enough need for repeat-run diagnostics, or when benchmark status policy explicitly needs stability metrics.