Skip to content

Fix fuzz job in ci.yml: treat context deadline as expected success#12526

Merged
pelikhan merged 4 commits into
mainfrom
copilot/fix-fuzz-job-timeout-issue
Jan 29, 2026
Merged

Fix fuzz job in ci.yml: treat context deadline as expected success#12526
pelikhan merged 4 commits into
mainfrom
copilot/fix-fuzz-job-timeout-issue

Conversation

Copilot AI commented Jan 29, 2026

Copy link
Copy Markdown
Contributor

Go 1.25+ exits with status 1 when fuzz tests hit their -fuzztime limit, causing the CI job to fail after the first target and skip the remaining 12 tests.

Changes

ci.yml - Fixed the fuzz job by wrapping each go test -fuzz invocation in a bash function that:

  • Captures output to file
  • Checks for "context deadline exceeded" message on non-zero exit
  • Returns success (0) if deadline reached, failure (1) otherwise
run_fuzz_test() {
  local fuzz_name=$1
  local package=$2
  local output_file="fuzz-results/${fuzz_name}.txt"
  
  if go test -run='^$' -fuzz="${fuzz_name}" -fuzztime=10s "${package}" 2>&1 | tee "${output_file}"; then
    echo "${fuzz_name} completed successfully"
    return 0
  else
    if grep -q "context deadline exceeded" "${output_file}"; then
      echo "${fuzz_name} completed (context deadline reached as expected)"
      return 0
    else
      echo "${fuzz_name} failed with unexpected error"
      return 1
    fi
  fi
}

run_fuzz_test "FuzzParseFrontmatter" "./pkg/parser/"
run_fuzz_test "FuzzScheduleParser" "./pkg/parser/"
# ... 11 more targets

ci-coach.md - Updated all references from agent-ci.yml to ci.yml so the CI optimization workflow targets the correct file (3 references updated, workflow recompiled).

agent-ci.yml - Deleted obsolete workflow file (the fuzz fix is now in ci.yml, the active CI workflow).

All 13 fuzz targets now run sequentially in ci.yml. Real errors still fail the job.

Original prompt

This section details on the original issue you should resolve

<issue_title>[CI Failure Doctor] 🏥 CI Failure Investigation - Run #32553</issue_title>
<issue_description>## Summary
The fuzz job now terminates immediately because the first fuzz target hits Go's context deadline and go test exits 1, so the rest of the suite never runs.

Failure Details

Root Cause Analysis

go test -run '^$' -fuzz=FuzzParseFrontmatter -fuzztime=10s ./pkg/parser/ exits with context deadline exceeded after ~11 seconds, and because the job uses set -o pipefail the non-zero status immediately fails the entire fuzz workflow. That context timeout is expected when the fuzz target hits the configured time limit, but the command currently propagates the error instead of being treated as a successful timeout.

Failed Jobs and Errors

  • fuzz: first command FuzzParseFrontmatter logs context deadline exceeded and exit status 1, halting the job before any other targets run.

Investigation Findings

  • Baseline coverage completes (45/45 completed) and the fuzz run begins, but the log shows fuzz: elapsed: 11s followed immediately by context deadline exceeded.
  • The go test invocations are running sequentially with -fuzztime=10s, so each one is supposed to return success when the time window expires, yet Go now reports the deadline as an error that flows through due to the job’s set -o pipefail guard.
  • There is no evidence of tooling bugs or crashes; the failure is purely due to hitting the configured timeout while fuzzing.

Recommended Actions

  • Wrap each go test -fuzz command so that a context deadline exceeded exit is treated as an expected success (or otherwise neutralize the non-zero exit) before allowing the pipeline to continue to the next target.
  • Investigate whether increasing -fuzztime (or tuning the baseline coverage steps) keeps the run short enough that Go no longer cancels the context, and adjust the job documentation so engineers know the timeout behavior changed in Go 1.25.

Prevention Strategies

Document the expectation that fuzz targets will exit with context deadline exceeded when they reach their configured run time and add a wrapper script that recognizes that message before applying set -o pipefail, ensuring a single target timing out can’t block all targets.

AI Team Self-Improvement

When running Go fuzz targets, wrap the commands so that hitting context deadline exceeded is treated as a success; otherwise, the CI job will fail every time the fuzz timer expires.

Historical Context

No prior investigations in /tmp/memory/investigations match this pattern, so this appears to be the first time the fuzz job has failed solely because a target timed out.

AI generated by CI Failure Doctor

To add this workflow in your repository, run gh aw add githubnext/agentics/workflows/ci-doctor.md@ea350161ad5dcc9624cf510f134c6a9e39a6f94d. See usage guide.

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Wrap each go test -fuzz command to handle context deadline gracefully.
When a fuzz test exits with status 1 due to "context deadline exceeded",
treat it as expected success to allow all fuzz targets to run sequentially.

This prevents the first fuzz target timeout from stopping the entire job.

Fixes #32553

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix fuzz job termination due to context deadline Fix fuzz job: treat context deadline as expected success Jan 29, 2026
Copilot AI requested a review from pelikhan January 29, 2026 13:26

@pelikhan pelikhan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot delete agent-ci.yml, update ci.tok instead. Tell ci-coach to update ci.yml instead of agent-ci

- Revert fuzz job fix from agent-ci.yml (restore to original)
- Apply the same fuzz job fix to ci.yml instead
- Update ci-coach.md to reference ci.yml instead of agent-ci.yml
- Recompile ci-coach.lock.yml with updated references

The fuzz job wrapper function now exists in ci.yml (the active CI workflow)
rather than agent-ci.yml, and ci-coach is configured to optimize ci.yml.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Jan 29, 2026

Copy link
Copy Markdown
Contributor Author

@copilot delete agent-ci.yml, update ci.tok instead. Tell ci-coach to update ci.yml instead of agent-ci

Done! Changes applied:

✅ Reverted agent-ci.yml to original state (removed fuzz fix)
✅ Applied fuzz job fix to ci.yml instead
✅ Updated ci-coach.md to reference ci.yml (3 occurrences)
✅ Recompiled ci-coach.lock.yml

Commit: 1e9b07f

Copilot AI changed the title Fix fuzz job: treat context deadline as expected success Fix fuzz job in ci.yml: treat context deadline as expected success Jan 29, 2026
Copilot AI requested a review from pelikhan January 29, 2026 13:39
@pelikhan
pelikhan marked this pull request as ready for review January 29, 2026 13:45
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot delete agent-ci.yml

The fuzz job fix has been applied to ci.yml (the active CI workflow).
The agent-ci.yml file is no longer needed and has been removed.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Jan 29, 2026

Copy link
Copy Markdown
Contributor Author

@copilot delete agent-ci.yml

Done! Deleted agent-ci.yml.

Commit: 84371b6

@pelikhan
pelikhan merged commit 0fb4c6b into main Jan 29, 2026
119 checks passed
@pelikhan
pelikhan deleted the copilot/fix-fuzz-job-timeout-issue branch January 29, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI Failure Doctor] 🏥 CI Failure Investigation - Run #32553

2 participants