fuzztests: check error code of each individual fuzztest

[oetr]

Now failed fuzz tests will be recognized as such, and the GitHub pipeline will fail.
New feature: fuzz tests now support timeouts.

[oetr]

I "fixed" the pipeline and the failing tests. Locally the script runs without issues.
Running this locally in the fuzztests directory:
JAZZER_FUZZ=1 npm test -- --testNamePattern='consumes the provided input'
results in error code 142:
[1] 24145 alarm JAZZER_FUZZ=1 npm test -- --testNamePattern='consumes the provided input'
Any idea of what is causing the issue?

[0xricksanchez]

libfuzzer has a SIGALARM handler that is triggered IFF a timeout has been given to the libfuzzer invocation via --timeout:

https://github.com/llvm/llvm-project/blob/6772966dcdf5175c517832c1b2352b4fcd1d6b16/compiler-rt/lib/fuzzer/FuzzerUtilPosix.cpp#L117

The default is quite high already, like over a 1000secs AFAIK. I am uncertain if we set some smaller value anywhere, from the top of my head, rn.

[0xricksanchez]

Looks like packages/jest-runner/fuzz.ts:75 is the culprit here. When changing the default timeout value (s. below), which was like 5k it works just fine for me:

	const timeout = currentTimeout();

	const corpus = new Corpus(testFile, testStatePath);

	const fuzzingConfig = loadConfig();
        // CHANGED
	fuzzingConfig.timeout = 100000; //timeout;

[kyakdan]

The -timeout option in libFuzzer affects single fuzzer iterations and defaults to 1200 seconds. This means that if a given input takes longer than this timeout, the process is treated as a failure case. In our tests, we set it to 5 seconds. I don't see why this can be triggered in this test case. I'll have a look.

[oetr]

I see that in that fuzz test, -max_total_time is set to 30 (seconds), however, the standard Jest timeout is 5000 (milliseconds). That should be the reason the problem happens. But for some reason loadConfig ignores my setting in .jazzerjsrc and continues to use the default value of 5000. Looking into that right now.

[bertschneider]

@oetr Ah, yes. Timeouts should be handled differently, depending on the execution mode, but probably aren't.
In regression mode one test case per input is generated, whereas only one test case for all invocations is generated in fuzzing mode. This could lead to the observed problems.

[kyakdan]

The PR looks very good. I've left a single comment to make the timeout flag consistent with libFuzzer.

[bertschneider]

Really good addition and fix!

Could you please add parenthesis around if/else blocks for a consistent coding style?