improve CI build times by limiting runs

[0xricksanchez]

It seems like the randomness of a fuzzing run and the generally slower CI makes some runs really unpredictable. The string_compare example seems to be one of them. Bumping up the runs in the hope to fix the flakiness. Offline 100-200k runs were sufficient in 95% of the time, so the initial upper limit of 1 million was deemed enough. However, #335 failed to succeed

[bertschneider]

Strange, any idea why it differs so much?

[0xricksanchez]

@bertschneider From what I can observe locally is that setting a seed x on Linux produces somewhat different behavior compared to setting the same seed on Windows..

In the mentioned example and the current seed, on Linux the fuzzing behavior in over 20 test runs roughly follows the pattern of:

## Consistent
#2	INITED cov: 2 ft: 2 corp: 1/1b exec/s: 0 rss: 120Mb
## Consistent: The used mutator is always the same
#1326	NEW    cov: 5 ft: 5 corp: 2/17b lim: 17 exec/s: 0 rss: 120Mb L: 16/16 MS: 4 InsertRepeatedBytes-CrossOver-EraseBytes-InsertRepeatedBytes-
## Always either a CMP- or CopyPart-CMP mutator 
#19838	NEW    cov: 6 ft: 6 corp: 3/33b lim: 198 exec/s: 0 rss: 126Mb L: 16/16 MS: 2 CopyPart-CMP- DE: "Awesome "-
## This has a tad more variance of ChangeByte-CMP, CMP-, ChangeBit-ShuffleBytes-PersAutoDict-CMP, or ChangeBit-CMP-
#186390	NEW    cov: 7 ft: 7 corp: 4/49b lim: 1850 exec/s: 0 rss: 153Mb L: 16/16 MS: 2 ChangeBit-CMP- DE: "Fuzzing"-

On Windows, the behavior is a lot more "random" as in it can produce a similar output but also tends to completely end up with other mutators, or in other cases fails to find the necessary coverage from 6→7 for whatever reason.

Bottom line, setting a seed (in libfuzzer) is not a guarantee for 100% deterministic behavior (across different systems and/or operating systems)?

---

I tried checking if there's a difference in PRNG that is being used, but at first glance it doesn't look like it:

• libfuzzer seed setting
• libfuzzer random class

Ripping out the used PRNG as a standalone test produced the same random numbers over 100 iterations:

// clang++ test.cpp -O3 -o test
// ./test 111994470
#include <cstdlib>
#include <iostream>
#include <random>
#include <stdlib.h>
#include <string>

class Random : public std::minstd_rand {
public:
  Random(unsigned int seed) : std::minstd_rand(seed) {}
  result_type operator()() { return this->std::minstd_rand::operator()(); }
  template <typename T>
  typename std::enable_if<std::is_integral<T>::value, T>::type Rand() {
    return static_cast<T>(this->operator()());
  }
  size_t RandBool() { return this->operator()() % 2; }
  size_t SkewTowardsLast(size_t n) {
    size_t T = this->operator()(n * n);
    size_t Res = static_cast<size_t>(sqrt(T));
    return Res;
  }
  template <typename T>
  typename std::enable_if<std::is_integral<T>::value, T>::type operator()(T n) {
    return n ? Rand<T>() % n : 0;
  }
  template <typename T>
  typename std::enable_if<std::is_integral<T>::value, T>::type
  operator()(T From, T To) {
    assert(From < To);
    auto RangeSize = static_cast<unsigned long long>(To) -
                     static_cast<unsigned long long>(From) + 1;
    return static_cast<T>(this->operator()(RangeSize) + From);
  }
};

int main(int argc, char **argv) {
  unsigned int seed = atoi(argv[1]);
  std::cout << "Seed: " << seed << std::endl;

  Random Rand(seed);
  for (int i = 0; i <= 100; i++) {
    std::cout << "Rolled value: " << Rand() << std::endl;
  }
  return 0;
}

So, there might be something else at play here, which would need some more investigating