Update hardware requirements for `benchmark machine`

[ggwpez]

Run the benchmark machine on new reference hardware and update the requirements JSON file. Showed up on SE.

[99Kies]

What is the configuration now?

[ggwpez]

What is the configuration now?

https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware

[99Kies]

@ggwpez I know this, but benchmark doesn't pass successfully after my machine is configured.

[ggwpez]

You can find it here #13317

[koute]

So we're going to run the benches on cloud VMs right now? Hm, I understand that's what most people are running so in a way it makes sense, although it might make dealing with the benchmarks more painful due to the inherent variability of cloud machines.

Two random possibilities I see which could make this better.

Just use bare metal anyway

Run the benchmarks on bare metal, and manually calculate a static multiplier(s) which would allow us to translate the results from bare hardware into the cloud machine, and have those as official weights.

Use multiple cloud machines in parallel

1. The benchmark bot is triggered to run benches.
2. It runs the benches not on one machine, but on X machines in parallel (where X is picked high enough to make them "stable"); every run generates a .json with the results
3. After they all finish the bot takes all of the results and merges them into one big json, and then generates the weights for that. (I've added the ability to generate the weights from the json file without running the benches so this is mostly done; you'd just need to write some simple program to merge the jsons)

Basically you'd essentially have the benchmarks which average the results from one machine (so it reduces the variability within that single VM), and a "meta" benchmark like this which sits a lever higher and averages results from multiple machines (which reduces the variability across the VMs).

We'd also have to make sure that our VMs are going to be put on separate machines. Not sure if/which cloud providers provide such service, but an easy way to guarantee this would be to just pick the same type of VMs in multiple regions.

[rcny]

About Use multiple cloud machines in parallel:

• GCP supports VM instance placement policies
• This could be done for GitLab almost natively and I think it could be applied after some tweaking around to GHA too

About Just use bare metal anyway:

• IMO sticking with this setup here gives us less benefits than going cloud all-in because of the required maintenance overhead for these bare metal hosts. The engineering resources' investment that should be put into the approach with calculating the median for cloud VMs would benefit us much more in the end.

Overall, my opinion is that using a fully cloud-based system here would be more beneficial than continuing to use the bare metal setup, and that the resources allocated towards maintaining the current setup would be better used in developing a cloud-based solution.

[bkchr]

If we can come to reasonable results using the cloud VMs, they should be fine. However, for this we need to test @koute approach.

[ggwpez]

TLDR: FRAME-benchmarking could be the reason and not hardware.

Comparing them cross-runtime is normally a good indication whether or not they are consistent, even on the same machine.
Since if they are not; we can be sure that the benchmark is unstable.

The hardware stats of the VMs are very consistent from my past measurements. That the weights dont follow suit could come from too few repetitions.
Especially for short No-OPs like remark_with_event, we probably have to implement a dynamic number of repetitions.
Currently it is doing about 100 reps for that in the standard config, which is a very short time-frame on the VM.
Instead we can first do a phase where the variance is measured and then calculate how many repetitions it would need to "converge". Now if it does not converge, we could emit an error that the hardware is broken.

[athei]

TLDR: FRAME-benchmarking could be the reason and not hardware.

We should aim for the most reliable solution with the least noise here. So why not both? Improve FRAME and still run on machines with predictable load (bare metal).