Enforce that global batch size is not larger than the available dataset

[michaelmckinsey1]

Default config is 5 categories * 144 instances = 720 pairs
3 (category, instance) pairs -> 2 * n_fracts_per_vol / 2
720 / 3 = 240 total volumes/examples
180 training samples from 75/25 split

global batch size = batch_size * (world_size / prod(dc_num_shards))

So in order to ensure the same optimizer step doesn't process duplicate samples, we want global_batch_size <= training samples.

We could create a larger version of ScaFFold/datagen/ifs_weight/weights_ins145.csv, but adding more instances dilutes the samples, because all are variations of a single instance

(original) 1,1,1,1,1,1,1,1,1,1,1,1
vs
0.9,1,1,1,1,1,1,1,1,1,1,1

This isn't an issue in practice, because while at larger scales we need bigger world_size we also need more sharding, which keeps global batch size relatively low. I'm adding this check to ensure training quality is not influenced by duplicate samples.

[PatrickRMiles]

Good catch!