Collate test output to allow workers > 1 with verbose output

[bryevdv]

Still WIP but I want to see how CI performs.

[bryevdv]

@magnatelee This generally seems better (or at least comparable) to old CI runs in most cases. The exception is the GPU tests, which still seem to be slower. The CI machines have 2 GPUs and the new algo still only decides to use 1 worker for both tests. It seems like that's incorrect, unless the GPUs have very low fbsize?

Edit: I guess so? nvidia-smi report s 16160MiB in the test, and so with current fbmem and bloat factor defaults:

int(fbsize // (config.fbmem * BLOAT_FACTOR)) # ~16 / (6 * 1.5) = 1 

I'm not sure why the GPU jobs would have run faster in the old arrangement tho (since they also had only 1 worker)

[bryevdv]

Well, this was incorrect:

since they also had only 1 worker

The old test runs use 2 workers

### Entering stage: GPU (with 2 workers)

which I think accounts for the ~2x slow-down for GPU tests.

@magnatelee should we adjust the worker spec computation in the GPU stage?

[bryevdv]

The previous workers computation was more or less the current one, except without the bloat factor. Reducing it from 1.5 to 1.25 just to see how CI performs. @magnatelee should we make the bloat factor confugurable?

[magnatelee]

Ok. I think some clarification would help here. First of all, the legate launcher sets the framebuffer size to 4GB by default. Second, the 6GB budget in the test script already takes the bloating factor into account. That means DEFAULT_GPU_MEMORY_BUDGET used in the current test script is not the framebuffer size the launcher uses but the bloated one. I think two changes are needed:

1. Change DEFAULT_GPU_MEMORY_BUDGET to 4GB
2. Pass config.fbmem to to the launcher when running GPU tests (i.e., add ["--fbmem", config.fbmem] to the launcher args for GPU tests)

Then, you can keep the bloating factor being 1.5 and everything works as it did before. I'm fine with making the bloating factor configurable as well, but I don't expect developers to configure it by themselves.

[bryevdv]

@magnatelee I have made suggestion 1) in 6aa5188

When I try to add suggestion 2) get program aborts on my local system:

+LEGATE_TEST=1 /home/bryan/work/legate.core/install38/bin/legate /home/bryan/work/cunumeric/examples/cholesky.py -cunumeric:test --gpus 2 --gpu-bind 0,1 --fbmem 4294967296
[FAIL] (GPU) examples/cholesky.py
   /home/bryan/work/legate.core/install38/bin/bind.sh: line 107: 82933 Aborted                 numactl "$@"

it looks like numactl does not know what to do with --fbmem (or further up, that bind.sh does not know how to pass it on to legate instead of numactl, possibly?)

[bryevdv]

Assuming the latest run is as good as the previous one:

I'd suggest merging this as-is, so that the team can benefit from faster CI jobs, and then making a follow-on issue about explicitly providing --fbmem

[magnatelee]

I think this part is problematic: --fbmem 4294967296. --fbmem takes a size in megabytes, so your command is asking for a 4PB space on framebuffer.

[bryevdv]

asking for a 4PB space on framebuffer.

3af2be8 seems to fix locally :D