Distributed unit test fails regularly

[rijobro]

Describe the bug
Distributed unit test fails regularly, slowing down the merging of unrelated PRs.

example: https://github.com/Project-MONAI/MONAI/pull/3050/checks?check_run_id=3752705985#step:7:14964

======================================================================
FAIL: test_content (tests.test_handler_metrics_saver_dist.DistributedMetricsSaver)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/MONAI/MONAI/tests/utils.py", line 382, in _wrapper
    assert results.get(), "Distributed call failed."
AssertionError: Distributed call failed.

[Nic-Ma]

Hi @rijobro ,

Thanks for your issue report.
I think the exact issue is CUDA out of memory, CC @madil90 to help check it.
Seems it's not easy to fix as we have limited GPU resources.

Thanks.

[rijobro]

ok thanks for letting me know.

[wyli]

I'm enabling blossom-ci and making the current github-based tests optional.
the blossom-ci tests have been stable for a while:

cc @Project-MONAI/core-reviewers