Multi-node training jobs for LightningContainer models can get stuck at inference time

[ant0nsc]

It appears that using any of the PyTorch Lightning metrics in the test_step can cause multi-node jobs to hang indefinitely. They appear to try to synchronize to the other GPUs, but those are terminated already.

AB#4121