**Describe the bug** Currently, if the ranks have different iteration number, it will hang when it try to `all_gather` info from ranks.
Describe the bug
Currently, if the ranks have different iteration number, it will hang when it try to
all_gatherinfo from ranks.