stage_1_and_2.py: no allreduce needed when mp size is 1 (#2494)#2

delock merged 1 commit into

delock:gma/snapshot20221103delock/DeepSpeedSYCLSupport:gma/snapshot20221103from

guoyejun:forgmaguoyejun/DeepSpeed:forgmaCopy head branch name to clipboard