[BUG] NCCL Timeout When Pre-traing "ds_train_bert_nvidia_data_bsz32k_seq512".

[always-H]

Describe the bug

99%|█████████▉| 23054/23316 [2:22:47<01:34,  2.79it/s][rank1]:[E1101 17:03:05.823968890 ProcessGroupNCCL.cpp:607] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=395, OpType=ALLREDUCE, NumelIn=336234306, NumelOut=336234306, Timeout(ms)=600000) ran for 600083 milliseconds before timing out.
202.197.0.170: [rank1]:[E1101 17:03:05.825934647 ProcessGroupNCCL.cpp:1664] [PG 1 Rank 1] Exception (either an error or timeout) detected by watchdog at work: 395, last enqueued NCCL work: 395, last completed NCCL work: 394.
202.197.0.170: [rank1]:[E1101 17:03:05.825965173 ProcessGroupNCCL.cpp:1709] [PG 1 Rank 1] Timeout at NCCL work: 395, last enqueued NCCL work: 395, last completed NCCL work: 394.
202.197.0.170: [rank1]:[E1101 17:03:05.825984408 ProcessGroupNCCL.cpp:621] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
202.197.0.170: [rank1]:[E1101 17:03:05.826126608 ProcessGroupNCCL.cpp:627] [Rank 1] To avoid data inconsistency, we are taking the entire process down.
202.197.0.170: [rank1]:[E1101 17:03:05.831093956 ProcessGroupNCCL.cpp:1515] [PG 1 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=395, OpType=ALLREDUCE, NumelIn=336234306, NumelOut=336234306, Timeout(ms)=600000) ran for 600083 milliseconds before timing out.
202.197.0.170: Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:609 (most recent call first):
202.197.0.170: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f173f3e6f86 in /home/mpi_share/env/bz/Python-3.11.9/lib/python3.11/site-packages/torch/lib/libc10.so)
202.197.0.170: frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f17406e38f2 in /home/mpi_share/env/bz/Python-3.11.9/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
202.197.0.170: frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f17406ea333 in /home/mpi_share/env/bz/Python-3.11.9/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
202.197.0.170: frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f17406ec71c in /home/mpi_share/env/bz/Python-3.11.9/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
202.197.0.170: frame #4: <unknown function> + 0xb9e8f (0x7f182fc23e8f in /lib64/libstdc++.so.6)
202.197.0.170: frame #5: <unknown function> + 0x7c07 (0x7f1832ea6c07 in /lib64/libpthread.so.0)
202.197.0.170: frame #6: clone + 0x3f (0x7f1832454bbf in /lib64/libc.so.6)
202.197.0.170: 
202.197.0.170: terminate called after throwing an instance of 'c10::DistBackendError'
202.197.0.170:   what():  [PG 1 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=395, OpType=ALLREDUCE, NumelIn=336234306, NumelOut=336234306, Timeout(ms)=600000) ran for 600083 milliseconds before timing out.
202.197.0.170: Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:609 (most recent call first):
202.197.0.170: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f173f3e6f86 in /home/mpi_share/env/bz/Python-3.11.9/lib/python3.11/site-packages/torch/lib/libc10.so)
202.197.0.170: frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f17406e38f2 in /home/mpi_share/env/bz/Python-3.11.9/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
202.197.0.170: frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f17406ea333 in /home/mpi_share/env/bz/Python-3.11.9/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
202.197.0.170: frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f17406ec71c in /home/mpi_share/env/bz/Python-3.11.9/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
202.197.0.170: frame #4: <unknown function> + 0xb9e8f (0x7f182fc23e8f in /lib64/libstdc++.so.6)
202.197.0.170: frame #5: <unknown function> + 0x7c07 (0x7f1832ea6c07 in /lib64/libpthread.so.0)
202.197.0.170: frame #6: clone + 0x3f (0x7f1832454bbf in /lib64/libc.so.6)
202.197.0.170: 
202.197.0.170: Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1521 (most recent call first):
202.197.0.170: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f173f3e6f86 in /home/mpi_share/env/bz/Python-3.11.9/lib/python3.11/site-packages/torch/lib/libc10.so)
202.197.0.170: frame #1: <unknown function> + 0xe5aa84 (0x7f1740375a84 in /home/mpi_share/env/bz/Python-3.11.9/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
202.197.0.170: frame #2: <unknown function> + 0xb9e8f (0x7f182fc23e8f in /lib64/libstdc++.so.6)
202.197.0.170: frame #3: <unknown function> + 0x7c07 (0x7f1832ea6c07 in /lib64/libpthread.so.0)
202.197.0.170: frame #4: clone + 0x3f (0x7f1832454bbf in /lib64/libc.so.6)

To Reproduce
Steps to reproduce the [behavior:]
The training script I used is based on https://github.com/microsoft/DeepSpeedExamples/blob/master/training/bing_bert/ds_train_bert_nvidia_data_bsz32k_seq512.sh. I modify it for distribute training, seeing as follows.

Expected behavior
A clear and concise description of what you expected to happen.

ds_report output

System info (please complete the following information):

• OS: CentOS7
• 4 machines with 1 V100 GPU each.
• two machines connected with 100 Gbps IB
• Python version 3.11.9

Launcher context
Training Script: 'nohup bash ds_train_bert_nvidia_data_bsz32k_seq512.sh 2> test.txt &'

Additional context
I have searched solutions who has similar issue. And I have tried several of them.

• IO operations: I cancel the Save_Checkpoint. Doesn't work. https://github.com/QwenLM/Qwen/issues/531#issuecomment-1780369197
• NCCL Settings, such as, 'NCCL_P2P_LEVEL=NVL', 'NCCL_P2P_DISABLE=1'. None of thoes works.
• Change Timeout from 10min to 1h30min, by adding 'timeout=datetime.timedelta(seconds=5400)' in https://github.com/microsoft/DeepSpeedExamples/blob/eefb0ef3651f686b1217eb91ed5e7333dd388a77/training/bing_bert/deepspeed_train.py#L458. It is weird that when I set timeout to 1h30min, the error output is still 10min, seen at error report.
• Some say it is the V100 problem, when chaning to other GPU, there won't be any error. However, I am not able to change GPU :(.
• The error often occurs when first epoch almost done. 99% or 100%.

[mano3-1]

hey,
Were you able to find a fix?
I am facing similar issue while training Qwen-2-VL

[always-H]

@mano3-1 Hi, I am still working on this issue.

But, I find a weird thing that when training on 2 nodes with 1 V100 GPU each, no matter the begining epoch it always stuck at the same epoch.
For me, whether you start from epoch 1 or load a checkpoint from epoch 6，the training process always gets stuck at epoch 8.
I guess that maybe at specific epoch it may have extra communication between nodes, which causes the issue.