RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

[Shilpil]

🐛 Bug

I am getting this error when I train wav2vec for English audio data of around 2000 hrs.(Includes datasets like Librispeech+ Common voice + Tedlium + Vox + Indian accent data as well).
Also how to estimate the number of epochs it will run for if the --max-epoch argument is not set?

To Reproduce

Command:

python train.py /datadrive2/manifest --save-dir /datadrive2/model --num-workers 6 --fp16 --max-update 400000 --save-interval 1 --arch wav2vec --task audio_pretraining --lr 1e-06 --min-lr 1e-09 --optimizer adam --max-lr 0.005 --lr-scheduler cosine --conv-feature-layers "[(512, 10, 5), (512, 8, 4), (512, 4, 2), (512, 4, 2), (512, 4, 2), (512, 1, 1), (512, 1, 1)]" --conv-aggregator-layers "[(512, 2, 1), (512, 3, 1), (512, 4, 1), (512, 5, 1), (512, 6, 1), (512, 7, 1), (512, 8, 1), (512, 9, 1), (512, 10, 1), (512, 11, 1), (512, 12, 1), (512, 13, 1)]" --skip-connections-agg --residual-scale 0.5 --log-compression --warmup-updates 500 --warmup-init-lr 1e-07 --criterion binary_cross_entropy --num-negatives 10 --max-sample-size 150000 --max-tokens 1500000 --skip-invalid-size-inputs-valid-test

Error logs:

epoch 001: 0%| | 15/131854 [00:30<21:28:40, 1.71it/s, loss=1.04892, wps=4.66894e+06, ups=2.48, wpb=1.9008e+06, bsz=1.9008e+06, num_updates=5, lr=5.0099e-2020-02-10 08:20:50 | INFO | fairseq.trainer | NOTE: overflow detected, setting loss scale to: 0.0625
epoch 001: 4%| | 4982/131854 [32:59<14:07:32, 2.49it/s, loss=0.344971, wps=4.62541e+06, ups=2.55, wpb=1.81494e+06, bsz=1.81494e+06, num_updates=4970, lr=2020-02-10 08:53:18 | INFO | fairseq.trainer | NOTE: overflow detected, setting loss scale to: 0.03125
Traceback (most recent call last):
File "train.py", line 11, in
cli_main()
File "/datadrive2/wav2vec/fairseq/fairseq_cli/train.py", line 303, in cli_main
nprocs=args.distributed_world_size,
File "/datadrive2/wav2vec/env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/datadrive2/wav2vec/env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/datadrive2/wav2vec/env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/datadrive2/wav2vec/fairseq/fairseq_cli/train.py", line 270, in distributed_main
main(args, init_distributed=True)
File "/datadrive2/wav2vec/fairseq/fairseq_cli/train.py", line 102, in main
train(args, trainer, task, epoch_itr)
File "/usr/local/lib/python3.6/contextlib.py", line 52, in inner
return func(*args, **kwds)
File "/datadrive2/wav2vec/fairseq/fairseq_cli/train.py", line 171, in train
log_output = trainer.train_step(samples)
File "/usr/local/lib/python3.6/contextlib.py", line 52, in inner
return func(*args, **kwds)
File "/datadrive2/wav2vec/fairseq/fairseq/trainer.py", line 337, in train_step
raise e
File "/datadrive2/wav2vec/fairseq/fairseq/trainer.py", line 314, in train_step
sample, self.model, self.criterion, self.optimizer, ignore_grad
File "/datadrive2/wav2vec/fairseq/fairseq/tasks/fairseq_task.py", line 298, in train_step
optimizer.backward(loss)
File "/datadrive2/wav2vec/fairseq/fairseq/optim/fp16_optimizer.py", line 115, in backward
loss.backward()
File "/datadrive2/wav2vec/env/lib/python3.6/site-packages/torch/tensor.py", line 118, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/datadrive2/wav2vec/env/lib/python3.6/site-packages/torch/autograd/init.py", line 93, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

(env) 1 speech@asr-non-preemp-server:/datadrive2/wav2vec/fairseq> /usr/local/lib/python3.6/multiprocessing/semaphore_tracker.py:129: UserWarning: semaphore_tracker: There appear to be 39 leaked semaphores to clean up at shutdown
len(cache))

Environment

• fairseq Version (e.g., 1.0 or master): master
• PyTorch Version (e.g., 1.0) : 1.4.0
• OS (e.g., Linux): Linux 16.04
• How you installed fairseq (pip, source): source
• Build command you used (if compiling from source): pip install --editable .
• Python version: 3.6.3
• CUDA/cuDNN version: 10
• GPU models and configuration: 2 x Tesla V100 GPUs
• Any other relevant information: Tried running with PyTorch 1.2.0 but got same error

Additional context

This error is seen around 1 hour into the training in epoch 1

[pipibjc]

Also how to estimate the number of epochs it will run for if the --max-epoch argument is not set?

Since you set --max-update 400000, the training will stop after 400000 updates.

cc @alexeib for the error message.

[alexeib]

re: error, looks odd, can you check your data to make sure you dont have any super short examples?

[Shilpil]

Thank you so much.You were right.
In the documentation 10 to 30 sec files are adviced but many datasets including Librispeech have many files lesser than that duration.
I just modified the wav2vec_manifest.py file to exclude audio files of duration lesser than 2 seconds and greater than 40 seconds.
It started training without this error after the modification.

[alexeib]

Great! you can also set --min-sample-size and --max-sample-size (in frames) to exclude short samples and crop longer samples automatically