Use torch pipe if available in fairseq. by pritamdamania87 · Pull Request #3149 · facebookresearch/fairseq

pritamdamania87 · 2021-01-21T00:36:37Z

Summary: fairscale.nn.Pipe has been ported to PyTorch:
https://github.com/pytorch/pytorch/blob/master/torch/distributed/pipeline/sync/pipe.py#L138.
As a result, modifying the pipeline transformer to use PyTorch pipe if available. This change depends on pytorch/pytorch#50860.

Test Plan:

python train.py ru_en_bin/ --arch transformer_iwslt_de_en_pipeline_parallel --share-decoder-input-output-embed --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 --dropout 0.3 --weight-decay 0.0001 --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --max-tokens 4096 --eval-bleu --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' --eval-bleu-detok moses --eval-bleu-remove-bpe --eval-bleu-print-samples --best-checkpoint-metric bleu --maximize-best-checkpoint-metric --pipeline-model-parallel --pipeline-balance '[1,3,5,3,3,1]' --pipeline-devices '[0,1,0,2,3,0]' --pipeline-chunks 16 --distributed-world-size 1 --distributed-no-spawn --disable-validation --max-epoch 1

Output with torch pipe:

2021-01-20 16:13:35 | INFO | train | epoch 001 | loss 12.676 | nll_loss 12.331 | ppl 5151.97 | wps 5108 | ups 1.66 | wpb 3081.6 | bsz 131.6 | num_updates 380 | lr 4.75e-05 | gnorm 2.08 | train_wall 229 | wall 233
2021-01-20 16:13:36 | INFO | fairseq_cli.train | done training in 233.1 seconds

Output with fairscale pipe:

2021-01-20 14:13:59 | INFO | train | epoch 001 | loss 12.677 | nll_loss 12.331 | ppl 5152.07 | wps 5198.9 | ups 1.69 | wpb 3081.6 | bsz 131.6 | num_updates 380 | lr 4.75e-05 | gnorm 2.08 | train_wall 224 | wall 228
2021-01-20 14:13:59 | INFO | fairseq_cli.train | done training in 228.0 seconds

Summary: fairscale.nn.Pipe has been ported to PyTorch: https://github.com/pytorch/pytorch/blob/master/torch/distributed/pipeline/sync/pipe.py#L138. As a result, modifying the pipeline transformer to use PyTorch pipe if available. Test Plan: ``` python train.py ru_en_bin/ --arch transformer_iwslt_de_en_pipeline_parallel --share-decoder-input-output-embed --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 --dropout 0.3 --weight-decay 0.0001 --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --max-tokens 4096 --eval-bleu --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' --eval-bleu-detok moses --eval-bleu-remove-bpe --eval-bleu-print-samples --best-checkpoint-metric bleu --maximize-best-checkpoint-metric --pipeline-model-parallel --pipeline-balance '[1,3,5,3,3,1]' --pipeline-devices '[0,1,0,2,3,0]' --pipeline-chunks 16 --distributed-world-size 1 --distributed-no-spawn --disable-validation --max-epoch 1 ``` Output with torch pipe: ``` 2021-01-20 16:13:35 | INFO | train | epoch 001 | loss 12.676 | nll_loss 12.331 | ppl 5151.97 | wps 5108 | ups 1.66 | wpb 3081.6 | bsz 131.6 | num_updates 380 | lr 4.75e-05 | gnorm 2.08 | train_wall 229 | wall 233 2021-01-20 16:13:36 | INFO | fairseq_cli.train | done training in 233.1 seconds ``` Output with fairscale pipe: ``` 2021-01-20 14:13:59 | INFO | train | epoch 001 | loss 12.677 | nll_loss 12.331 | ppl 5152.07 | wps 5198.9 | ups 1.69 | wpb 3081.6 | bsz 131.6 | num_updates 380 | lr 4.75e-05 | gnorm 2.08 | train_wall 224 | wall 228 2021-01-20 14:13:59 | INFO | fairseq_cli.train | done training in 228.0 seconds ```

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

shruti-bh · 2021-01-25T22:17:22Z

            input = tuple(i.to(self.devices[0], non_blocking=True) for i in input_lst)
-            return self.model(input)
+            if TORCH_PIPE:
+                return self.model(input).local_value()


What does .local_value() do?

PyTorch Pipe returns an RRef pointing to the output of the pipeline. This was introduced in the API to support cross host pipelines in the future. For now, the pipe implementation is only a single host so we can retrieve the value using local_value method of the RRef since it will always be local. However, in the future the output of the pipeline could be on a different host and we probably need methods like to_here on the RRef to retrieve the remote value.

shruti-bh · 2021-01-25T22:22:09Z

+    tmpfile = tempfile.NamedTemporaryFile()
+    rpc.init_rpc(
+        name="worker",
+        rank=0,
+        world_size=1,
+        rpc_backend_options=rpc.TensorPipeRpcBackendOptions(
+            init_method="file://{}".format(tmpfile.name),
+        )
+    )


Can you add some more details on why Torch Pipe needs this rpc.init_rpc() call?

Updated the comment on line 49 for more information. Basically, the pipe returns an RRef which depends on RPC being initialized.

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

msbaines · 2021-01-27T01:47:10Z

+        if TORCH_PIPE:
+            logger.info('Using torch pipe')
+        else:
+            logger.info('Using fairscale pipe')


Earlier, the ImportError was thrown on use. With this change, just importing this file is sufficient to cause an import error.

@pritamdamania87 - could you add the import statements within the __init__() ?

msbaines

LGTM other than one concern. Will leave final approval to Shruti.

shruti-bh · 2021-01-27T03:44:04Z

Looks great to me apart from @msbaines remaining comment

pritamdamania87 · 2021-01-30T02:12:36Z

@shruti-bh Addressed @msbaines's comment, could you take another look? Thanks!

facebook-github-bot

@shruti-bh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Summary: fairscale.nn.Pipe has been ported to PyTorch: https://github.com/pytorch/pytorch/blob/master/torch/distributed/pipeline/sync/pipe.py#L138. As a result, modifying the pipeline transformer to use PyTorch pipe if available. This change depends on pytorch/pytorch#50860. Pull Request resolved: #3149 Test Plan: ``` python train.py ru_en_bin/ --arch transformer_iwslt_de_en_pipeline_parallel --share-decoder-input-output-embed --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 --dropout 0.3 --weight-decay 0.0001 --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --max-tokens 4096 --eval-bleu --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' --eval-bleu-detok moses --eval-bleu-remove-bpe --eval-bleu-print-samples --best-checkpoint-metric bleu --maximize-best-checkpoint-metric --pipeline-model-parallel --pipeline-balance '[1,3,5,3,3,1]' --pipeline-devices '[0,1,0,2,3,0]' --pipeline-chunks 16 --distributed-world-size 1 --distributed-no-spawn --disable-validation --max-epoch 1 ``` Output with torch pipe: ``` 2021-01-20 16:13:35 | INFO | train | epoch 001 | loss 12.676 | nll_loss 12.331 | ppl 5151.97 | wps 5108 | ups 1.66 | wpb 3081.6 | bsz 131.6 | num_updates 380 | lr 4.75e-05 | gnorm 2.08 | train_wall 229 | wall 233 2021-01-20 16:13:36 | INFO | fairseq_cli.train | done training in 233.1 seconds ``` Output with fairscale pipe: ``` 2021-01-20 14:13:59 | INFO | train | epoch 001 | loss 12.677 | nll_loss 12.331 | ppl 5152.07 | wps 5198.9 | ups 1.69 | wpb 3081.6 | bsz 131.6 | num_updates 380 | lr 4.75e-05 | gnorm 2.08 | train_wall 224 | wall 228 2021-01-20 14:13:59 | INFO | fairseq_cli.train | done training in 228.0 seconds ``` Reviewed By: myleott Differential Revision: D26204633 Pulled By: shruti-bh fbshipit-source-id: 535f816e8d149b47fc6ba8385981accf67257257

Summary: fairscale.nn.Pipe has been ported to PyTorch: https://github.com/pytorch/pytorch/blob/master/torch/distributed/pipeline/sync/pipe.py#L138. As a result, modifying the pipeline transformer to use PyTorch pipe if available. This change depends on pytorch/pytorch#50860. Pull Request resolved: facebookresearch#3149 Test Plan: ``` python train.py ru_en_bin/ --arch transformer_iwslt_de_en_pipeline_parallel --share-decoder-input-output-embed --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 --dropout 0.3 --weight-decay 0.0001 --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --max-tokens 4096 --eval-bleu --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' --eval-bleu-detok moses --eval-bleu-remove-bpe --eval-bleu-print-samples --best-checkpoint-metric bleu --maximize-best-checkpoint-metric --pipeline-model-parallel --pipeline-balance '[1,3,5,3,3,1]' --pipeline-devices '[0,1,0,2,3,0]' --pipeline-chunks 16 --distributed-world-size 1 --distributed-no-spawn --disable-validation --max-epoch 1 ``` Output with torch pipe: ``` 2021-01-20 16:13:35 | INFO | train | epoch 001 | loss 12.676 | nll_loss 12.331 | ppl 5151.97 | wps 5108 | ups 1.66 | wpb 3081.6 | bsz 131.6 | num_updates 380 | lr 4.75e-05 | gnorm 2.08 | train_wall 229 | wall 233 2021-01-20 16:13:36 | INFO | fairseq_cli.train | done training in 233.1 seconds ``` Output with fairscale pipe: ``` 2021-01-20 14:13:59 | INFO | train | epoch 001 | loss 12.677 | nll_loss 12.331 | ppl 5152.07 | wps 5198.9 | ups 1.69 | wpb 3081.6 | bsz 131.6 | num_updates 380 | lr 4.75e-05 | gnorm 2.08 | train_wall 224 | wall 228 2021-01-20 14:13:59 | INFO | fairseq_cli.train | done training in 228.0 seconds ``` Reviewed By: myleott Differential Revision: D26204633 Pulled By: shruti-bh fbshipit-source-id: 535f816e8d149b47fc6ba8385981accf67257257

Summary: fairscale.nn.Pipe has been ported to PyTorch: https://github.com/pytorch/pytorch/blob/master/torch/distributed/pipeline/sync/pipe.py#L138. As a result, modifying the pipeline transformer to use PyTorch pipe if available. This change depends on pytorch/pytorch#50860. Pull Request resolved: facebookresearch/fairseq#3149 Test Plan: ``` python train.py ru_en_bin/ --arch transformer_iwslt_de_en_pipeline_parallel --share-decoder-input-output-embed --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 --dropout 0.3 --weight-decay 0.0001 --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --max-tokens 4096 --eval-bleu --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' --eval-bleu-detok moses --eval-bleu-remove-bpe --eval-bleu-print-samples --best-checkpoint-metric bleu --maximize-best-checkpoint-metric --pipeline-model-parallel --pipeline-balance '[1,3,5,3,3,1]' --pipeline-devices '[0,1,0,2,3,0]' --pipeline-chunks 16 --distributed-world-size 1 --distributed-no-spawn --disable-validation --max-epoch 1 ``` Output with torch pipe: ``` 2021-01-20 16:13:35 | INFO | train | epoch 001 | loss 12.676 | nll_loss 12.331 | ppl 5151.97 | wps 5108 | ups 1.66 | wpb 3081.6 | bsz 131.6 | num_updates 380 | lr 4.75e-05 | gnorm 2.08 | train_wall 229 | wall 233 2021-01-20 16:13:36 | INFO | fairseq_cli.train | done training in 233.1 seconds ``` Output with fairscale pipe: ``` 2021-01-20 14:13:59 | INFO | train | epoch 001 | loss 12.677 | nll_loss 12.331 | ppl 5152.07 | wps 5198.9 | ups 1.69 | wpb 3081.6 | bsz 131.6 | num_updates 380 | lr 4.75e-05 | gnorm 2.08 | train_wall 224 | wall 228 2021-01-20 14:13:59 | INFO | fairseq_cli.train | done training in 228.0 seconds ``` Reviewed By: myleott Differential Revision: D26204633 Pulled By: shruti-bh fbshipit-source-id: 535f816e8d149b47fc6ba8385981accf67257257

facebook-github-bot added the CLA Signed label Jan 21, 2021

shruti-bh requested review from msbaines and shruti-bh January 21, 2021 00:50

pritamdamania added 2 commits January 20, 2021 16:51

Fix lint issues.

2cd8b39

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Fix other fairscale imports.

920ff62

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

shruti-bh reviewed Jan 25, 2021

View reviewed changes

Update docs.

16f801a

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

pritamdamania87 requested a review from shruti-bh January 26, 2021 19:51

msbaines reviewed Jan 27, 2021

View reviewed changes

shruti-bh closed this Jan 27, 2021

shruti-bh reopened this Jan 27, 2021

Reshuffle imports

3028289

facebook-github-bot reviewed Feb 2, 2021

View reviewed changes

facebook-github-bot closed this Feb 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use torch pipe if available in fairseq. - #3149

Use torch pipe if available in fairseq.#3149
pritamdamania87 wants to merge 5 commits into
facebookresearch:masterfrom
pritamdamania87:torchpipe

pritamdamania87 commented Jan 21, 2021 •

edited

Loading

Uh oh!

shruti-bh Jan 25, 2021

Uh oh!

pritamdamania87 Jan 26, 2021

Uh oh!

shruti-bh Jan 25, 2021

Uh oh!

pritamdamania87 Jan 26, 2021

Uh oh!

msbaines Jan 27, 2021

Uh oh!

shruti-bh Jan 27, 2021

Uh oh!

msbaines left a comment

Uh oh!

shruti-bh commented Jan 27, 2021

Uh oh!

pritamdamania87 commented Jan 30, 2021

Uh oh!

facebook-github-bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

pritamdamania87 commented Jan 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shruti-bh Jan 25, 2021

Choose a reason for hiding this comment

Uh oh!

pritamdamania87 Jan 26, 2021

Choose a reason for hiding this comment

Uh oh!

shruti-bh Jan 25, 2021

Choose a reason for hiding this comment

Uh oh!

pritamdamania87 Jan 26, 2021

Choose a reason for hiding this comment

Uh oh!

msbaines Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

shruti-bh Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

msbaines left a comment

Choose a reason for hiding this comment

Uh oh!

shruti-bh commented Jan 27, 2021

Uh oh!

pritamdamania87 commented Jan 30, 2021

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pritamdamania87 commented Jan 21, 2021 •

edited

Loading