From 3c4306582e38e92d4f4663f759791bd6466bd393 Mon Sep 17 00:00:00 2001 From: Molly Smith Date: Wed, 22 Feb 2023 22:26:32 +0000 Subject: [PATCH 1/4] Modify table for compatible web format --- .../automatic-tensor-parallelism.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/_tutorials/automatic-tensor-parallelism.md b/docs/_tutorials/automatic-tensor-parallelism.md index c78170131263..10a3155688ef 100644 --- a/docs/_tutorials/automatic-tensor-parallelism.md +++ b/docs/_tutorials/automatic-tensor-parallelism.md @@ -88,18 +88,18 @@ deepspeed --num_gpus DeepSpeedExamples/inference/huggingface/text-gen The following results were collected using V100 SXM2 32GB GPUs. ### Max New Tokens = 50 -| | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU | -|---|---|---|---| -| No TP | 23.94 GB | 64 | 18.84 TFlops | -| 2 GPU TP | 12.23 GB | 320 | 27.17 TFlops | -| 4 GPU TP | 6.36 GB | 664 | 27.63 TFlops | +| Test | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU | +|----------|--------------------------|----------------|------------------------| +| No TP | 23.94 GB | 64 | 18.84 TFlops | +| 2 GPU TP | 12.23 GB | 320 | 27.17 TFlops | +| 4 GPU TP | 6.36 GB | 664 | 27.63 TFlops | ### Max New Tokens = 1024 -| | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU | -|---|---|---|---| -| No TP | 23.94 GB | 2 | 1.65 TFlops | -| 2 GPU TP | 12.23 GB | 20 | 4.61 TFlops | -| 4 GPU TP | 6.36 GB | 56 | 4.90 TFlops | +| Test | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU | +|----------|--------------------------|----------------|------------------------| +| No TP | 23.94 GB | 2 | 1.65 TFlops | +| 2 GPU TP | 12.23 GB | 20 | 4.61 TFlops | +| 4 GPU TP | 6.36 GB | 56 | 4.90 TFlops | # Supported Models From e26f7af78147cd885b2a5f259b64d1c4fa918d30 Mon Sep 17 00:00:00 2001 From: Molly Smith Date: Wed, 22 Feb 2023 22:31:43 +0000 Subject: [PATCH 2/4] Add tutorial links to navigation --- docs/_config.yml | 1 + docs/_data/navigation.yml | 2 ++ 2 files changed, 3 insertions(+) diff --git a/docs/_config.yml b/docs/_config.yml index f7350e81ffca..7127b8459fe2 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -35,6 +35,7 @@ collections: - advanced-install.md - getting-started.md - azure.md + - automatic-tensor-parallelism.md - bert-finetuning.md - bert-pretraining.md - cifar-10.md diff --git a/docs/_data/navigation.yml b/docs/_data/navigation.yml index 4449bb91ed12..6f7c443c7958 100755 --- a/docs/_data/navigation.yml +++ b/docs/_data/navigation.yml @@ -53,6 +53,8 @@ lnav: url: /getting-started/ - title: 'Getting started on Azure' url: /tutorials/azure/ + - title: 'Automatic Tensor Parallelism' + url: /tutorials/automatic-tensor-parallelism/ - title: 'Autotuning' url: /tutorials/autotuning/ - title: 'BingBertSQuAD Fine-tuning' From fac6a3f99c185550bef61396c47213260b4ade14 Mon Sep 17 00:00:00 2001 From: Molly Smith Date: Wed, 22 Feb 2023 22:43:19 +0000 Subject: [PATCH 3/4] Add news bit to main readme --- README.md | 1 + docs/_tutorials/automatic-tensor-parallelism.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 136f2d28734b..bfa03a6e8c9a 100755 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@ ## Latest News DeepSpeed trained the world's most powerful language models ([MT-530B](https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/), [BLOOM](https://huggingface.co/blog/bloom-megatron-deepspeed)); [learn how](https://www.deepspeed.ai/tutorials/large-models-w-deepspeed/). +* [2023/02] [Automatic Tensor Parallelism: Enables tensor parallelism by default without providing an injection policy](https://www.deepspeed.ai/tutorials/automatic-tensor-parallelism/) * [2022/12] [DeepSpeed Data Efficiency: A composable library that makes better use of data, increases training efficiency, and improves model quality](https://www.deepspeed.ai/2022/12/11/data-efficiency.html) * [2022/11] [Stable Diffusion Image Generation under 1 second w. DeepSpeed MII](https://github.com/microsoft/DeepSpeed-MII/tree/main/examples/benchmark/txt2img) * [2022/10] [DeepSpeed-MII: instant speedup on 24,000+ open-source DL models with up to 40x cheaper inference](https://www.deepspeed.ai/2022/10/10/mii.html) diff --git a/docs/_tutorials/automatic-tensor-parallelism.md b/docs/_tutorials/automatic-tensor-parallelism.md index 10a3155688ef..7de1c955bd9d 100644 --- a/docs/_tutorials/automatic-tensor-parallelism.md +++ b/docs/_tutorials/automatic-tensor-parallelism.md @@ -139,7 +139,7 @@ The following model families have been successfully tested with automatic tensor # Unsupported Models -The following models are not currently supported: +The following models are not currently supported with automatic tensor parallelism. They may still be compatible with other DeepSpeed features (i.e. kernel injection for Bloom): - bloom - codegen From 5d5d85bf4a775bb906affdf3837ce188d4baa206 Mon Sep 17 00:00:00 2001 From: Molly Smith <112220543+molly-smith@users.noreply.github.com> Date: Wed, 22 Feb 2023 16:35:01 -0800 Subject: [PATCH 4/4] Update docs/_tutorials/automatic-tensor-parallelism.md Co-authored-by: Michael Wyatt --- docs/_tutorials/automatic-tensor-parallelism.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/_tutorials/automatic-tensor-parallelism.md b/docs/_tutorials/automatic-tensor-parallelism.md index 7de1c955bd9d..e480fbf42ac8 100644 --- a/docs/_tutorials/automatic-tensor-parallelism.md +++ b/docs/_tutorials/automatic-tensor-parallelism.md @@ -139,7 +139,7 @@ The following model families have been successfully tested with automatic tensor # Unsupported Models -The following models are not currently supported with automatic tensor parallelism. They may still be compatible with other DeepSpeed features (i.e. kernel injection for Bloom): +The following models are not currently supported with automatic tensor parallelism. They may still be compatible with other DeepSpeed features (e.g., kernel injection for Bloom): - bloom - codegen