Automatic Tensor Parallelism Blog Links

README.md

@@ -12,6 +12,7 @@
 ## Latest News
 <b> DeepSpeed trained the world's most powerful language models ([MT-530B](https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/), [BLOOM](https://huggingface.co/blog/bloom-megatron-deepspeed)); [learn how](https://www.deepspeed.ai/tutorials/large-models-w-deepspeed/).</b>
 
+* [2023/02] [Automatic Tensor Parallelism: Enables tensor parallelism by default without providing an injection policy](https://www.deepspeed.ai/tutorials/automatic-tensor-parallelism/)
 * [2022/12] [DeepSpeed Data Efficiency: A composable library that makes better use of data, increases training efficiency, and improves model quality](https://www.deepspeed.ai/2022/12/11/data-efficiency.html)
 * [2022/11] [Stable Diffusion Image Generation under 1 second w. DeepSpeed MII](https://github.com/microsoft/DeepSpeed-MII/tree/main/examples/benchmark/txt2img)
 * [2022/10] [DeepSpeed-MII: instant speedup on 24,000+ open-source DL models with up to 40x cheaper inference](https://www.deepspeed.ai/2022/10/10/mii.html)

docs/_config.yml

@@ -35,6 +35,7 @@ collections:
       - advanced-install.md
       - getting-started.md
       - azure.md
+      - automatic-tensor-parallelism.md
       - bert-finetuning.md
       - bert-pretraining.md
       - cifar-10.md

docs/_data/navigation.yml

@@ -53,6 +53,8 @@ lnav:
         url: /getting-started/
       - title: 'Getting started on Azure'
         url: /tutorials/azure/
+      - title: 'Automatic Tensor Parallelism'
+        url: /tutorials/automatic-tensor-parallelism/
       - title: 'Autotuning'
         url: /tutorials/autotuning/
       - title: 'BingBertSQuAD Fine-tuning'

docs/_tutorials/automatic-tensor-parallelism.md

@@ -88,18 +88,18 @@ deepspeed --num_gpus <num_gpus> DeepSpeedExamples/inference/huggingface/text-gen
 The following results were collected using V100 SXM2 32GB GPUs.
 
 ### Max New Tokens = 50
-| | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU |
-|---|---|---|---|
-| No TP    | 23.94 GB | 64  | 18.84 TFlops |
-| 2 GPU TP | 12.23 GB | 320 | 27.17 TFlops |
-| 4 GPU TP | 6.36 GB  | 664 | 27.63 TFlops |
+| Test     | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU |
+|----------|--------------------------|----------------|------------------------|
+| No TP    | 23.94 GB                 | 64             | 18.84 TFlops           |
+| 2 GPU TP | 12.23 GB                 | 320            | 27.17 TFlops           |
+| 4 GPU TP | 6.36 GB                  | 664            | 27.63 TFlops           |
 
 ### Max New Tokens = 1024
-| | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU |
-|---|---|---|---|
-| No TP    | 23.94 GB | 2  | 1.65 TFlops |
-| 2 GPU TP | 12.23 GB | 20 | 4.61 TFlops |
-| 4 GPU TP | 6.36 GB  | 56 | 4.90 TFlops |
+| Test     | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU |
+|----------|--------------------------|----------------|------------------------|
+| No TP    | 23.94 GB                 | 2              | 1.65 TFlops            |
+| 2 GPU TP | 12.23 GB                 | 20             | 4.61 TFlops            |
+| 4 GPU TP | 6.36 GB                  | 56             | 4.90 TFlops            |
 
 # Supported Models
 
@@ -139,7 +139,7 @@ The following model families have been successfully tested with automatic tensor
 
 # Unsupported Models
 
-The following models are not currently supported:
+The following models are not currently supported with automatic tensor parallelism. They may still be compatible with other DeepSpeed features (e.g., kernel injection for Bloom):
 
 - bloom
 - codegen