diff --git a/README.md b/README.md index 136f2d28734b..bfa03a6e8c9a 100755 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@ ## Latest News DeepSpeed trained the world's most powerful language models ([MT-530B](https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/), [BLOOM](https://huggingface.co/blog/bloom-megatron-deepspeed)); [learn how](https://www.deepspeed.ai/tutorials/large-models-w-deepspeed/). +* [2023/02] [Automatic Tensor Parallelism: Enables tensor parallelism by default without providing an injection policy](https://www.deepspeed.ai/tutorials/automatic-tensor-parallelism/) * [2022/12] [DeepSpeed Data Efficiency: A composable library that makes better use of data, increases training efficiency, and improves model quality](https://www.deepspeed.ai/2022/12/11/data-efficiency.html) * [2022/11] [Stable Diffusion Image Generation under 1 second w. DeepSpeed MII](https://github.com/microsoft/DeepSpeed-MII/tree/main/examples/benchmark/txt2img) * [2022/10] [DeepSpeed-MII: instant speedup on 24,000+ open-source DL models with up to 40x cheaper inference](https://www.deepspeed.ai/2022/10/10/mii.html) diff --git a/docs/_config.yml b/docs/_config.yml index f7350e81ffca..7127b8459fe2 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -35,6 +35,7 @@ collections: - advanced-install.md - getting-started.md - azure.md + - automatic-tensor-parallelism.md - bert-finetuning.md - bert-pretraining.md - cifar-10.md diff --git a/docs/_data/navigation.yml b/docs/_data/navigation.yml index 4449bb91ed12..6f7c443c7958 100755 --- a/docs/_data/navigation.yml +++ b/docs/_data/navigation.yml @@ -53,6 +53,8 @@ lnav: url: /getting-started/ - title: 'Getting started on Azure' url: /tutorials/azure/ + - title: 'Automatic Tensor Parallelism' + url: /tutorials/automatic-tensor-parallelism/ - title: 'Autotuning' url: /tutorials/autotuning/ - title: 'BingBertSQuAD Fine-tuning' diff --git a/docs/_tutorials/automatic-tensor-parallelism.md b/docs/_tutorials/automatic-tensor-parallelism.md index c78170131263..e480fbf42ac8 100644 --- a/docs/_tutorials/automatic-tensor-parallelism.md +++ b/docs/_tutorials/automatic-tensor-parallelism.md @@ -88,18 +88,18 @@ deepspeed --num_gpus DeepSpeedExamples/inference/huggingface/text-gen The following results were collected using V100 SXM2 32GB GPUs. ### Max New Tokens = 50 -| | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU | -|---|---|---|---| -| No TP | 23.94 GB | 64 | 18.84 TFlops | -| 2 GPU TP | 12.23 GB | 320 | 27.17 TFlops | -| 4 GPU TP | 6.36 GB | 664 | 27.63 TFlops | +| Test | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU | +|----------|--------------------------|----------------|------------------------| +| No TP | 23.94 GB | 64 | 18.84 TFlops | +| 2 GPU TP | 12.23 GB | 320 | 27.17 TFlops | +| 4 GPU TP | 6.36 GB | 664 | 27.63 TFlops | ### Max New Tokens = 1024 -| | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU | -|---|---|---|---| -| No TP | 23.94 GB | 2 | 1.65 TFlops | -| 2 GPU TP | 12.23 GB | 20 | 4.61 TFlops | -| 4 GPU TP | 6.36 GB | 56 | 4.90 TFlops | +| Test | Memory Allocated per GPU | Max Batch Size | Max Throughput per GPU | +|----------|--------------------------|----------------|------------------------| +| No TP | 23.94 GB | 2 | 1.65 TFlops | +| 2 GPU TP | 12.23 GB | 20 | 4.61 TFlops | +| 4 GPU TP | 6.36 GB | 56 | 4.90 TFlops | # Supported Models @@ -139,7 +139,7 @@ The following model families have been successfully tested with automatic tensor # Unsupported Models -The following models are not currently supported: +The following models are not currently supported with automatic tensor parallelism. They may still be compatible with other DeepSpeed features (e.g., kernel injection for Bloom): - bloom - codegen