{"id":689370,"date":"2020-09-10T13:02:08","date_gmt":"2020-09-10T20:02:08","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=689370"},"modified":"2022-11-01T08:58:39","modified_gmt":"2022-11-01T15:58:39","slug":"deepspeed-extreme-scale-model-training-for-everyone","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/deepspeed-extreme-scale-model-training-for-everyone\/","title":{"rendered":"DeepSpeed: Extreme-scale model training for everyone"},"content":{"rendered":"<aside id=accordion-fee4f2a6-1c08-45d1-902c-9e241577b419 class=\"msr-table-of-contents-block accordion mb-5 pb-0\" data-bi-aN=\"table-of-contents\">\n\t<button class=\"btn btn-collapse bg-gray-100 mb-0 display-flex justify-content-between\" type=\"button\" data-mount=\"collapse\" data-target=\"#accordion-collapse-fee4f2a6-1c08-45d1-902c-9e241577b419\" aria-expanded=\"true\" aria-controls=\"accordion-collapse-fee4f2a6-1c08-45d1-902c-9e241577b419\">\n\t\t<span class=\"msr-table-of-contents-block__label subtitle\">In this article<\/span>\n\t\t<span class=\"msr-table-of-contents-block__current mr-4 text-gray-600 font-weight-normal\" aria-hidden=\"true\"><\/span>\n\t<\/button>\n\t<div id=\"accordion-collapse-fee4f2a6-1c08-45d1-902c-9e241577b419\" class=\"msr-table-of-contents-block__collapse-wrapper collapse show\" data-parent=\"#accordion-fee4f2a6-1c08-45d1-902c-9e241577b419\">\n\t\t<div class=\"accordion-body bg-gray-100 border-top pt-4\">\n\t\t\t<ol class=\"msr-table-of-contents-block__list\">\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"\" class=\"msr-table-of-contents-block__list-item-link\">3D parallelism: Scaling to trillion-parameter models<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"\" class=\"msr-table-of-contents-block__list-item-link\">ZeRO-Offload: 10x bigger model training using a single GPU<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"\" class=\"msr-table-of-contents-block__list-item-link\">DeepSpeed Sparse Attention: Powering 10x longer sequences with 6x faster execution<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"\" class=\"msr-table-of-contents-block__list-item-link\">1-bit Adam: 5x less communication and 3.4x faster training<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<\/ul>\n\t\t<\/div>\n\t<\/div>\n\t<span class=\"msr-table-of-contents-block__progress-bar\"><\/span>\n<\/aside>\n\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p>In February, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters\/\">we announced DeepSpeed<\/a>, an open-source deep learning training optimization library, and ZeRO (Zero Redundancy Optimizer), a novel memory optimization technology in the library, which vastly advances large model training by improving scale, speed, cost, and usability. DeepSpeed has enabled researchers to create Turing Natural Language Generation (<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/turing-nlg-a-17-billion-parameter-language-model-by-microsoft\">Turing-NLG<\/a>), the largest language model with 17 billion parameters and state-of-the-art accuracy at the time of its release. In May, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/zero-2-deepspeed-shattering-barriers-of-deep-learning-speed-scale\/\">we released ZeRO-2<\/a>\u2014supporting model training of 200 billion parameters up to 10x faster compared to state of the art\u2014along with a list of compute, I\/O, and convergence optimizations powering the fastest BERT training. From there, we have been continuing to innovate at a fast rate, pushing the boundaries of speed and scale for deep learning training.<\/p>\n\n\n\n<p>Today, we are happy to share our new advancements that not only push deep learning training to the extreme, but also democratize it for more people\u2014from data scientists training on massive supercomputers to those training on low-end clusters or even on a single GPU. More specifically, DeepSpeed adds four new system technologies that further the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/ai-at-scale\/\">AI at Scale<\/a> initiative to innovate across Microsoft\u2019s AI products and platforms. These offer extreme compute, memory, and communication efficiency, and they power model training with billions to trillions of parameters. The technologies also allow for extremely long input sequences and power on hardware systems with a single GPU, high-end clusters with thousands of GPUs, or low-end clusters with very slow ethernet networks.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Trillion parameter model training with 3D parallelism:<\/strong> DeepSpeed enables a flexible combination of three parallelism approaches\u2014ZeRO-powered data parallelism, pipeline parallelism, and tensor-slicing model parallelism. 3D parallelism adapts to the varying needs of workload requirements to power <strong>extremely large models<\/strong> with over a <strong>trillion <\/strong>parameters while achieving near-perfect memory-scaling and throughput-scaling efficiency. In addition, its improved communication efficiency allows users to train multi-billion-parameter models 2\u20137x faster on regular clusters with limited network bandwidth.<\/li><li><strong>10x bigger model training on a single GPU with ZeRO-Offload:<\/strong> We extend ZeRO-2 to leverage both CPU and GPU memory for training large models. Using a machine with <strong>a single NVIDIA V100 GPU<\/strong>, our users can run <strong>models of up to 13 billion parameters<\/strong> without running out of memory, 10x bigger than the existing approaches, while obtaining competitive throughput. This feature democratizes multi-billion-parameter model training and opens the window for many deep learning practitioners to explore bigger and better models. Read the paper: <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/zero-offload-democratizing-billion-scale-model-training\/\">https:\/\/www.microsoft.com\/en-us\/research\/publication\/zero-offload-democratizing-billion-scale-model-training\/ <\/a><\/li><li><strong>Powering 10x longer sequences and 6x faster execution through DeepSpeed Sparse Attention:<\/strong> DeepSpeed offers sparse attention kernels\u2014an instrumental technology to support long sequences of model inputs, whether for text, image, or sound. Compared with the classic dense Transformers, it powers <strong>an order-of-magnitude longer input sequence<\/strong> and obtains up to 6x faster execution with comparable accuracy. It also outperforms state-of-the-art sparse implementations with 1.5\u20133x faster execution. Furthermore, our sparse kernels support efficient execution of flexible sparse format and empower users to innovate on their custom sparse structures.<\/li><li><strong>1-bit Adam with up to 5x communication volume reduction: <\/strong>Adam is an effective and (probably the most well-utilized) optimizer for training many large-scale deep learning models. However, Adam is generally not compatible with communication-efficient optimization algorithms. Therefore, the communication cost could become a bottleneck while scaling across distributed devices. We introduce a new algorithm, 1-bit Adam with efficient implementation, which <strong>reduces communication volume by up to 5x<\/strong> while achieving similar convergence efficiency to Adam. We observe up to 3.5x faster distributed training in communication-constrained scenarios, allowing for scaling to different types of GPU clusters and networks. Read the paper: <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/1-bit-adam-communication-efficient-large-scale-training-with-adams-convergence-speed\/ \">https:\/\/www.microsoft.com\/en-us\/research\/publication\/1-bit-adam-communication-efficient-large-scale-training-with-adams-convergence-speed\/ <\/a><\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_MainHero_HighRes-1024x636.jpg\" target=\"_blank\" rel=\"noopener noreferrer\"><img loading=\"lazy\" decoding=\"async\" width=\"1237\" height=\"768\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_MainHero_HighRes.jpg\" alt=\"3d Parallelism, ZeRO-Offload, Sparse Attention, 1-bit Adam\" class=\"wp-image-690747\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_MainHero_HighRes.jpg 1237w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_MainHero_HighRes-300x186.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_MainHero_HighRes-1024x636.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_MainHero_HighRes-768x477.jpg 768w\" sizes=\"(max-width: 1237px) 100vw, 1237px\" \/><\/a><\/figure>\n\n\n\n<p>This blog post explores these four lines of technology in greater depth. We have been making all of these exciting new optimizations available in <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/DeepSpeed\">open-source library, DeepSpeed<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n\t<div class=\"border-bottom border-top border-gray-300 mt-5 mb-5 msr-promo text-center text-md-left alignwide\" data-bi-aN=\"promo\" data-bi-id=\"1045002\">\n\t\t\n\n\t\t<p class=\"msr-promo__label text-gray-800 text-center text-uppercase\">\n\t\t<span class=\"px-4 bg-white display-inline-block font-weight-semibold small\">Spotlight: Event<\/span>\n\t<\/p>\n\t\n\t<div class=\"row pt-3 pb-4 align-items-center\">\n\t\t\t\t\t\t<div class=\"msr-promo__media col-12 col-md-5\">\n\t\t\t\t<a class=\"bg-gray-300\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/cvpr-2024\/\" aria-label=\"Microsoft at CVPR 2024\" data-bi-cN=\"Microsoft at CVPR 2024\" target=\"_blank\">\n\t\t\t\t\t<img decoding=\"async\" class=\"w-100 display-block\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/05\/CVPR_WebBanner_1920x720.png\" alt=\"teal background triangular pattern\" \/>\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t<div class=\"msr-promo__content p-3 px-5 col-12 col-md\">\n\n\t\t\t\t\t\t\t\t\t<h2 class=\"h4\">Microsoft at CVPR 2024<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<p class=\"large\">Microsoft is a proud sponsor and active participant of\u00a0<a href=\"https:\/\/cvpr2023.thecvf.com\/Conferences\/2024\" target=\"_blank\" rel=\"noreferrer noopener\">CVPR 2024<\/a>, which focuses on advancements in computer vision and pattern recognition. <\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<div class=\"wp-block-buttons justify-content-center justify-content-md-start\">\n\t\t\t\t\t<div class=\"wp-block-button\">\n\t\t\t\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/cvpr-2024\/\" class=\"btn btn-brand glyph-append glyph-append-chevron-right\" aria-label=\"Learn more\" data-bi-cN=\"Microsoft at CVPR 2024\" target=\"_blank\">\n\t\t\t\t\t\t\tLearn more\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div><!--\/.msr-promo__content-->\n\t<\/div><!--\/.msr-promo__inner-wrap-->\n\t<\/div><!--\/.msr-promo-->\n\t\n\n\n<h2 id=\"3d-parallelism-scaling-to-trillion-parameter-models\">3D parallelism: Scaling to trillion-parameter models<\/h2>\n\n\n\n<p>With the rapid growth of compute available on modern GPU clusters, training a powerful trillion-parameter model with incredible capabilities is no longer a far-fetched dream but rather a near-future reality. DeepSpeed has combined three powerful technologies to enable training trillion-scale models and to scale to thousands of GPUs: data parallel training, model parallel training, and pipeline parallel training. This symbiosis scales deep learning training far beyond what each of the strategies can offer in isolation. 3D parallelism simultaneously addresses the two fundamental challenges toward training trillion-parameter models:<em> memory efficiency<\/em> and <em>compute efficiency<\/em>. As a result, DeepSpeed can scale to fit the most massive models in memory without sacrificing speed.<\/p>\n\n\n\n\n\n<p><strong>Memory Efficiency: <\/strong>The memory requirements to train a trillion-parameter model are far beyond what is available in a single GPU device. Training using the Adam optimizer in mixed precision requires approximately 16 terabytes (TB) of memory just to store the model states (parameters, gradients, and optimizer states). For comparison, the state-of-the-art NVIDIA A100 GPUs have just 40 gigabytes (GB) of memory. It would require the collective memory of 400 such GPUs just to store the model states.<\/p>\n\n\n\n<p>Activations consume additional memory that increases with the batch size. A trillion-parameter model trained with only unit batch size produces over 1 TB of activation memory. Activation <em>checkpointing <\/em>reduces this memory to approximately 20 GB by trading for additional compute, but the memory requirements remain prohibitively large for training.<\/p>\n\n\n\n<p>The model states and activations must be efficiently partitioned across the available multiple GPU devices to enable such a model to even begin training without running out of memory.<\/p>\n\n\n\n<p><strong>Compute Efficiency: <\/strong>Training a trillion-parameter model end-to-end requires approximately 5,000 zettaflops (that\u2019s 5 with <em>24 zeros<\/em> after it; based on the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2001.08361\">laws of scaling<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> work from OpenAI). It would take 4,000 NVIDIA A100 GPUS running at 50% compute efficiency about 100 days to train such a model.<\/p>\n\n\n\n<p>While large super-computing GPU clusters can have well over 4,000 GPUs, achieving high compute efficiency at this scale is challenging due to the batch size constraints. Compute efficiency increases as the computation time increases over the communication time. This ratio is proportional to the batch size. However, the batch size that a model can be trained with has an upper bound\u2014beyond that the convergence efficiency deteriorates rapidly.<\/p>\n\n\n\n<p>One of the largest models in the world, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2005.14165\">GPT-3<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, was trained using a batch size of about 1,500. With 4,000 GPUs, even a liberal batch size of 4,000 would only allow for a batch size of 1 per GPU and limit scalability.<\/p>\n\n\n\n\n\n<p><strong>Data parallelism<\/strong> is a ubiquitous technique in deep learning in which each input batch of training data is split among the data parallel workers. Gradients must be communicated and aggregated after backward propagation to ensure that consistent steps are taken by the optimizer. Data parallelism has several distinct advantages, including compute efficiency and minimal implementation effort. However, data parallelism relies on scaling the batch size with the number of data parallel workers, which cannot be done indefinitely without affecting convergence.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Memory efficiency: <\/strong>Data parallelism replicates the model and optimizer across all workers, and therefore is not memory efficient. DeepSpeed developed <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters\/\">ZeRO<\/a>, a collection of optimizations that improve the memory efficiency of data parallelism. This work relies on ZeRO stage 1, which partitions the optimizer states among data parallel workers to reduce redundancy.<br><\/li><li><strong>Compute efficiency: <\/strong>The amount of computation performed by each worker is constant as we increase the degree of parallelism. Data parallelism can achieve near-perfect scaling at small scales. However, the communication cost of aggregating gradients among data parallel workers scales with the model size and limits compute efficiency on large models or systems with low communication bandwidth. <em>Gradient accumulation<\/em> is a common strategy for amortizing this communication cost by further increasing the batch size and performing multiple forward and backward propagations on <em>micro-batches <\/em>while locally accumulating gradients before aggregating and taking an optimizer step.<\/li><\/ul>\n\n\n\n<p><strong>Model Parallelism<\/strong> is a broad class of techniques that partitions the individual layers of the model across workers. By its nature, the computations and communications of model parallelism are specific to a model architecture and therefore can have large initial implementation effort. DeepSpeed leverages NVIDIA\u2019s <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/NVIDIA\/Megatron-LM\">Megatron-LM<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> in this work for massive model-parallel Transformer-based language models. Model parallelism reduces the memory proportional to the number of workers. Model parallelism is the most memory efficient among the three types of parallelism at the cost of the lowest compute efficiency.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Memory efficiency: <\/strong>Model parallelism reduces the memory footprint proportional to the number of workers. Crucially, it is the only approach that reduces the activation memory for individual network layers. DeepSpeed further improves memory efficiency by partitioning the activation memory among model-parallel workers.<br><\/li><li><strong>Compute efficiency: <\/strong>Model parallelism has poor computational efficiency due to additional communication of activations in each forward and backward propagation. Model parallelism requires high communication bandwidth to be efficient and does not scale well beyond a single node where the communication bandwidth is limited. Furthermore, each model-parallel worker decreases the amount of computation performed between each communication stage, impacting compute efficiency. Model parallelism is often used in conjunction with data parallelism to trade between memory and compute efficiency.<\/li><\/ul>\n\n\n\n<p><strong>Pipeline parallelism<\/strong> training engine is included in this release of DeepSpeed! Pipeline parallelism divides the layers of the model into <em>stages<\/em> that can be processed in parallel. As one stage completes the forward pass for a micro-batch, the activation memory is communicated to the next stage in the pipeline. Similarly, as the next stage completes its backward propagation, gradients are communicated backwards through the pipeline. Multiple micro-batches must be kept in flight to ensure pipeline stages compute in parallel. Several approaches, such as <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/pipedream-a-more-effective-way-to-train-deep-neural-networks-using-pipeline-parallelism\/\">PipeDream<\/a>, have been developed to trade off memory and compute efficiency as well as convergence behavior. The DeepSpeed approach extracts parallelism through gradient accumulation to maintain the same convergence behavior as traditional data- and model-parallel training with the same total batch size.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Memory efficiency: <\/strong>Pipeline parallelism reduces memory proportional to the number of pipeline stages, allowing model size to scale linearly with the number of workers. However, pipeline parallelism does not reduce the memory footprint for the activations of each layer. Additionally, each worker must store the activations for all micro-batches in flight. In effect, the activation memory on the first stage of the pipeline is approximately the same as the total activation memory for a single micro-batch. A trillion-parameter model would need approximately 19 GB of memory for the activations of a micro-batch, consuming almost half the available memory of the new NVIDIA A100 GPU.<br><\/li><li><strong>Compute efficiency: <\/strong>Pipeline parallelism has the lowest communication volume since it only communicates data proportional to the activation size of the layers between stage boundaries. However, it cannot scale indefinitely. Like model parallelism, increasing the pipeline size decreases the computation per pipeline stage, which also decreases the compute-to-communication ratio. Pipeline parallelism also requires each of its stages to be perfectly load balanced to achieve good efficiency.<br><br>Furthermore, pipeline parallelism incurs a bubble overhead from filling and emptying the pipeline at the beginning and end of each training batch. Training with gradient accumulation steps (and thus batch size) that is 4x or 8x the number of pipeline stages achieves 81% and 90% scaling efficiency from one pipeline stage, respectively.<\/li><\/ul>\n\n\n\n\n\n<h3 id=\"achieving-both-memory-and-compute-efficiency-with-3d-parallelism\">Achieving both memory and compute efficiency with 3D parallelism<\/h3>\n\n\n\n<p>Data, model, and pipeline parallelism each perform a specific role in improving memory and compute efficiency. Figure 1 illustrates our 3D strategy.<\/p>\n\n\n\n<p><em>Memory Efficiency:<\/em> The layers of the model are divided into pipeline stages, and the layers of each stage are further divided via model parallelism. This 2D combination simultaneously reduces the memory consumed by the model, optimizer, and activations. However, we cannot partition the model indefinitely without succumbing to communication overheads which limits compute efficiency.<\/p>\n\n\n\n<p><em>Compute Efficiency:<\/em> To allow the number of workers to scale beyond model and pipeline parallelism without sacrificing compute efficiency, we use ZeRO-powered data parallelism (ZeRO-DP). ZeRO-DP not only improves memory efficiency further via optimizer state partition, but also allows scaling to arbitrarily large number of GPUs with minimal communication overhead by exploiting topology aware mapping.<\/p>\n\n\n\n<p><em>Topology aware 3D mapping <\/em>(Figure 2)<em>: <\/em>Each dimension in 3D parallelism is carefully mapped onto the workers to achieve maximum compute efficiency by exploiting two key architectural properties.<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li><strong>Optimizing for intra- and inter-node communication bandwidth<\/strong>: Model parallelism has the largest communication overhead of the three strategies, and so we prioritize placing model parallel groups within a node to utilize the larger intra-node bandwidth. Here we apply NVIDIA Megatron-LM for tensor-slicing style of model parallelism. Data parallel groups are\u202fplaced within a node when model parallelism does not span\u202fall the workers in a\u202fnode.\u202f Otherwise, they are placed across nodes.\u202f Pipeline parallelism has the lowest communication volume, and so we can schedule pipeline stages across nodes without being limited by the communication bandwidth.<\/li><li><strong>Bandwidth amplification via parallelism in communication:<\/strong> The size of the gradients communicated by each data parallel group decreases linearly via both pipeline and model parallelism, and thus the total communication volume is decreased from pure data parallelism.\u202f Furthermore, each data parallel group performs its communication independently and in parallel among a subset of localized workers. As a result, the effective bandwidth for\u202fdata parallel\u202fcommunication\u202fis amplified by a combination of reduced communication volume and increased locality and parallelism.<\/li><\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"615\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure-1_highres-1024x615.png\" alt=\"Diagram showing Example 3D parallelism with 32 workers. Layers of the neural network are divided among four pipeline stages. Layers within each pipeline stage are further partitioned among four model parallel workers. Lastly, each pipeline is replicated across two data parallel instances, and ZeRO partitions the optimizer states across the data parallel replicas.\" class=\"wp-image-690744\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure-1_highres-1024x615.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure-1_highres-300x180.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure-1_highres-768x461.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure-1_highres-1536x923.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure-1_highres-2048x1230.png 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 1: Example 3D parallelism with 32 workers. Layers of the neural network are divided among four pipeline stages. Layers within each pipeline stage are further partitioned among four model parallel workers. Lastly, each pipeline is replicated across two data parallel instances, and ZeRO partitions the optimizer states across the data parallel replicas.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"606\" height=\"350\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure2_highres.png\" alt=\"Colorful blocks showing  Mapping of workers in Figure 1 to GPUs on a system with eight nodes, each with four GPUs. Coloring denotes GPUs on the same node.\" class=\"wp-image-690741\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure2_highres.png 606w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure2_highres-300x173.png 300w\" sizes=\"(max-width: 606px) 100vw, 606px\" \/><figcaption>Figure 2: Mapping of workers in Figure 1 to GPUs on a system with eight nodes, each with four GPUs. Coloring denotes GPUs on the same node.<\/figcaption><\/figure>\n\n\n\n\n\n<p>A trillion-parameter model could be scaled across 4,096 NVIDIA A100 GPUs using 8-way model parallelism, 64-way pipeline parallelism, and 8-way data parallelism.<\/p>\n\n\n\n<p>By combining model parallelism and pipeline parallelism, 3D parallelism achieves excellent memory efficiency and compute efficiency across multiple nodes. Model parallelism brings memory efficiency for the activations and model states within a node, while pipeline parallelism allows for memory efficiency of model states across nodes without sacrificing compute efficiency compared to using model parallelism alone. In our trillion-parameter example with a micro-batch size of 1, our model would consume 30 GB of memory for model states and 2.5 GB for partitioned activations after activation checkpointing with the aforementioned 3D parallelism. This results in a total memory footprint of 32.5 GB. With such a configuration, NVIDIA A100 GPUs with 40 GB of memory have more than enough space to fit and train such a model.<\/p>\n\n\n\n<p>Combining model parallelism with pipeline parallelism also allows pipeline parallelism to achieve high compute efficiency with minimal bubble overhead even at very small batch sizes. With 8-way model parallelism, using a micro-batch of 1 per model would result in an effective micro-batch of 1\/8 per GPU. Therefore, pipeline parallelism can achieve a 90% compute efficiency using a gradient accumulation step of 8x the pipeline parallelism degree and with an aggregate per-GPU batch size of only 1. When combined with data parallelism, this results in an effective batch size of 4,096 on 4,096 GPUs, which can still achieve 90% pipeline efficiency.<\/p>\n\n\n\n<p><strong>But what compute efficiency results from data parallelism? Doesn\u2019t data parallelism require large batch per GPU to remain efficient?<\/strong><\/p>\n\n\n\n<p>Model parallelism can reduce the effective batch size to be less than 1 per GPU. This allows pipeline parallelism to hide the pipeline bubble overhead even with small batch sizes. Note that by using pipeline parallelism across nodes, we are effectively allowing communication between data parallel nodes at each stage of the pipeline to happen independently and in parallel with the other pipeline stages. In fact, in a fully connected network topology common in high-end GPU clusters, this has a significant implication on the effective communication bandwidth available for data parallel training. Since each node at a pipeline stage can communicate in parallel with its corresponding data parallel nodes, the effective communication bandwidth is directly proportional to the number of pipeline stages. With 64 pipeline-parallel stages, the effective bandwidth is 64x the bandwidth to and from a single node. With such large effective bandwidth pipeline parallelism enables data parallelism to scale effectively, even at small batch sizes where the compute-to-communication ratio is very low.<\/p>\n\n\n\n\n\n<h3 id=\"powering-trillion-parameter-model-training-with-linear-efficiency-scaling\">Powering trillion-parameter model training with linear efficiency scaling<\/h3>\n\n\n\n<p>DeepSpeed can train a language model with one <strong><em>trillion <\/em><\/strong>parameters using as few as 800 NVIDIA V100 GPUs (Figure 3). We demonstrate simultaneous memory and compute efficiency by scaling the size of the model and observing linear growth, both in terms of the size of the model and the throughput of the training. In every configuration, we can train approximately 1.4 billion parameters per GPU, which is the largest model size that a single GPU can support without running out of memory, indicating perfect memory scaling. We also obtain close to perfect-linear compute efficiency scaling and a throughput of 47 teraflops per V100 GPU. This is impressive scaling and throughput for the given hardware.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1226\" height=\"608\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-3_Section-1.jpg\" alt=\"Figure 3: Model size (in billions of parameters) and training throughput (in petaflops) as a function of GPUs. DeepSpeed can train a model with 1 trillion parameters using 800 NVIDIA V100 Tensor Core GPUs with 32 GB of memory. Each configuration uses 16-way model parallelism provided by NVIDIA Megatron-LM, and the remaining GPUs are arranged using pipeline parallelism. The trillion-parameter model has 298 layers of Transformers with a hidden dimension of 17,408 and is trained with sequence length 2,048 and batch size 2,048. For smaller models, we decrease the number of Transformer layers and the batch size proportionally to the number of GPUs.\" class=\"wp-image-689928\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-3_Section-1.jpg 1226w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-3_Section-1-300x149.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-3_Section-1-1024x508.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-3_Section-1-768x381.jpg 768w\" sizes=\"(max-width: 1226px) 100vw, 1226px\" \/><figcaption>Figure 3: Model size (in billions of parameters) and training throughput (in petaflops) as a function of GPUs. DeepSpeed can train a model with 1 trillion parameters using 800 NVIDIA V100 Tensor Core GPUs with 32 GB of memory. Each configuration uses 16-way model parallelism provided by <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/NVIDIA\/Megatron-LM\">NVIDIA Megatron-LM<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, and the remaining GPUs are arranged using pipeline parallelism. The trillion-parameter model has 298 layers of Transformers with a hidden dimension of 17,408 and is trained with sequence length 2,048 and batch size 2,048. For smaller models, we decrease the number of Transformer layers and the batch size proportionally to the number of GPUs.<\/figcaption><\/figure>\n\n\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters\/\"><img loading=\"lazy\" decoding=\"async\" width=\"963\" height=\"538\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2-_section-2.jpg\" alt=\"Figure 4: System performance using 800 GPUs to train a GPT-3 scale model with 180 billion parameters using 2D and 3D parallelism. The model has 100 Transformer layers with hidden dimension 12,288 and 96 attention heads. The model is trained with batch size 2,048 and sequence length 2,048. ZeRO-1 is enabled alongside data parallelism. P, M, and D denote the pipeline, model, and data parallel dimensions, respectively.\" class=\"wp-image-689958\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2-_section-2.jpg 963w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2-_section-2-300x168.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2-_section-2-768x429.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2-_section-2-343x193.jpg 343w\" sizes=\"(max-width: 963px) 100vw, 963px\" \/><\/a><figcaption>Figure 4: System performance using 800 GPUs to train a GPT-3 scale model with 180 billion parameters using 2D and 3D parallelism. The model has 100 Transformer layers with hidden dimension 12,288 and 96 attention heads. The model is trained with batch size 2,048 and sequence length 2,048. ZeRO-1 is enabled alongside data parallelism. P, M, and D denote the pipeline, model, and data parallel dimensions, respectively.<\/figcaption><\/figure>\n\n\n\n<p>In Figure 4, we use the recent <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2005.14165\">GPT-3<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> model architecture, with over 175 billion parameters, as a benchmark for 3D parallelism: \u00ad<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>We first evaluate the <strong>2D configurations<\/strong> (C1-C3). Configurations C1 and C2 use only pipeline and model parallelism\u2014they can train the model but achieve low throughput due to over-decomposing the problem and having low GPU utilization. C3 attempts to use only pipeline and data parallelism but is unable to fit the problem in memory without reducing the size of activations via Megatron\u2019s model parallelism.<\/li><li>The <strong>3D configurations <\/strong>(C4-C10) are arranged by increasing degree of pipeline parallelism; the best performance is achieved by the middle configurations that balance the parallelism in order to be memory-, computation-, and communication-efficient.<\/li><li>The best 3D approaches achieve 49 teraflops per GPU, over 40% of the theoretical hardware peak.<\/li><\/ul>\n\n\n\n\n\n<p>We demonstrate the communication benefits of hybrid parallelism in Figure 5 while training a 1.5-billion-parameter GPT-2 model. We train on four nodes of a cluster with low inter-node bandwidth in order to emphasize the communication stages of training:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Model parallelism<\/strong> is not advantageous in this case due to the low intra-node bandwidth and smaller model size.<br><\/li><li><strong>Pipeline parallelism<\/strong> communicates over an order of magnitude less volume than the data and model parallel configurations and is 7x faster at small batch sizes.<br><\/li><li><strong>Data parallelism<\/strong> uses gradient accumulation to amortize communication overhead as the batch size increases, but pipeline parallel configurations still achieve over twice the performance of data parallelism at larger batch sizes.<br><\/li><li>The <strong>hybrid pipeline and data parallel configuration<\/strong> avoids the gradient communication bottleneck by restricting data parallel groups to GPUs within a node, so gradient communications benefit from the faster intra-node bandwidth.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"934\" height=\"676\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-4_Section-1.jpg\" alt=\"Figure 5: Throughput as a function of batch size while training GPT-2 (1.5B parameters) with sequence length 1,024. Training uses four nodes, each with four NVIDIA V100 GPUs with 16 GB of memory. The GPUs are connected with 50 Gigabits-per-second (Gbps) intra-node bandwidth and 4 Gbps inter-node bandwidth. DP denotes data parallelism with ZeRO-1 enabled. All methods scale batch size via increasing steps of gradient accumulation.\" class=\"wp-image-689964\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-4_Section-1.jpg 934w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-4_Section-1-300x217.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-4_Section-1-768x556.jpg 768w\" sizes=\"(max-width: 934px) 100vw, 934px\" \/><figcaption>Figure 5: Throughput as a function of batch size while training GPT-2 (1.5B parameters) with sequence length 1,024. Training uses four nodes, each with four NVIDIA V100 GPUs with 16 GB of memory. The GPUs are connected with 50 Gigabits-per-second (Gbps) intra-node bandwidth and 4 Gbps inter-node bandwidth. DP denotes data parallelism with ZeRO-1 enabled. All methods scale batch size via increasing steps of gradient accumulation.<\/figcaption><\/figure>\n\n\n\n\n\n<h2 id=\"zero-offload-10x-bigger-model-training-using-a-single-gpu\">ZeRO-Offload: 10x bigger model training using a single GPU<\/h2>\n\n\n\n<p>ZeRO-Offload pushes the boundary of the maximum model size that can be trained efficiently using minimal GPU resources, by exploiting computational and memory resources on both GPUs and their host CPUs. It allows training up to 13-billion-parameter models on a single NVIDIA V100 GPU, 10x larger than the state-of-the-art while retaining high training throughput of over 30 teraflops per GPU.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--right\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication <\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/zero-offload-democratizing-billion-scale-model-training\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"ZeRO-Offload: Democratizing Billion-Scale Model Training\" data-bi-aN=\"margin-callout\" data-bi-cN=\"ZeRO-Offload: Democratizing Billion-Scale Model Training\">\n\t\t\t\tZeRO-Offload: Democratizing Billion-Scale Model Training&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>By enabling multi-billion-parameter model training on a single GPU, ZeRO-Offload democratizes large model training, making it accessible to deep learning practitioners with limited resources.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_deepspeed3_figure6_highres-1024x552.jpg\" alt=\"Bar graph showing largest models can be trained using default PyTorch and ZeRO-Offload on a single GPU.\" class=\"wp-image-690726\" width=\"719\" height=\"395\"\/><figcaption>Figure 6: The largest models can be trained using default PyTorch and ZeRO-Offload on a single GPU.<\/figcaption><\/figure>\n\n\n\n<p>The key technology behind ZeRO-Offload is our new capability to offload optimizer states and gradients onto CPU memory, building on top of <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/zero-2-deepspeed-shattering-barriers-of-deep-learning-speed-scale\/\">ZeRO-2<\/a>. This approach allows ZeRO-Offload to minimize the compute efficiency loss from CPU offloading while also achieving the same, and sometimes even better, efficiency of the original ZeRO-2. The figure below shows the architecture of ZeRO-Offload.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"546\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2_section-1-1024x546.png\" alt=\"Figure 7: ZeRO-Offload overview. \" class=\"wp-image-689934\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2_section-1-1024x546.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2_section-1-300x160.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2_section-1-768x409.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2_section-1-710x380.png 710w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2_section-1.png 1212w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 7: ZeRO-Offload overview. <\/figcaption><\/figure>\n\n\n\n\n\n<p>Training multi-billion-parameter models like GPT and T5 require many GPUs to fit the model and its states in GPU memory. Large model training has been mostly carried out with model parallelism across multiple GPU devices to solve the memory limitation problem. Recently, we released ZeRO, a memory efficient optimizer that partitions model states (optimizer states, gradients, and model weights) across data parallel GPUs, allowing multi-billion-parameter models to be trained without requiring model parallelism. However, ZeRO still requires a large number of data parallel GPUs to hold the partitioned model states, limiting the access of large model training to a few with access to such resources.<\/p>\n\n\n\n<p>ZeRO-Offload democratizes large model training by making it possible even on a single GPU. To allow training multi-billion-parameter models without using multiple GPUs, ZeRO-Offload inherits the optimizer state and gradient partitioning from ZeRO-2. Unlike ZeRO-2, instead of having each GPU keep a partition of the optimizer state and gradients, ZeRO-Offload offloads both to host CPU memory. Optimizer states are kept in CPU memory for the entire training. Gradients, on the other hand, are computed and averaged using reduce-scatter on the GPUs during the backward pass, and each data-parallel process then offloads the averaged gradients belonging to its partition to the CPU memory (<em>g offload <\/em>in Figure 7) while discarding the rest.<\/p>\n\n\n\n<p>Once the gradients are available on the CPU, optimizer state partitions are updated in parallel by each data parallel process directly on the CPU (<em>p update<\/em> in Figure 7). After the update, parameter partitions are moved back to GPU followed by an all-gather operation on the GPU to gather all the updated parameters (<em>g swap <\/em>in Figure 7). ZeRO-Offload also exploits overlapping between communication (such as <em>g offload <\/em>and <em>g swap<\/em>) and computation (such as the backward pass and <em>p update<\/em>) using separate CUDA streams to maximize training efficiency.<\/p>\n\n\n\n\n\n<p><strong>10x model scale:<\/strong> On a single 32 GB V100 GPU, Figure 6 shows that the biggest model that can be trained by PyTorch has 1.3 billion parameters, while ZeRO-Offload allows for training models of 13 billion parameters, which is 10 times bigger. This is because ZeRO-Offload keeps the optimizer states (which consume a large portion of GPU memory) in host memory during the entire training process while also offloading gradients to CPU as they are computed in the backward pass. As a result, the saved GPU memory can be used in hosting bigger models for training.<\/p>\n\n\n\n<p><strong>Efficient&nbsp;training&nbsp;throughput<\/strong>:&nbsp;Figure 8 shows&nbsp;that&nbsp;when training a 10-billion-parameter model,&nbsp;ZeRO-Offload&nbsp;provides&nbsp;over&nbsp;30 teraflops throughput per GPU even when training with only a single GPU, and its throughput increases close to perfect linearly with the increasing number of GPUs.&nbsp;<\/p>\n\n\n\n<p>ZeRO-Offload complements ZeRO-2&nbsp;well, supporting&nbsp;efficient training of large models on a small number of GPUs. From 1 to 16 GPUs,&nbsp;ZeRO-Offload turns the model training from infeasible to feasible by leveraging CPU memory, reducing GPU memory required for the model. On 32 GPUs,&nbsp;ZeRO-Offload slightly outperforms ZeRO-2; the improvement comes from additional memory savings on GPU from&nbsp;ZeRO-Offload, which allows training with larger batch sizes and increases the GPU computation efficiency despite the overhead of CPU offloading.&nbsp;With more GPUs&nbsp;(such as&nbsp;64&nbsp;and&nbsp;128), ZeRO-2 outperforms&nbsp;ZeRO-Offload since both can now run similar batch sizes.&nbsp;On one hand,&nbsp;though,&nbsp;ZeRO-2&nbsp;does&nbsp;not have the overhead of moving data to CPU, while on the other hand, the optimizer step calculation on GPU is much faster than on CPU. In summary,&nbsp;ZeRO-Offload&nbsp;complements&nbsp;ZeRO-2 and extends&nbsp;ZeRO&nbsp;family of optimizations to cover the full spectrum of large model training from a single device to thousands of devices.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"403\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure8_HighRes-1024x403.jpg\" alt=\"Bar graph showing The training throughput is compared for ZeRO-Offload and ZeRO-2 using 128 GPUs to train a 10-billion parameter GPT-2 model. \" class=\"wp-image-690729\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure8_HighRes-1024x403.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure8_HighRes-300x118.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure8_HighRes-768x302.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure8_HighRes-1536x605.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure8_HighRes.jpg 1555w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 8: The training throughput is compared for ZeRO-Offload and ZeRO-2 using 128 GPUs to train a 10-billion parameter GPT-2 model. <\/figcaption><\/figure>\n\n\n\n\n\n<h2 id=\"deepspeed-sparse-attention-powering-10x-longer-sequences-with-6x-faster-execution\">DeepSpeed Sparse Attention: Powering 10x longer sequences with 6x faster execution<\/h2>\n\n\n\n<p>Attention-based deep learning models, such as Transformers, are highly effective in capturing relationships between tokens in an input sequence, even across long distances. As a result, they are used with text, image, and sound-based inputs, where the sequence length can be in thousands of tokens. However, despite the effectiveness of attention modules to capture long term dependencies, in practice their application to long sequence input is limited by compute and memory requirements of the attention computation that grow quadratically, \\(O(n^2 )\\), with the sequence length \\(n\\).<\/p>\n\n\n\n<p>To address this limitation, <strong>DeepSpeed offers a suite of sparse attention kernels<\/strong>\u2014an instrumental technology that can reduce the compute and memory requirement of attention computation by orders of magnitude via block-sparse computation. The suite not only alleviates the memory bottleneck of attention calculation, but also performs sparse computation efficiently. Its APIs allow convenient integration with any transformer-based models. Along with providing a wide spectrum of sparsity structures, it has the flexibility of handling any user-defined block-sparse structures. <\/p>\n\n\n\n<p>More specifically, sparse attention (SA) can be designed to compute local attention between nearby tokens, or global attention via summary tokens computed with local attention. Moreover, SA can also allow random attention or any combination of local, global, and random attention as shown in Figure 10 with blue, orange, and green blocks, respectively. As a result, SA decreases the memory footprint to \\(O (wn)\\), in which 1\\(&lt;w \u2264 n\\) is a parameter, whose value depends on the attention structure.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure10_Sparse-Attention_highres.jpg\" alt=\"Box of colorful small blocks that shows the Variable Sparsity structure\" class=\"wp-image-690732\" width=\"423\" height=\"419\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure10_Sparse-Attention_highres.jpg 846w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure10_Sparse-Attention_highres-300x297.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure10_Sparse-Attention_highres-150x150.jpg 150w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure10_Sparse-Attention_highres-768x761.jpg 768w\" sizes=\"(max-width: 423px) 100vw, 423px\" \/><figcaption>Figure 10: Variable Sparsity structure<\/figcaption><\/figure>\n\n\n\n<p><strong>Efficient implementation on GPUs: <\/strong>While a basic implementation of sparse attention may show a benefit of memory savings, computationally it can be even worse than full computation. This is mainly due to the divergence and un-coalesced memory access that sparse data adds to the full picture. In general, developing efficient sparse kernels, particularly on GPUs, is challenging. DeepSpeed offers efficient sparse attention kernels developed in <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/ptillet\/triton\">Triton<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. These kernels are structured in block-sparse paradigm that enables aligned memory access, alleviates thread divergence, and balances workloads on processors.<\/p>\n\n\n\n<p><strong>System performance: <\/strong>SA powers over<strong> 10x longer sequences <\/strong>and <strong>up to 6.3x faster computation <\/strong>as shown in Figure 11. The left figure shows the longest sequence length runnable in BERT-Base and BERT-Large models under three settings: dense, dense with activation checkpoint, and sparse (SA) with activation checkpoint. SA empowers 10x and 16x longer sequences when compared with dense for BERT-Base and BERT-Large, respectively. Furthermore, SA reduces total computation compared with dense and improves training speed: the boost is higher with increased sequence length, and it is up to 6.3x faster for BERT-Base and 5.3x for BERT-Large.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/Blog_DeepSpeed3_Figure11_updated-v3-1024x465.jpg\" target=\"_blank\" rel=\"noopener noreferrer\"><img loading=\"lazy\" decoding=\"async\" width=\"1248\" height=\"702\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-11_final.jpg\" alt=\"Figure 11: Maximum possible sequence length for BERT models (left); Training time of BERT-Base (center) and BERT-Large (right) on a single NVIDIA V100 GPU with varying sequence length.\" class=\"wp-image-690876\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-11_final.jpg 1248w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-11_final-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-11_final-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-11_final-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-11_final-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-11_final-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-11_final-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-11_final-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-11_final-960x540.jpg 960w\" sizes=\"(max-width: 1248px) 100vw, 1248px\" \/><\/a><figcaption>Figure 11: Maximum possible sequence length for BERT models (left); Training time of BERT-Base (center) and BERT-Large (right) on a single NVIDIA V100 GPU with varying sequence length.<\/figcaption><\/figure>\n\n\n\n\n\n<p>Related works along the line of sparse attention (<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/1904.10509.pdf\">Sparse Transformer<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/2004.05150.pdf\">Longformer<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/2007.14062.pdf\">BigBird<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>) have shown comparable or higher accuracy than full attention. Our experience is well aligned. In addition to lower memory overhead and faster computation, we also observe cases in production models where SA reaches higher accuracy and faster convergence. The following chart illustrates the accuracy of training a production model based on BERT for <em>long document comprehension <\/em>(2,048 sequence length). The experiment is performed in three settings: dense starting from scratch, SA starting from scratch, and SA continued training from a checkpoint of using dense with a sequence length of 512. We have observed that, for pretraining from scratch, SA converges faster with higher accuracy compared with dense. Furthermore, continuing training from a pretrained checkpoint with SA performs even better, with respect to both time and accuracy.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1084\" height=\"759\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-12_update.jpg\" alt=\"Figure 12: Accuracy of long document comprehension application\" class=\"wp-image-690756\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-12_update.jpg 1084w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-12_update-300x210.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-12_update-1024x717.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Figure-12_update-768x538.jpg 768w\" sizes=\"(max-width: 1084px) 100vw, 1084px\" \/><figcaption>Figure 12: Accuracy of long document comprehension application<\/figcaption><\/figure>\n\n\n\n\n\n<p>We compared SA with Longformer, a state-of-the-art sparse structure and implementation. In our experiment, SA uses \u201c<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/1904.10509.pdf\">Fixed<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>\u201d sparsity, and two implementations have comparable accuracy. On system performance, SA outperforms Longformer both in training and inference:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>1.5x faster execution pretraining MLM on Wikitext103<\/li><li>3x faster execution inference on BERT-Base (batch size 1, sequence length 2,048)<\/li><\/ul>\n\n\n\n\n\n<p><strong>Flexibility to handle any block-sparse structure:<\/strong> DeepSpeed sparse attention suite does not target any specific sparse structure but enables model scientists to explore any block sparse structure with efficient system support. Currently, we have added popular sparse structures, like <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/1904.10509.pdf\">Fixed <span class=\"sr-only\"> (opens in new tab)<\/span><\/a>(from OpenAI Sparse Transformer), <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/2007.14062.pdf\">BigBird <span class=\"sr-only\"> (opens in new tab)<\/span><\/a>(from Google), and BSLongformer (Block-Sparse implementation of AI2 <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/pdf\/2004.05150.pdf\">Longformer<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>). We also define a template to have \u201cvariable\u201d structure, shown in Figure 10, which can be used to simply customize any block-sparse random, local, or global attention pattern.<\/p>\n\n\n\n<h2 id=\"1-bit-adam-5x-less-communication-and-3-4x-faster-training\">1-bit Adam: 5x less communication and 3.4x faster training<\/h2>\n\n\n\n<p>Scalable training of large models (like BERT and GPT-3) requires careful optimization rooted in model design, architecture, and system capabilities. From a system standpoint, communication has become a major bottleneck, especially on commodity systems with standard TCP interconnects that offer limited network bandwidth.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--left\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication <\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/1-bit-adam-communication-efficient-large-scale-training-with-adams-convergence-speed\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"1-bit Adam: Communication Efficient Large-Scale Training with Adam\u2019s Convergence Speed\" data-bi-aN=\"margin-callout\" data-bi-cN=\"1-bit Adam: Communication Efficient Large-Scale Training with Adam\u2019s Convergence Speed\">\n\t\t\t\t1-bit Adam: Communication Efficient Large-Scale Training with Adam\u2019s Convergence Speed&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>Communication compression is an important technique to reduce training time on such systems. One of the most effective ways to compress communication is via error compensation compression, which offers robust convergence speed, even under 1-bit compression. However, state-of-the-art error compensation techniques only work with basic optimizers like Stochastic Gradient Descent (SGD) and momentum SGD, which are linearly dependent on the gradients. They do not work with non-linear gradient-based optimizers like Adam, which offers state-of-the-art convergence efficiency and accuracy for many tasks, including training of BERT-like models.<\/p>\n\n\n\n<p>For a powerful optimizer like Adam, the non-linear dependency on gradient (in the variance term) makes it challenging to develop error compensation-based compression techniques, limiting the practical value of the state-of-the-art communication compression techniques.<\/p>\n\n\n\n\n\n<p>One way of communication compression is 1-bit compression, which can be expressed as:<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/LaTex-Equation-1_DeepSpeed-1.jpg\" alt=\"latex equation\" class=\"wp-image-689982\" width=\"301\" height=\"110\"\/><\/figure>\n\n\n\n<p>With this compression, we could achieve a 32x reduction of memory size by representing each number using one bit. The problem is that using this straightforward method would significantly degrade the convergence speed, which makes this method inapplicable. To solve this problem, recent studies show that by using error compensation compression, we could expect almost the same convergence rate with communication compression.<\/p>\n\n\n\n<p>The idea of error compensation can be summarized as: 1) doing compression, 2) memorizing the compression error, and then 3) adding the compression error back in during the next iteration. For SGD, doing error compression leads to:<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/LaTex-Equation_2_DeepSpeed.jpg\" alt=\"latex equation\" class=\"wp-image-689985\" width=\"778\" height=\"89\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/LaTex-Equation_2_DeepSpeed.jpg 688w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/LaTex-Equation_2_DeepSpeed-300x34.jpg 300w\" sizes=\"(max-width: 778px) 100vw, 778px\" \/><\/figure>\n\n\n\n<p>Where \\(C(\u22c5)\\) is the 1-bit compression operator. The good thing about doing this error compensation is that the history compression error \\(e_t\\)  and \\(e_t-1\\)  would be canceled by itself eventually, which can be seen by:<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/LaTex-Equation-3_DeepSpeed.jpg\" alt=\"latex equation\" class=\"wp-image-689988\" width=\"418\" height=\"75\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/LaTex-Equation-3_DeepSpeed.jpg 417w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/LaTex-Equation-3_DeepSpeed-300x53.jpg 300w\" sizes=\"(max-width: 418px) 100vw, 418px\" \/><\/figure>\n\n\n\n<p>This strategy has been proven to work for all optimization algorithms that are linearly dependent on the gradient, such as SGD and Momentum SGD.<\/p>\n\n\n\n\n\n<p>We provide an overview of the Adam algorithm below. The update rules are as follows.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/LaTex-Equations-3_DeepSpeed.jpg\" alt=\"latex equation\" class=\"wp-image-689994\" width=\"722\" height=\"278\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/LaTex-Equations-3_DeepSpeed.jpg 638w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/LaTex-Equations-3_DeepSpeed-300x115.jpg 300w\" sizes=\"(max-width: 722px) 100vw, 722px\" \/><\/figure>\n\n\n\n<p>As shown in the equations above, the variance term \\(v_t\\) is nonlinearly dependent on the gradient \\(g_t\\). If we apply basic error compensation compression to Adam, we observe that Adam will not converge as shown in Figure 13.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"814\" height=\"625\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-1-_Section-5.png\" alt=\"Figure 13: Inapplicability of Error-compensation Compression for Adam due to non-linear dependence on the gradient\" class=\"wp-image-689997\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-1-_Section-5.png 814w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-1-_Section-5-300x230.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-1-_Section-5-768x590.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-1-_Section-5-80x60.png 80w\" sizes=\"(max-width: 814px) 100vw, 814px\" \/><figcaption>Figure 13: Inapplicability of Error-compensation Compression for Adam due to non-linear dependence on the gradient<\/figcaption><\/figure>\n\n\n\n\n\n<h4 id=\"compressing-communication-with-1-bit-adam\">Compressing communication with 1-bit Adam<\/h4>\n\n\n\n<p>To compress communication while using the Adam optimizer, we develop <strong>1-bit Adam<\/strong>, which addresses the non-linearity in gradients via preconditioning. We observe that the magnitude of changes on the non-linear term, variance (\\(v_t\\)), decrease significantly after a few epochs of training and setting \\(v_t\\) constant afterwards will not change the convergence speed. The proposed 1-bit Adam optimizer, as shown in Figure 14, consists of two parts: the <strong>warmup<\/strong> stage, which is essentially the vanilla Adam algorithm; and the <strong>compression <\/strong>stage, which keeps the variance term constant and compresses the remaining linear term, that is the momentum, into 1-bit representation.<\/p>\n\n\n\n<p>The compression stage of the algorithm is controlled by a threshold parameter (as shown in Figure 14). When we detect that the change in \u201cvariance\u201d falls below a certain threshold, we switch to the compression stage. Our study shows that only 15-20% of the overall training steps are needed for the warmup stage.<\/p>\n\n\n\n\n\n<p>The weight update rule for 1-bit Adam is governed by the following equations. For the <em>i<\/em><img loading=\"lazy\" decoding=\"async\" width=\"5\" height=\"18\" src=\"\"><sup>&nbsp;th<\/sup> worker, in the compression stage:<\/p>\n\n\n\n<figure class=\"wp-block-image alignwide size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"392\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Equations-5_Latex-1024x392.jpg\" alt=\"equations\" class=\"wp-image-690000\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Equations-5_Latex-1024x392.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Equations-5_Latex-300x115.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Equations-5_Latex-768x294.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-Equations-5_Latex.jpg 1183w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"975\" height=\"526\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2_Section-4.png\" alt=\"Figure 14: Comparison of distributed training steps in classic Adam and the proposed 1-bit compressed Adam algorithm\" class=\"wp-image-689943\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2_Section-4.png 975w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2_Section-4-300x162.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-2_Section-4-768x414.png 768w\" sizes=\"(max-width: 975px) 100vw, 975px\" \/><figcaption>Figure 14: Comparison of distributed training steps in classic Adam and the proposed 1-bit compressed Adam algorithm<\/figcaption><\/figure>\n\n\n\n<h4 id=\"addressing-system-challenges-for-1-bit-adam\">Addressing system challenges for 1-bit Adam<\/h4>\n\n\n\n<p>Besides the algorithmic challenge, there are two system challenges in applying 1-bit Adam in training systems. First, we need efficient kernels that convert the momentum to 1-bit representations. Second, we need efficient communication schemes to exchange this compressed momentum across different GPUs. The goal of compression is to reduce the overall training time so that commodity systems with bandwidth-limited interconnects can be used to train large models. We address these challenges in DeepSpeed and introduce a fully optimized 1-bit Adam implementation for training on communication-constrained systems.<\/p>\n\n\n\n<h4 id=\"benefits-of-1-bit-adam-on-communication-constrained-systems\">Benefits of 1-bit Adam on communication-constrained systems<\/h4>\n\n\n\n<p>1-bit Adam offers the same convergence as Adam, incurs up to <strong>5x less communication<\/strong> that enables up to <strong>3.5x higher throughput<\/strong> for BERT-Large pretraining and up to <strong>2.7x higher throughput<\/strong> for SQuAD fine-tuning. This end-to-end throughput improvement is enabled by the 6.6x (Figure 15 left) and 6.2x (Figure 15 right) speedup observed during the compression stage. It is worth mentioning that our 1-bit Adam optimizer scales so well on a 40 Gigabit Ethernet system that its performance is comparable to Adam\u2019s scalability on a 40 Gigabit InfiniBand QDR system. We note that the effective bandwidth on 40 Gigabit Ethernet is 4.1 Gbps based on iPerf benchmarks, whereas InfiniBand provides near-peak bandwidth of 32 Gbps based on InfiniBand perftest microbenchmarks.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"349\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3-Figure-3-Section-4-1024x349.jpg\" alt=\"Figure 15: Scalability of 1-bit Adam for BERT-Large Pretraining (left) and SQuAD Fine-tuning (right) on NVIDIA V100 GPUs. The batch sizes are 16\/GPU and 3\/GPU for BERT pretraining and SQuAD fine-tuning, respectively.\" class=\"wp-image-689946\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3-Figure-3-Section-4-1024x349.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3-Figure-3-Section-4-300x102.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3-Figure-3-Section-4-768x262.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3-Figure-3-Section-4.jpg 1459w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 15: Scalability of 1-bit Adam for BERT-Large Pretraining (left) and SQuAD Fine-tuning (right) on NVIDIA V100 GPUs. The batch sizes are 16\/GPU and 3\/GPU for BERT pretraining and SQuAD fine-tuning, respectively.<\/figcaption><\/figure>\n\n\n\n\n\n<p><strong>Same convergence as Adam<\/strong>: One major question for using 1-bit Adam is the convergence speed, and we find that 1-bit Adam can achieve the same convergence speed and comparable testing performance using the same number of training samples as shown in Figure 16.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"892\" height=\"651\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-4_Section-5.png\" alt=\"Figure 16: 1-bit Adam converges like Adam using the same number of training samples.\" class=\"wp-image-690003\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-4_Section-5.png 892w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-4_Section-5-300x219.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-4_Section-5-768x561.png 768w\" sizes=\"(max-width: 892px) 100vw, 892px\" \/><figcaption>Figure 16: 1-bit Adam converges like Adam using the same number of training samples.<\/figcaption><\/figure>\n\n\n\n<p>Detailed BERT-Base and BERT-Large results are shown in Table 1. We see that the scores are on par with or better than the original model for both the uncompressed and compressed cases.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"810\" height=\"170\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed_Table-1.png\" alt=\"table 1\" class=\"wp-image-690006\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed_Table-1.png 810w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed_Table-1-300x63.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed_Table-1-768x161.png 768w\" sizes=\"(max-width: 810px) 100vw, 810px\" \/><figcaption>Table 1: Verifying correctness of 1-bit Adam on various testing tasks<\/figcaption><\/figure>\n\n\n\n<p><strong>Up to 5x less communication:<\/strong> 1-bit Adam provides the same convergence as Adam and reduces the communication volume by 16x during the compression stage for 16-bit (FP16) training. For BERT pretraining, this leads to an overall communication reduction of 5x as we observed the warmup stage to be just 15% of the end-to-end training time.<\/p>\n\n\n\n<p>The formula to calculate the communication volume ratio of the original versus 1-bit Adam is as follows:<\/p>\n\n\n\n<p class=\"has-text-align-center\">1 \/ (warmup + (1 \u2013 warmup)\/16)<\/p>\n\n\n\n<p><strong>1-bit Adam is 3.5x faster for training BERT-Large: <\/strong>We present two main results for training BERT-Large on systems with two different bandwidth-limited interconnects: 1) 40 Gbps Ethernet (Figure 17 left) and 2) 40 Gbps InfiniBand QDR (Figure 17 right). During the compression phase, we observe up to 6.6x higher throughput on the system with Ethernet and up to 2x higher throughput on the system with InfiniBand, resulting in end-to-end speed up (including both warmup and compression stages) of 3.5x and 2.7x, respectively. The major benefit of 1-bit Adam comes from the communication volume reduction\u2014enabled by our compressed momentum exchange\u2014and from our custom <em>allreduce<\/em> operation that implements efficient 1-bit communication using non-blocking gather operations followed by an <em>allgather<\/em> operation.<\/p>\n\n\n\n<p>It is important to note that one can also increase total batch size to reduce communication using optimizers like LAMB instead of Adam for BERT pretraining. However, 1-bit Adam avoids the need for rigorous hyperparameter tuning, which is often more difficult for large batches from our experience. Furthermore, 1-bit Adam also works very well for workloads that have small critical batch size (cannot converge well with large batch size) like many fine-tuning tasks.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"343\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-5_Section-5-1024x343.jpg\" alt=\"Figure 17: Performance of 1-bit Adam for BERT-Large training on 40 Gbps Ethernet (left) and InfiniBand (right) interconnect during the compression stage.\" class=\"wp-image-690009\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-5_Section-5-1024x343.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-5_Section-5-300x101.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-5_Section-5-768x257.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-5_Section-5.jpg 1313w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 17: Performance of 1-bit Adam for BERT-Large training on 40 Gbps Ethernet (left) and InfiniBand (right) interconnect during the compression stage.<\/figcaption><\/figure>\n\n\n\n<p><strong>1-bit Adam is 2.7x faster for SQuAD fine-tuning: <\/strong>1-bit Adam offers scalability not only on large-scale training tasks but also on tasks like SQuAD fine-tuning. As shown in Figure 18, 1-bit Adam scales well on both Ethernet- and InfiniBand-based systems and offers up to 6.2x higher throughput (during the compression stage) on the Ethernet-based system, resulting in 2.7x end-to-end speedup (25% warmup plus 75% compression stage). For SQuAD fine-tuning, we observed that a total batch size of 96 offers the best F1 score. Batch sizes larger than this value lower the convergence rate and require additional hyperparameter tuning. Therefore, in order to scale to 32 GPUs, we can only apply a small batch size of 3-4 per GPU. This makes fine-tuning tasks communication intensive and hard to scale. 1-bit Adam addresses the scaling challenge well, obtaining 3.4x communication reduction without enlarging batch size, and it results in a 2.7x end-to-end speedup.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"316\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-6_Section-5-1024x316.jpg\" alt=\"Figure 18: Performance of 1-bit Adam for SQuAD fine-tuning on 40 Gbps Ethernet (left) and InfiniBand (right) interconnect during the compression stage.\" class=\"wp-image-690012\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-6_Section-5-1024x316.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-6_Section-5-300x93.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-6_Section-5-768x237.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed-3_Figure-6_Section-5.jpg 1295w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>Figure 18: Performance of 1-bit Adam for SQuAD fine-tuning on 40 Gbps Ethernet (left) and InfiniBand (right) interconnect during the compression stage.<\/figcaption><\/figure>\n\n\n\n\n\n<p>Please visit <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/deepspeed\/\">DeepSpeed website<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/DeepSpeed\">Github repository<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> for the codes, tutorials and documentations about these new technologies! We are also integrating some of these techniques into&nbsp;<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/aka.ms\/onnxruntime\">ONNX Runtime<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n<h3 id=\"about-our-great-collaborators\">About our great collaborators:<\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>We would like to acknowledge our academic collaborator, Philippe Tillet from Harvard University, for helping us co-develop sparse attention kernels through <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/www.eecs.harvard.edu\/~htk\/publication\/2019-mapl-tillet-kung-cox.pdf\">Triton<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> compiler.<\/li><li>ZeRO-Offload was co-developed with our intern Jie Ren from UC Merced. We would also like to thank Dong Li from UC Merced, as well as Bharadwaj Pudipeddi and Maral Mesmakhouroshahi from Microsoft <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2002.05645\">L2L work<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, for their discussions on the topic.<\/li><li>1-bit Adam was co-developed with our intern Hanlin Tang from University of Rochester. <\/li><li>We would like to thank the great collaboration from NVIDIA, especially the Megatron-LM team.<\/li><\/ul>\n\n\n\n<h3 id=\"about-the-deepspeed-team\">About the DeepSpeed Team:<\/h3>\n\n\n\n<p>We are a group of system researchers and engineers\u2014Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Reza Yazdani Aminabadi, Elton Zheng, Arash Ashari, Jing Zhao, Minjia Zhang, Niranjan Uma Naresh, Shaden Smith, Ammar Ahmad Awan, Conglong Li, Yuxiong He (team lead) \u2014 who are enthusiastic about performance optimization of large-scale systems. We have recently focused on deep learning systems, optimizing deep learning\u2019s speed to train, speed to convergence, and speed to develop!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In February, we announced DeepSpeed, an open-source deep learning training optimization library, and ZeRO (Zero Redundancy Optimizer), a novel memory optimization technology in the library, which vastly advances large model training by improving scale, speed, cost, and usability. DeepSpeed has enabled researchers to create Turing Natural Language Generation (Turing-NLG), the largest language model with 17 [&hellip;]<\/p>\n","protected":false},"author":38838,"featured_media":690915,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556,13545],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-689370","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-human-language-technologies","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[678390,649749],"related-events":[],"related-researchers":[{"type":"guest","value":"deepspeed-team","user_id":"690909","display_name":"DeepSpeed Team","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/deepspeed\/#!people\" aria-label=\"Visit the profile page for DeepSpeed Team\">DeepSpeed Team<\/a>","is_active":true,"last_first":"Team, DeepSpeed","people_section":0,"alias":"deepspeed-team"},{"type":"user_nicename","value":"Rangan Majumder","user_id":38931,"display_name":"Rangan Majumder","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/ranganm\/\" aria-label=\"Visit the profile page for Rangan Majumder\">Rangan Majumder<\/a>","is_active":false,"last_first":"Majumder, Rangan","people_section":0,"alias":"ranganm"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"500\" height=\"281\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed_3_FeatureImage_MCRSite.png\" class=\"img-object-cover\" alt=\"\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed_3_FeatureImage_MCRSite.png 500w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed_3_FeatureImage_MCRSite-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/09\/DeepSpeed_3_FeatureImage_MCRSite-343x193.png 343w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/>","byline":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/deepspeed\/#!people\" title=\"Go to researcher profile for DeepSpeed Team\" aria-label=\"Go to researcher profile for DeepSpeed Team\" data-bi-type=\"byline author\" data-bi-cN=\"DeepSpeed Team\">DeepSpeed Team<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/ranganm\/\" title=\"Go to researcher profile for Rangan Majumder\" aria-label=\"Go to researcher profile for Rangan Majumder\" data-bi-type=\"byline author\" data-bi-cN=\"Rangan Majumder\">Rangan Majumder<\/a>, and Junhua Wang","formattedDate":"September 10, 2020","formattedExcerpt":"In February, we announced DeepSpeed, an open-source deep learning training optimization library, and ZeRO (Zero Redundancy Optimizer), a novel memory optimization technology in the library, which vastly advances large model training by improving scale, speed, cost, and usability. DeepSpeed has enabled researchers to create Turing&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/689370"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38838"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=689370"}],"version-history":[{"count":99,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/689370\/revisions"}],"predecessor-version":[{"id":894474,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/689370\/revisions\/894474"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/690915"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=689370"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=689370"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=689370"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=689370"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=689370"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=689370"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=689370"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=689370"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=689370"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=689370"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=689370"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}