{"id":811273,"date":"2022-01-19T09:19:40","date_gmt":"2022-01-19T17:19:40","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=811273"},"modified":"2023-06-12T10:08:04","modified_gmt":"2023-06-12T17:08:04","slug":"deepspeed-advancing-moe-inference-and-training-to-power-next-generation-ai-scale","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/deepspeed-advancing-moe-inference-and-training-to-power-next-generation-ai-scale\/","title":{"rendered":"DeepSpeed: Advancing MoE inference and training to power next-generation AI scale"},"content":{"rendered":"\n<figure class=\"wp-block-image alignwide size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"1441\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-scaled.jpg\" alt=\"DeepSpeed shares findings and innovations for MoE models and systems that 1) reduce training cost by 5x, 2) reduce MoE parameter size by up to 3.7x and 3) reduce MoE inference latency by 7.3x at an unprecedented scale and offer up to 4.5x faster and 9x cheaper inference for MoE models compared to quality-equivalent dense models.\" class=\"wp-image-813712\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-scaled.jpg 2560w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-1536x865.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-2048x1153.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-scaled-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-scaled-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-1920x1080.jpg 1920w\" sizes=\"(max-width: 2560px) 100vw, 2560px\" \/><\/figure>\n\n\n<aside id=accordion-803de00d-6f70-43f5-a481-d72232138710 class=\"msr-table-of-contents-block accordion mb-5 pb-0\" data-bi-aN=\"table-of-contents\">\n\t<button class=\"btn btn-collapse bg-gray-100 mb-0 display-flex justify-content-between\" type=\"button\" data-mount=\"collapse\" data-target=\"#accordion-collapse-803de00d-6f70-43f5-a481-d72232138710\" aria-expanded=\"true\" aria-controls=\"accordion-collapse-803de00d-6f70-43f5-a481-d72232138710\">\n\t\t<span class=\"msr-table-of-contents-block__label subtitle\">In this article<\/span>\n\t\t<span class=\"msr-table-of-contents-block__current mr-4 text-gray-600 font-weight-normal\" aria-hidden=\"true\"><\/span>\n\t<\/button>\n\t<div id=\"accordion-collapse-803de00d-6f70-43f5-a481-d72232138710\" class=\"msr-table-of-contents-block__collapse-wrapper collapse show\" data-parent=\"#accordion-803de00d-6f70-43f5-a481-d72232138710\">\n\t\t<div class=\"accordion-body bg-gray-100 border-top pt-4\">\n\t\t\t<ol class=\"msr-table-of-contents-block__list\">\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"#deepspeed-moe-for-nlg-reducing-the-training-cost-of-language-models-by-five-times\" class=\"msr-table-of-contents-block__list-item-link\">DeepSpeed-MoE for NLG: Reducing the training cost of language models by five times<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"#pr-moe-and-mixture-of-students-reducing-the-model-size-and-improving-parameter-efficiency\" class=\"msr-table-of-contents-block__list-item-link\">PR-MoE and Mixture-of-Students: Reducing the model size and improving parameter efficiency<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"#deepspeed-moe-inference-serving-moe-models-at-unprecedented-scale-and-speed\" class=\"msr-table-of-contents-block__list-item-link\">DeepSpeed-MoE inference: Serving MoE models at unprecedented scale and speed<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"#looking-forward-to-the-next-generation-of-ai-scale\" class=\"msr-table-of-contents-block__list-item-link\">Looking forward to the next generation of AI Scale<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<\/ul>\n\t\t<\/div>\n\t<\/div>\n\t<span class=\"msr-table-of-contents-block__progress-bar\"><\/span>\n<\/aside>\n\n\n\n<p>In the last three years, the largest trained dense models have increased in size by over 1,000 times, from a few hundred million parameters to over 500 billion parameters in <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model\/\">Megatron-Turing NLG 530B<\/a> (MT-NLG). Improvements in model quality with size suggest that this trend will continue, with larger model sizes bringing better model quality. However, sustaining the growth in model size is getting more difficult due to the increasing compute requirements.<\/p>\n\n\n\n<p>There have been numerous efforts to reduce compute requirements to train large models without sacrificing model quality. To this end, architectures based on <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/deepspeed-powers-8x-larger-moe-model-training-with-high-performance\/\">Mixture of Experts (MoE)<\/a> have paved a promising path, enabling sub-linear compute requirements with respect to model parameters and allowing for improved model quality without increasing training cost.<\/p>\n\n\n\n<p>However, MoE models have their own challenges. First, the scope of MoE models is primarily limited on encoder-decoder models and sequence-to-sequence tasks. Second, MoE models require more parameters to achieve the same model quality as their dense counterparts, which requires more memory for training and inference even though MoE models require less compute. Lastly, a critical consideration is that MoE models\u2019 large size makes inference difficult and costly.<\/p>\n\n\n\n<p>To address these above challenges, the DeepSpeed team, as part of Microsoft\u2019s \u202f<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/ai-at-scale\/\">AI at Scale<\/a> initiative, has been exploring new applications and optimizations for MoE models at scale. These can lower the training and inference cost of large models, while also enabling the ability to train and serve the next generation of models affordably on today&#8217;s hardware. Here, we are happy to share our findings and innovations for MoE models and systems that 1) <strong>reduce training cost by 5x<\/strong>, 2) <strong>reduce MoE parameter size by up to 3.7x<\/strong> <strong>and <\/strong>3) <strong>reduce MoE inference latency by 7.3x at an unprecedented scale and offer up to 4.5x faster and 9x cheaper inference for MoE models compared to quality-equivalent dense models:<\/strong><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>5x reduction in training cost for natural language generation (NLG) models<\/strong>: We extend the scope of MoE models to beyond just encoder-decoder models and sequence-to-sequence tasks, demonstrating that MoE can reduce the training cost of NLG models like those in the GPT family or MT-NLG by 5x while obtaining the same model quality. Data scientists can now train models of superior quality previously only possible with 5x more hardware resources.<\/li>\n\n\n\n<li><strong><strong>Reduced model size and improved parameter efficiency with Pyramid-Residual-MoE (PR-MoE)<\/strong> <strong>Architecture and Mixture-of-Students (MoS):<\/strong> <\/strong>The training cost reduction of MoE is not free and comes at the expense of increasing the total number of parameters required to achieve the same model quality as dense models. PR-MoE is a hybrid dense and MoE model created using residual connections, applying experts only where they are most effective. PR-MoE reduces MoE model parameter size by up to 3x with no change to model quality. In addition, we leverage staged knowledge distillation to learn a Mixture-of-Students model that further leads to up to 3.7x model size reduction while retaining similar model quality.<\/li>\n\n\n\n<li><strong>Fast and economical MoE inference at unprecedented scale:<\/strong> The DeepSpeed-MoE (DS-MoE) inference system enables efficient scaling of inference workloads on hundreds of GPUs, providing up to 7.3x reduction in inference latency and cost when compared with existing systems. It offers ultra-fast inference latencies (25 ms) for trillion-parameter MoE models. DS-MoE also offers up to 4.5x faster and 9x cheaper inference for MoE models compared to quality-equivalent dense models by combining both system and model optimizations.<\/li>\n<\/ol>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--left\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">PUBLICATION<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/deepspeed-moe-advancing-mixture-of-experts-inference-and-training-to-power-next-generation-ai-scale\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale\" data-bi-aN=\"margin-callout\" data-bi-cN=\"DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale\">\n\t\t\t\tDeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>Each of these advances is explored further in the blog post below. For more about the technical details, please read our <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/deepspeed-moe-advancing-mixture-of-experts-inference-and-training-to-power-next-generation-ai-scale\/\">paper<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"deepspeed-moe-for-nlg-reducing-the-training-cost-of-language-models-by-five-times\">DeepSpeed-MoE for NLG: Reducing the training cost of language models by five times<\/h2>\n\n\n\n<p>While recent works like <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2006.16668\">GShard<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2101.03961\">Switch Transformers<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> have shown that the MoE model structure can reduce large model pretraining cost for encoder-decoder model architecture, their impact on the much more compute-intensive transformer-based autoregressive NLG models has been mostly unknown.<\/p>\n\n\n\n<p>Given the tremendous compute and energy requirements for training NLG models, we explore opportunities where MoE can reduce their training cost.\u202f<strong>We show that MoE can be applied to NLG models to significantly improve their model quality with the same training cost.\u202fAlso, MoE\u202fcan achieve 5x reduction in training cost to achieve the same model quality of a dense NLG model.<\/strong>\u202fFor example, we achieved the quality of a 6.7B-parameter dense NLG model at the cost of training a 1.3B-parameter dense model.&nbsp; Our observation about MoE training cost savings aligns with parallel explorations from <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2112.06905\" target=\"_blank\" rel=\"noreferrer noopener\">Du et al.<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2112.10684\" target=\"_blank\" rel=\"noreferrer noopener\">Artetxe et al.<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, where they also demonstrated the savings for models with bigger sizes.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"our-moe-based-nlg-model-architecture\">Our MoE-based NLG model architecture<\/h3>\n\n\n\n<p>To create an MoE-based NLG model, we studied a transformer-based NLG model similar to those of the GPT family. To complete training in a reasonable timeframe, the following models were selected: 350M (24 layers, 1024 hidden size, 16 attention heads), 1.3B (24 layers, 2048 hidden size, 16 attention heads), and 6.7B (32 layers, 4096 hidden size, 32 attention heads). We use \u201c350M+MoE-128\u201d to denote a MoE model that uses 350M dense model as the base model and adds 128 experts on every other feedforward layer.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"moe-training-infrastructure-and-dataset\">MoE training infrastructure and dataset<\/h3>\n\n\n\n<p>We pretrained both the dense and MoE versions of the above models using <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/deepspeed.ai\/\">DeepSpeed<\/a> on 128 NVIDIA Ampere A100 GPUs (<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/virtual-machines\/nda100-v4-series\">Azure ND A100 instances<\/a>). These Azure instances are powered by the latest <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/Azure\/azhpc-images\">Azure HPC docker images<\/a> that provide a fully optimized environment and best performing library versions of NCCL, Mellanox OFED, Sharp, and CUDA. DeepSpeed uses a combination of data-parallel and expert-parallel training to effectively scale&nbsp;<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/deepspeed-powers-8x-larger-moe-model-training-with-high-performance\/\">MoE model training<\/a> and is capable of training MoE models with trillions of parameters on hundreds of GPUs.<\/p>\n\n\n\n<p>We used the same training data as described in the&nbsp;<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model\/\">MT-NLG blog<\/a> post. For a fair comparison, we use 300 billion tokens to train both dense and MoE models.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"moe-leads-to-better-quality-for-nlg-models\">MoE leads to better quality for NLG models<\/h3>\n\n\n\n<p>Figure 1 shows that the validation loss for the MoE versions of the model is significantly better than their dense counterparts. Furthermore, validation loss of the 350M+MoE-128 model is on par validation loss of the 1.3B dense model with 4x larger base. This is also true for 1.3B+MoE-128 in comparison with 6.7B dense model with 5x larger base. Furthermore, the model quality is on par not only with the validation loss but also with six zero-shot evaluation tasks as shown in Table 1, demonstrating that these models have very similar model quality.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-regular alignwide\"><center><table><tbody><tr bgcolor=\"#0078d4\"><td style=\"color: white; vertical-align: middle\"><strong>Case<\/strong><\/td><td style=\"color: white; vertical-align: middle\"><strong>Model size<\/strong><\/td><td style=\"color: white; vertical-align: middle\"><strong>LAMBADA: completion prediction<\/strong><\/td><td style=\"color: white; vertical-align: middle\"><strong>PIQA: commonsense reasoning<\/strong><\/td><td style=\"color: white; vertical-align: middle\"><strong>BoolQ: reading comprehension<\/strong><\/td><td style=\"color: white; vertical-align: middle\"><strong>RACE-h: reading comprehension<\/strong><\/td><td style=\"color: white; vertical-align: middle\"><strong>TriviaQA: question answering<\/strong><\/td><td style=\"color: white; vertical-align: middle\"><strong>WebQs: question answering<\/strong><\/td><\/tr><tr bgcolor=\"#b2d6f2\"><td colspan=\"8\"><strong>Dense NLG:<\/strong><\/td><\/tr><tr><td>(1) 350M<\/td><td>350M<\/td><td>0.5203<\/td><td>0.6931<\/td><td>0.5364<\/td><td>0.3177<\/td><td>0.0321<\/td><td>0.0157<\/td><\/tr><tr><td>(2) 1.3B<\/td><td>1.3B<\/td><td>0.6365<\/td><td>0.7339<\/td><td>0.6339<\/td><td>0.3560<\/td><td>0.1005<\/td><td>0.0325<\/td><\/tr><tr><td>(3) 6.7B<\/td><td>6.7B<\/td><td>0.7194<\/td><td>0.7671<\/td><td>0.6703<\/td><td>0.3742<\/td><td>0.2347<\/td><td>0.0512<\/td><\/tr><tr bgcolor=\"#b2d6f2\"><td colspan=\"8\"><strong>Standard MoE NLG:<\/strong><\/td><\/tr><tr><td>(4) 350M+MoE-128<\/td><td>13B<\/td><td>0.6270<\/td><td>0.7459<\/td><td>0.6046<\/td><td>0.3560<\/td><td>0.1658<\/td><td>0.0517<\/td><\/tr><tr><td>(5) 1.3B+MoE-128<\/td><td>52B<\/td><td>0.6984<\/td><td>0.7671<\/td><td>0.6492<\/td><td>0.3809<\/td><td>0.3129<\/td><td>0.0719<\/td><\/tr><tr bgcolor=\"#b2d6f2\"><td colspan=\"8\"><strong>PR-MoE NLG:<\/strong><\/td><\/tr><tr><td>(6) 350M+PR-MoE-32\/64<\/td><td>4B<\/td><td>0.6365<\/td><td>0.7399<\/td><td>0.5988<\/td><td>0.3569<\/td><td>0.1630<\/td><td>0.0473<\/td><\/tr><tr><td>(7) 1.3B+PR-MoE-64\/128<\/td><td>31B<\/td><td>0.7060<\/td><td>0.7775<\/td><td>0.6716<\/td><td>0.3809<\/td><td>0.2886<\/td><td>0.0773<\/td><\/tr><tr bgcolor=\"#b2d6f2\"><td colspan=\"8\"><strong>PR-MoE NLG + MoS:<\/strong><\/td><\/tr><tr><td>(8) 350M+PR-MoE-32\/64 + MoS-21L<\/td><td>3.5B<\/td><td>0.6346<\/td><td>0.7334<\/td><td>0.5807<\/td><td>0.3483<\/td><td>0.1369<\/td><td>0.0522<\/td><\/tr><tr><td>(9) 1.3B+PR-MoE-64\/128 + MoS-21L<\/td><td>27B<\/td><td>0.7017<\/td><td>0.7769<\/td><td>0.6566<\/td><td>0.3694<\/td><td>0.2905<\/td><td>0.0822<\/td><\/tr><\/tbody><\/table><figcaption><em>Table 1: Zero-shot evaluation results (last six columns) for different dense and MoE NLG models. All zero-shot evaluation results use the accuracy metric.<\/em><\/figcaption><\/center><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Figure 1: Token-wise validation loss curves for dense and MoE NLG models with different model sizes. This shows that the validation loss for the MoE versions of the model is significantly better than their dense counterparts\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/moe_valid_loss_350_to_6p7.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1077\" height=\"740\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/moe_valid_loss_350_to_6p7.png\" alt=\"Figure 1: Token-wise validation loss curves for dense and MoE NLG models with different model sizes. This shows that the validation loss for the MoE versions of the model is significantly better than their dense counterparts\" class=\"wp-image-813478\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/moe_valid_loss_350_to_6p7.png 1077w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/moe_valid_loss_350_to_6p7-300x206.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/moe_valid_loss_350_to_6p7-1024x704.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/moe_valid_loss_350_to_6p7-768x528.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/moe_valid_loss_350_to_6p7-800x550.png 800w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/moe_valid_loss_350_to_6p7-240x165.png 240w\" sizes=\"(max-width: 1077px) 100vw, 1077px\" \/><\/a><figcaption class=\"wp-element-caption\">Figure 1: Token-wise validation loss curves for dense and MoE NLG models with different model sizes.<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"same-quality-with-5x-less-training-cost\">Same quality with 5x less training cost<\/h3>\n\n\n\n<p>As shown in the results above, adding MoE with 128 experts to the NLG model significantly improves its quality. However, these experts do not change the compute requirements of the model as each token is only processed by a single expert. Therefore, the compute requirements for a dense model and its corresponding MoE models with the same base are similar.<\/p>\n\n\n\n<p>More concretely, training 1.3B+MoE-128 requires roughly the same amount of compute operations as a 1.3B dense model while offering much better quality. Our results show that by applying MoE, the model quality of a 6.7B-parameter dense model can be achieved at the training cost of a 1.3B-parameter dense model, resulting in an effective training compute reduction of 5x.<\/p>\n\n\n\n<p>This compute cost reduction can directly be translated into throughput gain, training time and training cost reduction by leveraging the efficient DeepSpeed MoE training system. Table 2 shows the training throughput of 1.3B+MoE-128 compared with the 6.7B dense model on 128 NVIDIA A100 GPUs.<\/p>\n\n\n\n<figure class=\"wp-block-table aligncenter is-style-regular\"><table><thead><tr><th class=\"has-text-align-left\" data-align=\"left\"><\/th><th class=\"has-text-align-center\" data-align=\"center\">Training samples per sec<\/th><th class=\"has-text-align-center\" data-align=\"center\">Throughput gain\/ Cost Reduction<\/th><\/tr><\/thead><tbody><tr><td class=\"has-text-align-left\" data-align=\"left\"> 6.7B dense <\/td><td class=\"has-text-align-center\" data-align=\"center\">70<\/td><td class=\"has-text-align-center\" data-align=\"center\">1x<\/td><\/tr><tr><td class=\"has-text-align-left\" data-align=\"left\"> 1.3B+MoE-128 <\/td><td class=\"has-text-align-center\" data-align=\"center\">372<\/td><td class=\"has-text-align-center\" data-align=\"center\">5x<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\"><em>Table 2: Training throughput (on 128 A100 GPUs) of an MoE-based model versus a dense model, where both achieve the same model quality.<\/em><\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"pr-moe-and-mixture-of-students-reducing-the-model-size-and-improving-parameter-efficiency\">PR-MoE and Mixture-of-Students: Reducing the model size and improving parameter efficiency<\/h2>\n\n\n\n<p>While MoE-based models achieve the same quality with 5x training cost reduction in the NLG example, the resulting model has roughly 8x the parameters of the corresponding dense model. For example, a 6.7B dense model has 6.7 billion parameters and 1.3B+MoE-128 has 52 billion parameters. Training such a massive MoE model requires significantly more memory; inference latency and cost could also increase since the primary inference bottleneck is often the memory bandwidth needed to read model weights.<\/p>\n\n\n\n<p>To reduce model size and improve parameter efficiency, we\u2019ve made innovations in the MoE model architecture that reduce the overall model size by up to 3 times without affecting model quality. We also leverage knowledge distillation to learn a Mixture-of-Students (MoS) model, with a smaller model capacity as the teacher PR-MoE but preserve the teacher model accuracy.<\/p>\n\n\n\n\t<div class=\"border-bottom border-top border-gray-300 mt-5 mb-5 msr-promo text-center text-md-left alignwide\" data-bi-aN=\"promo\" data-bi-id=\"670821\">\n\t\t\n\n\t\t<p class=\"msr-promo__label text-gray-800 text-center text-uppercase\">\n\t\t<span class=\"px-4 bg-white display-inline-block font-weight-semibold small\">Spotlight: Microsoft research newsletter<\/span>\n\t<\/p>\n\t\n\t<div class=\"row pt-3 pb-4 align-items-center\">\n\t\t\t\t\t\t<div class=\"msr-promo__media col-12 col-md-5\">\n\t\t\t\t<a class=\"bg-gray-300\" href=\"https:\/\/info.microsoft.com\/ww-landing-microsoft-research-newsletter.html\" aria-label=\"Microsoft Research Newsletter\" data-bi-cN=\"Microsoft Research Newsletter\" target=\"_blank\">\n\t\t\t\t\t<img decoding=\"async\" class=\"w-100 display-block\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/09\/Newsletter_Banner_08_2019_v1_1920x1080.png\" alt=\"\" \/>\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t<div class=\"msr-promo__content p-3 px-5 col-12 col-md\">\n\n\t\t\t\t\t\t\t\t\t<h2 class=\"h4\">Microsoft Research Newsletter<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<p class=\"large\">Stay connected to the research community at Microsoft.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<div class=\"wp-block-buttons justify-content-center justify-content-md-start\">\n\t\t\t\t\t<div class=\"wp-block-button is-style-fill-chevron\">\n\t\t\t\t\t\t<a href=\"https:\/\/info.microsoft.com\/ww-landing-microsoft-research-newsletter.html\" class=\"btn btn-brand glyph-append glyph-append-chevron-right\" aria-label=\"Microsoft Research Newsletter\" data-bi-cN=\"Microsoft Research Newsletter\" target=\"_blank\">\n\t\t\t\t\t\t\tSubscribe today\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div><!--\/.msr-promo__content-->\n\t<\/div><!--\/.msr-promo__inner-wrap-->\n\t<\/div><!--\/.msr-promo-->\n\t\n\n\n<h3 class=\"wp-block-heading\" id=\"two-intuitions-for-improving-moe-architecture\">Two intuitions for improving MoE architecture<\/h3>\n\n\n\n<p><strong>Intuition-I: <\/strong>The standard MoE architecture has the same number and structure of experts in all MoE layers. This relates to a fundamental question in the deep learning community, which has been well-studied in computer vision: <strong>do all the layers in a deep neural network learn the same representation?<\/strong> Shallow layers learn general representations and deep layers learn more objective-specific representations. This also leads transfer learning in computer vision to freeze shallow layers for fine-tuning. This phenomenon, however, has not been well-explored in natural language processing (NLP), particularly for MoE.<\/p>\n\n\n\n<p>To investigate the question, we compare the performance of two different half-MoE architectures. More specifically, we put MoE layers in the first half of the model and leave the second half\u2019s layers identical to the dense model. We switch the MoE layers to the second half and use dense at the first half. The results show that deeper layers benefit more from large number of experts. This confirms that not all MoE layers learn the same level of representations.<\/p>\n\n\n\n<p><strong>Intuition-II:<\/strong> To improve the generalization performance of MoE models, there are two common methods: 1) increasing the number of experts while keeping the capacity (that is, for each token, the number of experts it goes through) to be the same; 2) doubling the capacity at the expense of slightly more computation (33%) while keeping the same number of experts. However, for method 1, the memory requirement for training resources needs to be increased due to larger number of experts. For method 2, higher capacity also doubles the communication volume which can significantly slow down training and inference.<strong> Is there a way to keep the training and inference efficiency while getting generalization performance gain?<\/strong><\/p>\n\n\n\n<p>One intuition of why larger capacity helps accuracy is that those extra experts can help correct the \u201crepresentation\u201d of the first expert. However, does this first expert need to be changed every time? Or can we fix the first and only assign different extra experts to different tokens?<\/p>\n\n\n\n<p>To investigate this, we perform a comparison in two ways: doubling the capacity and fixing one expert while varying the second expert across different experts. For the latter, a token will always pass a dense multilayer perceptron (MLP) module and an expert from MoE module. Therefore, we can achieve the benefit of using two experts per layer but still use one communication. We find out that the generalization performance of these two is on-par with each other. However, the training\/inference speed of our new design is faster.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"new-moe-architecture-pyramid-residual-moe\">New MoE Architecture: Pyramid-Residual MoE<\/h3>\n\n\n\n<p>We propose a novel MoE architecture, Pyramid-Residual MoE (PR-MoE). Figure 2 (right) shows its architecture. Following Intuition-I, PR-MoE utilizes more experts in the last few layers as compared to previous layers, which gives a reverse pyramid design. Following Intuition II, we propose a Residual-MoE structure, where each token separately passes one fixed MLP layer and one chosen expert. Combining them results in the PR-MoE model, where all standard MoE layers are replaced by the new PR-MoE layer.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"The left side shows the architecture of standard MoE model, where each expert layer have same amount number of experts. The right side shows the PR-MoE architecture. Two noticeable differences are (1) PR-MoE has more experts at the last two experts; (2) A token will always pass an MLP module plus a selected expert.\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure2_DeepSpeed-scaled.jpg\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"1523\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure2_DeepSpeed-scaled.jpg\" alt=\"The left side shows the architecture of standard MoE model, where each expert layer have same amount number of experts. The right side shows the PR-MoE architecture. Two noticeable differences are (1) PR-MoE has more experts at the last two experts; (2) A token will always pass an MLP module plus a selected expert.\" class=\"wp-image-811843\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure2_DeepSpeed-scaled.jpg 2560w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure2_DeepSpeed-300x178.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure2_DeepSpeed-1024x609.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure2_DeepSpeed-768x457.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure2_DeepSpeed-1536x914.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure2_DeepSpeed-2048x1218.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure2_DeepSpeed-240x143.jpg 240w\" sizes=\"(max-width: 2560px) 100vw, 2560px\" \/><\/a><figcaption class=\"wp-element-caption\">Figure 2: The illustration of standard MoE (left) and PR-MoE (right).<\/figcaption><\/figure>\n\n\n\n<p><strong><strong>Same quality as standard models with up to 3x model size reduction:<\/strong><\/strong> We evaluate PR-MoE on two model sizes, with bases of 350M and 1.3B parameters, and compare the performance with larger standard MoE architectures. The results are shown in Table 1 above. For both cases, PR-MoE uses much fewer experts but achieves comparable accuracy as standard MoE models. In the 350M model, PR-MoE only uses less than one third of the parameters that the standard MoE uses. In the 1.3B case, PR-MoE only uses about 60 percent of the parameters required for standard MoE.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"mixture-of-students-distillation-for-even-smaller-model-size-and-faster-inference\">Mixture-of-Students: Distillation for even smaller model size and faster inference<\/h3>\n\n\n\n<p>Model compression and distillation present additional opportunities to improve inference performance further. While there are many ways for model compression, such as quantization and pruning, we focus on reducing the number of layers of each expert in MoE and using knowledge distillation to compress the resulting student model to achieve a similar performance to the teacher MoE.<\/p>\n\n\n\n<p>Since MoE structure brings significant benefits by enabling sparse training and inference, our task-agnostic distilled MoE model, which we call <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/deepspeed-moe-advancing-mixture-of-experts-inference-and-training-to-power-next-generation-ai-scale\/\">Mixture of Students (MoS),<\/a> inherits these benefits while still providing the flexibility to compress into a dense model. We note that while existing work primarily considers small transformers (a few hundred parameters) and dense encoder-based LM models (like BERT), we focus on studying knowledge distillation for sparse MoE-based auto-generative language models on a multi-billion parameter scale. Furthermore, given the excellent performance of PR-MoE, we combine PR-MoE with MoS to further reduce the MoE model size.<\/p>\n\n\n\n\n\n<p>To apply knowledge distillation for MoE, we first train a teacher MoE model using the same training hyperparameters and datasets as in the previous section. The teacher model is 350M+PR-MoE-32\/64 and 1.3B+PR-MoE-64\/128, respectively. We reduce the depth of the teacher model to 21 (12.5%) to obtain a student model, and we force the student to imitate the outputs from the teacher MoE on the training dataset.<\/p>\n\n\n\n<p>In particular, we take the knowledge distillation loss as a weighted sum of the cross-entropy loss between predictions and the given hard label and the Kullback\u2013Leibler (KL) divergence loss between the predictions and the teacher\u2019s soft label. In practice, we observe that distillation may adversely affect MoS accuracy. In particular, while knowledge distillation loss improves validation accuracy initially, it begins to hurt accuracy towards the end of training.<\/p>\n\n\n\n<p>We hypothesize that because the PR-MoE already reduces the capacity compared with the standard MoE by exploiting the architecture change (for example, reducing experts in lower layers), further reducing the depth of the model causes the student to have insufficient capacity, making it fall into the underfitting regime. Therefore, we take a staged distillation approach, where we decay the impact from knowledge distillation gradually in the training process.<\/p>\n\n\n\n\n\n<p>Our study shows that it is possible to reach similar performance\u2014such as in zero-shot evaluation on many downstream tasks\u2014for a smaller MoE model pretrained with knowledge distillation. The MoS achieve comparable accuracy to the teacher MoE model, retaining 99.3% and 99.1% of the performance despite having 12.5% fewer layers. This enables an additional 12.5% model size reduction. When combined with PR-MoE, it leads to up to 3.7x model size reduction.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"deepspeed-moe-inference-serving-moe-models-at-unprecedented-scale-and-speed\">DeepSpeed-MoE inference: Serving MoE models at unprecedented scale and speed<\/h2>\n\n\n\n<p>Optimizing for MoE inference latency and cost is crucial for MoE models to be useful in practice. During inference the batch size is generally small, so the inference latency of an MoE model depends primarily on time it takes to load the model parameters from main memory, contrasting with the conventional belief that lesser compute should lead to faster inference. So, inference performance mainly depends on two factors: the overall model size and the overall achievable memory bandwidth.<\/p>\n\n\n\n<p>In the previous section, we presented PR-MoE and distillation to optimize the model size. This section presents our solution to maximize the achievable memory bandwidth by creating a multi-GPU MoE inferencing system that can leverage the aggregated memory bandwidth across dozens of distributed GPUs to speed up inference. Together, DeepSpeed offers an unprecedented scale and efficiency to serve massive MoE models with<strong> 7.3x better <em>latency and cost compared to baseline MoE systems, and up to 4.5x faster and 9x cheaper MoE inference compared to quality-equivalent dense models.<\/em><\/strong><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"moe-inference-performance-is-an-interesting-paradox\">MoE inference performance is an interesting paradox<\/h3>\n\n\n\n<p>From the best-case view, each token of an MoE model only activates a single expert at each MoE layer, resulting in a critical data path that is equivalent to the base model size, orders-of-magnitude smaller than the actual model size. For example, when inferencing with a 1.3B+MoE-128 model, each input token needs just 1.3 billion parameters, even though the overall model size is 52 billion parameters.<\/p>\n\n\n\n<p>From the worst-case view, the aggregate parameters needed to process a group of tokens can be as large as the full model size, in the example, the entire 52 billion parameters, making it challenging to achieve short latency and high throughput.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"design-goals-for-the-ds-moe-inference-system\">Design goals for the DS-MoE inference system<\/h3>\n\n\n\n<p>The design goal of our optimizations is to steer the performance toward the best-case view. This requires careful orchestration and partitioning of the model to group and route all tokens with the same critical data path together to reduce data access per device and achieve maximum aggregate bandwidth. An overview of how DS-MoE tackles this design goal by embracing multi-dimensional parallelism inherent in MoE models is illustrated in Figure 3.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Figure 3: Illustration of the DS-MoE design that embraces the complexity of multi-dimensional parallelism for different partitions (expert and non-expert) of the model.\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/expert-parallelism-scaled.jpg\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"570\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/expert-parallelism-1024x570.jpg\" alt=\"Figure 3: Illustration of the DS-MoE design that embraces the complexity of multi-dimensional parallelism for different partitions (expert and non-expert) of the model.\" class=\"wp-image-813544\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/expert-parallelism-1024x570.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/expert-parallelism-300x167.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/expert-parallelism-768x427.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/expert-parallelism-1536x855.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/expert-parallelism-2048x1139.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/expert-parallelism-240x134.jpg 240w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><figcaption class=\"wp-element-caption\">Figure 3: DS-MoE design that embraces the complexity of multi-dimensional parallelism for different partitions (expert and non-expert) of the model.<\/figcaption><\/figure>\n\n\n\n<p><strong>DS-MoE inference system is centered around three well-coordinated optimizations:<\/strong><\/p>\n\n\n\n\n\n<p>The DS-MoE Inference system is designed to minimize the critical data path per device and maximize the achievable aggregate memory bandwidth across devices, which is achieved by: 1) expert parallelism and expert-slicing on expert parameters and 2) data parallelism and tensor-slicing for non-expert parameters.<\/p>\n\n\n\n<p><strong>Expert parallelism and expert-slicing for expert parameters:<\/strong> We partition experts across devices, group all tokens of using the same experts under the same critical data path, and parallelize processing of the token groups with different critical paths among different devices using expert parallelism.<\/p>\n\n\n\n<p>In the example of 1.3B+MoE-128, when expert parallelism is equal to 128, each GPU only processes a single token group corresponding to the experts on that device. This results in a sequential path that is 1.3 billion parameters per device, 5x smaller than its quality-equivalent dense model with 6.7B parameters. Therefore, in theory, an MoE-based model has the potential to run up to 5x faster than its quality-equivalent dense model using expert parallelism assuming no communication overhead, a topic we discuss in the next section.<\/p>\n\n\n\n<p>In addition, we propose \u201cexpert-slicing\u201d to leverage the concept of tensor-slicing for the parameters within an expert. This additional dimension of parallelism is helpful for latency stringent scenarios that we scale to more devices than the number of experts.<\/p>\n\n\n\n<p><strong>Data parallelism and Tensor-slicing for non-expert parameters: <\/strong>Within a node, we use tensor-slicing to partition the non-expert parameters, leveraging aggregate GPU memory bandwidth of all GPUs to accelerate the processing. While it is possible to perform tensor-slicing across nodes, the communication overhead of tensor-slicing along with reduced compute granularity generally makes inter-node tensor-slicing inefficient. To scale non-expert parameters across multiple nodes, we use data parallelism by creating non-expert parameter replicas processing different batches across nodes that incurs no communication overhead or reduction in compute granularity.<\/p>\n\n\n\n<p>Figure 3 above shows an example scenario for distributed MoE inference highlighting different parts of the MoE model, how the model and data are partitioned, and what form of parallelism is used to deal with each piece.<\/p>\n\n\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--left\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">GITHUB<\/span>\n\t\t\t<a href=\"https:\/\/github.com\/microsoft\/msccl\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Synthesized Collective Communication Library\" data-bi-aN=\"margin-callout\" data-bi-cN=\"Synthesized Collective Communication Library\">\n\t\t\t\tSynthesized Collective Communication Library&nbsp;<span class=\"glyph-append glyph-append-share glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t\t\t<p class=\"annotations__caption text-neutral-400 mt-2\">Microsoft Collective Communication Library (MSCCL) is a platform to execute custom collective communication algorithms for multiple accelerators supported by Microsoft Azure.<\/p>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>Expert parallelism requires all-to-all communication between all expert parallel devices. By default, DS-MoE uses NCCL for this communication via torch. distributed interface, but we observe major overhead when it is used at scale. To optimize this, we develop a custom communication interface to use Microsoft <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/msccl\">SCCL<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and achieve better performance than NCCL. Despite the plug-in optimizations, it is difficult to scale expert parallelism to many devices as the latency increases linearly with the increase in devices. To address this critical scaling challenge, we design two new communication optimization strategies that exploit the underlying point-to-point NCCL operations and custom CUDA kernels to perform necessary data-layout transformations.<\/p>\n\n\n\n<p><strong>Hierarchical All-to-All<\/strong>: We implement a hierarchical all-to-all as a two-step process with a data-layout transformation, followed by an intra-node all-to-all, followed by a second data-layout transformation and a final inter-node all-to-all. This reduces the communication hops from O (p) to O (G+p\/G), where G is the number of GPUs in a node and p is the total number of GPU devices. Figure 4 shows the design overview of this implementation. Despite the 2x increase in communication volume, this hierarchical implementation allows for better scaling for small batch sizes as communication at this message size is more latency-bound than bandwidth-bound.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Illustration of the proposed hierarchical all-to-all design\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure4_DeepSpeed.jpg\"><img loading=\"lazy\" decoding=\"async\" width=\"2255\" height=\"1208\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure4_DeepSpeed.jpg\" alt=\"Illustration of the proposed hierarchical all-to-all design\" class=\"wp-image-811882\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure4_DeepSpeed.jpg 2255w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure4_DeepSpeed-300x161.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure4_DeepSpeed-1024x549.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure4_DeepSpeed-768x411.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure4_DeepSpeed-1536x823.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure4_DeepSpeed-2048x1097.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure4_DeepSpeed-710x380.jpg 710w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure4_DeepSpeed-240x129.jpg 240w\" sizes=\"(max-width: 2255px) 100vw, 2255px\" \/><\/a><figcaption class=\"wp-element-caption\">Figure 4: Illustration of the proposed hierarchical all-to-all design<\/figcaption><\/figure>\n\n\n\n<p><strong>Parallelism Coordinated Communication Optimization:<\/strong> Combining expert parallelism and tensor-slicing with data parallelism within a single model is non-trivial. Tensor-slicing splits individual operators across GPUs and requires all-reduce between them, while expert parallelism places expert operators across GPUs without splitting them and requires all-to-all between them. By design, a na\u00efve approach to handle these communication steps will be inefficient.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Illustration of the parallelism coordinated all-to-all optimization\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure5_DeepSpeed.png\"><img loading=\"lazy\" decoding=\"async\" width=\"4119\" height=\"2376\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure5_DeepSpeed.png\" alt=\"Illustration of the parallelism coordinated all-to-all optimization\" class=\"wp-image-813517\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure5_DeepSpeed.png 4119w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure5_DeepSpeed-300x173.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure5_DeepSpeed-1024x591.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure5_DeepSpeed-768x443.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure5_DeepSpeed-1536x886.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure5_DeepSpeed-2048x1181.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure5_DeepSpeed-240x138.png 240w\" sizes=\"(max-width: 4119px) 100vw, 4119px\" \/><\/a><figcaption class=\"wp-element-caption\">Figure 5: Illustration of the parallelism coordinated communication<\/figcaption><\/figure>\n\n\n\n<p>To this end, we propose a novel design, as shown in Figure 5, that performs all-to-all only on a subset of devices that share the same tensor-slicing rank instead of all expert-parallel processes. As a result, the latency of all-to-all can be reduced to O(p\/L) instead of O(p) where L is the tensor-slicing parallelism degree. This reduced latency enables us to scale inference to hundreds of GPU devices.<\/p>\n\n\n\n\n\n<p>DS-MoE inference system consists of highly optimized kernels targeting both transformer and MoE-related operations. These kernels aim for maximizing the bandwidth utilization by fusing the operations that work in producer-consumer fashion. In addition to computation required for the transformer layers (explained in this <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression\/\">blog post<\/a>), MoE models require the following additional operations:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>a gating function that determines the assignment of tokens to experts, where the result is represented as a sparse tensor.<\/li>\n\n\n\n<li>a sparse einsum operator, between the one-hot tensor and all the tokens, which sorts the ordering of the tokens based on the assigned expert ID.<\/li>\n\n\n\n<li>a final einsum that scales and re-sorts the tokens back to their original ordering.<\/li>\n<\/ol>\n\n\n\n<p>The gating function includes numerous operations to create token-masks, select top-k experts, and perform cumulative-sum and sparse matrix-multiply, all of which are not only wasteful due to the sparse tenor representation, but also extremely slow due to many kernel call invocations. Moreover, the sparse einsums have a complexity of SxExMxc (number of tokens S, number of experts E, model dimension M, and expert capacity c that is typically 1), but E-1 out of E operators for each token are multiplication and addition with zeros.<\/p>\n\n\n\n<p>We optimize these operators using dense representation and kernel-fusion. First, we fuse the gating function into a single kernel, and use a dense token-to-expert mapping table to represent the assignment from tokens to experts, greatly reducing the kernel launch overhead, as well as memory and compute overhead from the sparse representation.<\/p>\n\n\n\n<p>Second, to optimize the remaining two sparse einsums, we implement them as data layout transformations using the above-mentioned mapping table, to first sort them based on the expert id and then back to its original ordering without requiring any sparse einsum, reducing the complexity of these operations from SxExMxc to SxMxc. Combined, these optimizations result in over 6x reduction in MoE kernel related latency.<\/p>\n\n\n\n\n\n<h3 class=\"wp-block-heading\" id=\"low-latency-and-high-throughput-at-unprecedented-scale\">Low latency and high throughput at unprecedented scale<\/h3>\n\n\n\n<p>In modern production environments, powerful DL models are often served using hundreds of GPU devices to meet the traffic demand and deliver low latency. Here we demonstrate the performance of DS-MoE Inference System on a 256 A100 with 40 GB GPUs. Table 3 shows various model configurations used for performance comparisons in this section.<\/p>\n\n\n\n<figure class=\"wp-block-table aligncenter is-style-regular\"><table><thead><tr><th class=\"has-text-align-center\" data-align=\"center\"><strong>Model<\/strong><\/th><th class=\"has-text-align-center\" data-align=\"center\"><strong>Size (billions)<\/strong><\/th><th class=\"has-text-align-center\" data-align=\"center\"><strong># of Layers<\/strong><\/th><th class=\"has-text-align-center\" data-align=\"center\"><strong>Hidden size<\/strong><\/th><th class=\"has-text-align-center\" data-align=\"center\"><strong>Model-Parallel degree<\/strong><\/th><th class=\"has-text-align-center\" data-align=\"center\"><strong>Expert-Parallel degree<\/strong><\/th><\/tr><\/thead><tbody><tr><td class=\"has-text-align-center\" data-align=\"center\"><strong>2.4B+MoE-128<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\">107.7<\/td><td class=\"has-text-align-center\" data-align=\"center\">16<\/td><td class=\"has-text-align-center\" data-align=\"center\">3,584<\/td><td class=\"has-text-align-center\" data-align=\"center\">1<\/td><td class=\"has-text-align-center\" data-align=\"center\">128<\/td><\/tr><tr><td class=\"has-text-align-center\" data-align=\"center\"><strong>8B+MoE-128<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\">349.0<\/td><td class=\"has-text-align-center\" data-align=\"center\">40<\/td><td class=\"has-text-align-center\" data-align=\"center\">4,096<\/td><td class=\"has-text-align-center\" data-align=\"center\">4<\/td><td class=\"has-text-align-center\" data-align=\"center\">128<\/td><\/tr><tr><td class=\"has-text-align-center\" data-align=\"center\"><strong>24B+MoE-128<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\">1,046.9<\/td><td class=\"has-text-align-center\" data-align=\"center\">30<\/td><td class=\"has-text-align-center\" data-align=\"center\">8,192<\/td><td class=\"has-text-align-center\" data-align=\"center\">8<\/td><td class=\"has-text-align-center\" data-align=\"center\">128<\/td><\/tr><tr><td class=\"has-text-align-center\" data-align=\"center\"><strong>47B+MoE-128<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\">2,024.0<\/td><td class=\"has-text-align-center\" data-align=\"center\">58<\/td><td class=\"has-text-align-center\" data-align=\"center\">8,192<\/td><td class=\"has-text-align-center\" data-align=\"center\">8<\/td><td class=\"has-text-align-center\" data-align=\"center\">128<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">Table 3. The configuration of different MoE models used for the performance evaluation of Figure 6.<\/figcaption><\/figure>\n\n\n\n<p>We scale MoE models from 107 billion parameters to 2 trillion parameters. To offer a strong baseline for comparison, we utilize a full-featured distributed <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/deepspeed-powers-8x-larger-moe-model-training-with-high-performance\/\">PyTorch implementation<\/a> that is <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/deepspeed-powers-8x-larger-moe-model-training-with-high-performance\/\">capable of both tensor-slicing and expert-parallelism<\/a>. Figure 6 shows the results for all these model configurations:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>DeepSpeed MoE achieves up to 7.3x reduction in latency while achieving up to 7.3x higher throughput compared to the baseline.<\/li>\n\n\n\n<li>By effectively exploiting hundreds of GPUs in parallel, DeepSpeed MoE achieves an unprecedented scale for inference at incredibly low latencies &#8211; a staggering trillion parameter MoE model can be inferenced under 25ms.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2560\" height=\"748\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure6_DeepSpeed-scaled.jpg\" alt=\"Figure 6: Latency and throughput Improvement offered by DeepSpeed-Inference-MoE (Optimized) over PyTorch (Baseline) for different model sizes (107 billion to 2 trillion parameters). We use 128 GPUs for all configurations for baseline, and 128\/256 GPUs for DeepSpeed (256 GPUs for the trillion-scale models). The throughputs show that DeepSpeed MoE achieves up to 7.3x reduction in latency while achieving up to 7.3x higher throughput compared to the baseline\" class=\"wp-image-811876\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure6_DeepSpeed-scaled.jpg 2560w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure6_DeepSpeed-300x88.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure6_DeepSpeed-1024x299.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure6_DeepSpeed-768x224.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure6_DeepSpeed-1536x449.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure6_DeepSpeed-2048x599.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure6_DeepSpeed-240x70.jpg 240w\" sizes=\"(max-width: 2560px) 100vw, 2560px\" \/><figcaption class=\"wp-element-caption\">Figure 6: Latency and throughput Improvement offered by DeepSpeed-Inference-MoE (Optimized) over PyTorch (Baseline) for different model sizes (107 billion to 2 trillion parameters). We use 128 GPUs for all configurations for baseline, and 128\/256 GPUs for DeepSpeed (256 GPUs for the trillion-scale models). The throughputs shown here are per GPU and should be multiplied by number of GPUs to get the aggregate throughput of the cluster.<\/figcaption><\/figure>\n\n\n\n<p>By combining the system optimizations offered by the DS-MoE inference system and model innovations of PR-MoE and MoS, DeepSpeed MoE delivers two more benefits:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Reduce the minimum number of GPUs required to perform inference on these models. Figure 7 shows a comparison of three model variants along with the baseline: 1) standard MoE Model (8b-MoE-128), 2) PR-MoE model, and 3) PR-MoE+MoS model. The PR-MoE+MoS model performs the best as expected. The key observation is that the PR-MoE and MoS optimizations allow us to use 16 GPUs instead of 32 GPUs to perform this inference.<\/li>\n\n\n\n<li>Further improve both latency and throughput of various MoE model sizes (as shown in Figure 8).<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Figure 7: Graph showing 2x fewer resources needed for MoE inference when using PR-MoE+MoS. PR-MoE and MoS optimizations also allow us to use 16 GPUs instead of 32 GPUs to perform this inference.\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure7_DeepSpeed.png\"><img loading=\"lazy\" decoding=\"async\" width=\"2019\" height=\"983\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure7_DeepSpeed.png\" alt=\"Figure 7: Graph showing 2x fewer resources needed for MoE inference when using PR-MoE+MoS. PR-MoE and MoS optimizations also allow us to use 16 GPUs instead of 32 GPUs to perform this inference.\" class=\"wp-image-813499\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure7_DeepSpeed.png 2019w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure7_DeepSpeed-300x146.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure7_DeepSpeed-1024x499.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure7_DeepSpeed-768x374.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure7_DeepSpeed-1536x748.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure7_DeepSpeed-240x117.png 240w\" sizes=\"(max-width: 2019px) 100vw, 2019px\" \/><\/a><figcaption class=\"wp-element-caption\">Figure 7: 2x fewer resources needed for MoE inference when using PR-MoE+MoS.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Figure 8: Graph showing inference latency comparing standard-MoE with PR-MoE and PR-MoE + MoS compression on different GPU count and model sizes.  This shows PR-MoE + MoS achieves up to a 10x latency improvement compared to the baseline.\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure8_DeepSpeed.jpg\"><img loading=\"lazy\" decoding=\"async\" width=\"2337\" height=\"1009\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure8_DeepSpeed.jpg\" alt=\"Figure 8: Graph showing inference latency comparing standard-MoE with PR-MoE and PR-MoE + MoS compression on different GPU count and model sizes.  This shows PR-MoE + MoS achieves up to a 10x latency improvement compared to the baseline.\" class=\"wp-image-811870\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure8_DeepSpeed.jpg 2337w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure8_DeepSpeed-300x130.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure8_DeepSpeed-1024x442.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure8_DeepSpeed-768x332.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure8_DeepSpeed-1536x663.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure8_DeepSpeed-2048x884.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure8_DeepSpeed-240x104.jpg 240w\" sizes=\"(max-width: 2337px) 100vw, 2337px\" \/><\/a><figcaption class=\"wp-element-caption\">Figure 8: Inference latency comparing standard-MoE with PR-MoE and PR-MoE + MoS compression on different GPU count and model sizes<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"better-inference-latency-and-throughput-than-quality-equivalent-dense-models\">Better inference latency and throughput than quality-equivalent dense models<\/h3>\n\n\n\n<p>To better understand the inference performance of MoE models compared to quality-equivalent dense models, it is important to note that although MoE models are 5x faster and cheaper to train, that may not be true for inference. Inference performance has different bottlenecks and its primary factor is the amount of data read from memory instead of computation.<\/p>\n\n\n\n<p>We show inference latency and throughput for two MoE models compared to their quality-equivalent dense models: a) 52 billion-parameter MoE (1.3B-MoE-128) model compared to a 6.7 billion-parameter dense model and b) 1.5 trillion-parameter MoE model compared to a 175 billion-parameter dense model in Figures 9 and 10, respectively.<\/p>\n\n\n\n<p>When using PyTorch, MoE model inference is more expensive and slower compared to its quality-equivalent dense models. This is true for both model sizes. However, the optimizations in DS-MoE reverse this trend and make MoE model inference both faster and cheaper compared to quality-equivalent dense models. This is a critical result, showing MoE\u2019s benefits over dense beyond training but also on inference latency and cost, which is important to real-world deployments.<\/p>\n\n\n\n<p>When comparing the results of Figure 9 with Figure 10, we observe that the benefits of MoE models over dense models become even larger with the increase of model size. While the 52 billion-parameter MoE model is 2.4x faster and cheaper than the 6.7 billion-parameter dense model, the 1.5 trillion-parameter MoE model is 4.5x faster and 9x cheaper than the 175 billion-parameter dense model. The benefits increase for larger models because DS-MoE leverages parallelism-coordinated optimization to reduce communication overhead when using tensor-slicing on non-expert part of model. Furthermore, we can take advantage of expert-slicing at this scale, which enables us to scale to a higher number of GPUs compared to the PyTorch baseline. In addition, for the larger 1.5 trillion-parameter MoE model, we observed 2x additional improvement in throughput over latency as shown in Figure 10. This is because MoE models can run with half the tensor-slicing degree of the dense model (8-way vs. 16-way) and thus two times higher batch size.<\/p>\n\n\n\n<p>Overall, DeepSpeed MoE delivers up to 4.5x faster and up to 9x cheaper MoE model inference compared to serving quality-equivalent dense models using PyTorch. With benefits that scale with model size and hardware resources, as shown from these results, it makes us believe that MoE models will be crucial to bring the next generation of advances in AI scale.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Figure 9: Inference latency comparison of a 52 billion-parameter MoE model and its quality-equivalent 6.7 billion-parameter dense model. It shows the 52 billion-parameter MoE model is 2.4x faster and cheaper than the 6.7 billion-parameter dense model\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure9_DeepSpeed.jpg\"><img loading=\"lazy\" decoding=\"async\" width=\"2135\" height=\"1137\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure9_DeepSpeed.jpg\" alt=\"Figure 9: Inference latency comparison of a 52 billion-parameter MoE model and its quality-equivalent 6.7 billion-parameter dense model. It shows the 52 billion-parameter MoE model is 2.4x faster and cheaper than the 6.7 billion-parameter dense model\" class=\"wp-image-811867\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure9_DeepSpeed.jpg 2135w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure9_DeepSpeed-300x160.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure9_DeepSpeed-1024x545.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure9_DeepSpeed-768x409.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure9_DeepSpeed-1536x818.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure9_DeepSpeed-2048x1091.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure9_DeepSpeed-240x128.jpg 240w\" sizes=\"(max-width: 2135px) 100vw, 2135px\" \/><\/a><figcaption class=\"wp-element-caption\">Figure 9: Inference latency comparison of a 52 billion-parameter MoE model and its quality-equivalent 6.7 billion-parameter dense model. We use 1 GPU for 6.7 billion-parameter model as it offers the lowest latency. We use 128 GPUs for the 52 billion-parameter model. The quality-equivalence has been verified by experiments presented in the training section.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><a data-bi-bhvr=\"14\"  data-bi-cn=\"Figure 10: Measured inference latency comparison of a 1.5 trillion-parameter MoE model and its quality-equivalent 175 billion dense model. It shows the 1.5 trillion-parameter MoE model is 4.5x faster and 9x cheaper than the 175 billion-parameter dense model.\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure10_DeepSpeed.jpg\"><img loading=\"lazy\" decoding=\"async\" width=\"1818\" height=\"1184\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure10_DeepSpeed.jpg\" alt=\"Figure 10: Measured inference latency comparison of a 1.5 trillion-parameter MoE model and its quality-equivalent 175 billion dense model. It shows the 1.5 trillion-parameter MoE model is 4.5x faster and 9x cheaper than the 175 billion-parameter dense model.\" class=\"wp-image-811864\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure10_DeepSpeed.jpg 1818w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure10_DeepSpeed-300x195.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure10_DeepSpeed-1024x667.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure10_DeepSpeed-768x500.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure10_DeepSpeed-1536x1000.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/Figure10_DeepSpeed-240x156.jpg 240w\" sizes=\"(max-width: 1818px) 100vw, 1818px\" \/><\/a><figcaption class=\"wp-element-caption\">Figure 10: Measured inference latency comparison of a 1.5 trillion-parameter MoE model and its quality-equivalent 175 billion dense model. We assume the quality equivalence of these two models with the hypothesis that the scaling law of the smaller scale experiments of Figure 9 holds, as well as from the observations of the published literature.<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"looking-forward-to-the-next-generation-of-ai-scale\">Looking forward to the next generation of AI Scale<\/h2>\n\n\n\n<p>With the exponential growth of model size recently, we have arrived at the boundary of what modern supercomputing clusters can do to train and serve large models. It is no longer feasible to achieve better model quality by simply increasing the model size due to insurmountable requirements on hardware resources. The choices we have are to wait for the next generation of hardware or to innovate and improve the training and inference efficiency using current hardware.<\/p>\n\n\n\n<p>We, along with recent literature, have demonstrated how MoE-based models can reduce the training cost of even the largest NLG models by several times compared to their quality-equivalent dense counterparts, offering the possibility to train the next scale of AI models on current generation of hardware. However, prior to this blog post, to our knowledge there have been no existing works on how to serve the MoE models (with many more parameters) with latency and cost better than the dense models. This is a challenging issue that blocks practical use.<\/p>\n\n\n\n<p>To enable practical and efficient inference for MoE models, we offer novel PR-MoE model architecture and MoS distillation technique to significantly reduce the memory requirements of these models. We also offer an MoE inference framework to achieve incredibly low latency and cost at an unprecedented model scale. Combining these innovations, we are able to make these MoE models not just feasible to serve but able to be used for inference at lower latency and cost than their quality-equivalent dense counterparts.<\/p>\n\n\n\n<p>As a whole, the new innovations and infrastructures offer a promising path towards training and inference of the next generation of AI scale, without requiring an increase in compute resources. A shift from dense to sparse MoE models can open a path to new directions in the large model landscape, where deploying higher-quality models is widely possible with fewer resources and is more sustainable by reducing the environmental impact of large-scale AI.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--right\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">GITHUB<\/span>\n\t\t\t<a href=\"https:\/\/github.com\/Azure\/azureml-examples\/tree\/main\/python-sdk\/workflows\/train\/deepspeed\/transformers\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Deepspeed Transformers Example\" data-bi-aN=\"margin-callout\" data-bi-cN=\"Deepspeed Transformers Example\">\n\t\t\t\tDeepspeed Transformers Example&nbsp;<span class=\"glyph-append glyph-append-share glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p><strong>Software: <\/strong>The best place to train and serve models using DeepSpeed is the Microsoft <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/azure.microsoft.com\/en-us\/overview\/ai-platform\/\">Azure AI<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> platform. To get started with DeepSpeed on Azure, follow the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.deepspeed.ai\/tutorials\/azure\/\">tutorial<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and experiment with different models using our <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/Azure\/azureml-examples\/tree\/main\/python-sdk\/workflows\/train\/deepspeed\/transformers\">Azure ML examples<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. You can also measure your model&#8217;s energy consumption using the latest <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/techcommunity.microsoft.com\/t5\/green-tech-blog\/charting-the-path-towards-sustainable-ai-with-azure-machine\/ba-p\/2866923\" target=\"_blank\" rel=\"noreferrer noopener\">Azure Machine Learning resource metrics.<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/p>\n\n\n\n<p>With this release of DeepSpeed, we are releasing a generic end-to-end framework for training and inference of MoE-based models. The MoE training support and optimizations are made available in full. The MoE inference optimizations will be released in two phases. The generic flexible parallelism framework for MoE inference is being released today. Optimizations related to computation kernels and communication will be released in future.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--left\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">GITHUB<\/span>\n\t\t\t<a href=\"https:\/\/github.com\/microsoft\/DeepSpeed\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"DeepSpeed\" data-bi-aN=\"margin-callout\" data-bi-cN=\"DeepSpeed\">\n\t\t\t\tDeepSpeed&nbsp;<span class=\"glyph-append glyph-append-share glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t\t\t<p class=\"annotations__caption text-neutral-400 mt-2\">DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.<\/p>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p>To enable experimentation with DeepSpeed MoE optimizations, we are also releasing two extensions of the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.deepspeed.ai\/news\/2021\/12\/09\/deepspeed-moe-nlg.html\">NLG example<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> that enables 5x reduction in training cost for MT-NLG like models: 1) PR-MoE model extension to enable 3x improvement in parameter efficiency and model size reduction and 2) Model code extensions so users can easily experiment with MoE inference at scale. Please find the code, tutorials, and documents at DeepSpeed <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/DeepSpeed\">GitHub<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.deepspeed.ai\/\">website<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"about-our-great-collaborators\">About our great collaborators<\/h3>\n\n\n\n<p>This work was done in collaboration with Brandon Norick, Zhun Liu, and Xia Song from the Turing Team, Young Jin Kim, Alex Muzio, and Hany Hassan Awadalla from the Z-Code Team, and both Saeed Maleki and Madan Musuvathi from the SCCL team.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"about-the-deepspeed-team\">About the DeepSpeed Team<\/h3>\n\n\n\n<p>We are a group of system researchers and engineers\u2014Samyam Rajbhandari, Ammar Ahmad Awan, Jeff Rasley, Reza Yazdani Aminabadi, Minjia Zhang, Zhewei Yao, Conglong Li, Olatunji Ruwase, Elton Zheng, Shaden Smith, Cheng Li, Du Li, Yang Li, Xiaoxia Wu, Jeffery Zhu (PM), Yuxiong He (team lead)\u2014who are enthusiastic about performance optimization of large-scale systems. We have recently focused on deep learning systems, optimizing deep learning\u2019s speed to train, speed to convergence, and speed to develop! If this type of work interests you, the DeepSpeed team is hiring both researchers and engineers! Please visit our <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/careers.microsoft.com\/us\/en\/search-results?keywords=http%3A%2F%2Fdeepspeed.ai\">careers page.<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In the last three years, the largest trained dense models have increased in size by over 1,000 times, from a few hundred million parameters to over 500 billion parameters in Megatron-Turing NLG 530B (MT-NLG). Improvements in model quality with size suggest that this trend will continue, with larger model sizes bringing better model quality. However, [&hellip;]<\/p>\n","protected":false},"author":39507,"featured_media":813712,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243984],"msr-impact-theme":[],"msr-promo-type":[243993],"msr-podcast-series":[],"class_list":["post-811273","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-blog-homepage-featured","msr-promo-type-webinar"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[678390,649749],"related-events":[],"related-researchers":[{"type":"guest","value":"deepspeed-team","user_id":"690909","display_name":"DeepSpeed Team","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/deepspeed\/#!people\" aria-label=\"Visit the profile page for DeepSpeed Team\">DeepSpeed Team<\/a>","is_active":true,"last_first":"Team, DeepSpeed","people_section":0,"alias":"deepspeed-team"},{"type":"guest","value":"andrey-proskurin","user_id":"740575","display_name":"Andrey  Proskurin","author_link":"<a href=\"https:\/\/ca.linkedin.com\/in\/andreyproskurin\" aria-label=\"Visit the profile page for Andrey  Proskurin\">Andrey  Proskurin<\/a>","is_active":true,"last_first":"Proskurin, Andrey ","people_section":0,"alias":"andrey-proskurin"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-scaled-960x540.jpg\" class=\"img-object-cover\" alt=\"DeepSpeed shares findings and innovations for MoE models and systems that 1) reduce training cost by 5x, 2) reduce MoE parameter size by up to 3.7x and 3) reduce MoE inference latency by 7.3x at an unprecedented scale and offer up to 4.5x faster and 9x cheaper inference for MoE models compared to quality-equivalent dense models.\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-scaled-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-1536x865.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-2048x1153.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-scaled-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2022\/01\/1400x788_Deepspeed_MoE_no_logo_still-1-1920x1080.jpg 1920w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/deepspeed\/#!people\" title=\"Go to researcher profile for DeepSpeed Team\" aria-label=\"Go to researcher profile for DeepSpeed Team\" data-bi-type=\"byline author\" data-bi-cN=\"DeepSpeed Team\">DeepSpeed Team<\/a> and <a href=\"https:\/\/ca.linkedin.com\/in\/andreyproskurin\" title=\"Go to researcher profile for Andrey  Proskurin\" aria-label=\"Go to researcher profile for Andrey  Proskurin\" data-bi-type=\"byline author\" data-bi-cN=\"Andrey  Proskurin\">Andrey  Proskurin<\/a>","formattedDate":"January 19, 2022","formattedExcerpt":"In the last three years, the largest trained dense models have increased in size by over 1,000 times, from a few hundred million parameters to over 500 billion parameters in Megatron-Turing NLG 530B (MT-NLG). Improvements in model quality with size suggest that this trend will&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/811273"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/39507"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=811273"}],"version-history":[{"count":72,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/811273\/revisions"}],"predecessor-version":[{"id":951540,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/811273\/revisions\/951540"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/813712"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=811273"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=811273"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=811273"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=811273"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=811273"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=811273"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=811273"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=811273"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=811273"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=811273"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=811273"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}