{"id":1123686,"date":"2025-02-05T09:32:32","date_gmt":"2025-02-05T17:32:32","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1123686"},"modified":"2025-02-05T09:32:36","modified_gmt":"2025-02-05T17:32:36","slug":"advances-to-low-bit-quantization-enable-llms-on-edge-devices","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/advances-to-low-bit-quantization-enable-llms-on-edge-devices\/","title":{"rendered":"Advances to low-bit quantization enable LLMs on edge devices"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1400\" height=\"788\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1.jpg\" alt=\"Three white icons that represent artificial intelligence, systems, and networking. These icons sit on a purple to pink gradient background.\" class=\"wp-image-1124472\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1.jpg 1400w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-1280x720.jpg 1280w\" sizes=\"auto, (max-width: 1400px) 100vw, 1400px\" \/><\/figure>\n\n\n\n<p>Large language models (LLMs) are increasingly being deployed on edge devices\u2014hardware that processes data locally near the data source, such as smartphones, laptops, and robots. Running LLMs on these devices supports advanced AI and real-time services, but their massive size, with hundreds of millions of parameters, requires significant memory and computational power, limiting widespread adoption. Low-bit quantization, a technique that compresses models and reduces memory demands, offers a solution by enabling more efficient operation.<\/p>\n\n\n\n<p>Recent advances in low-bit quantization have made mixed-precision matrix multiplication (mpGEMM) viable for LLMs. This deep learning technique allows data of the same or different formats to be multiplied, such as int8*int1, int8*int2, or FP16*int4. By combining a variety of precision levels, mpGEMM strikes a balance among speed, memory efficiency, and computational accuracy.&nbsp;<\/p>\n\n\n\n<p>However, most hardware supports only symmetric computations\u2014operations on data of similar formats\u2014creating challenges for mixed-precision calculations during General Matrix Multiplication (GEMM), a critical operation for LLMs. Overcoming these hardware limitations is essential to fully benefit from mpGEMM and support asymmetrical computations.&nbsp;<\/p>\n\n\n\n<p>To unlock the potential of low-bit quantization on resource-constrained edge devices, hardware must natively support mpGEMM. To address this, we developed the following three approaches for computing kernels and hardware architectures:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Ladder<\/strong> <strong>data type compiler:<\/strong> Supports various low-precision data types by converting unsupported types into hardware-compatible ones without data loss, while also generating high-performance conversion code.&nbsp;<\/li>\n\n\n\n<li><strong>T-MAC mpGEMM library:<\/strong> Implements GEMM using a lookup table (LUT) approach, eliminating multiplications to significantly reduce computational overhead. Optimized for diverse CPUs, T-MAC delivers several times the speed of other libraries.&nbsp;<\/li>\n\n\n\n<li><strong>LUT Tensor Core hardware architecture:<\/strong> Introduces a cutting-edge design for next-generation AI hardware, tailored for low-bit quantization and mixed-precision computations.<\/li>\n<\/ul>\n\n\n\n<p>The following sections describe these techniques in detail.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"ladder-bridging-the-gap-between-custom-data-and-hardware-limits\">Ladder: Bridging the gap between custom data and hardware limits<\/h2>\n\n\n\n<p>Cutting-edge hardware accelerators, such as GPUs, TPUs, and specialized chips, are designed to speed up computationally intensive tasks like deep learning by efficiently handling large-scale operations. These accelerators now integrate lower-bit computing units, such as FP32, FP16, and even FP8, into their architectures.&nbsp;&nbsp;<\/p>\n\n\n\n<p>However, constraints in chip area and hardware costs limit the availability of these units for standard data types. For instance, the NVIDIA V100 Tensor Core GPU supports only FP16, while the A100 supports int2, int4, and int8 but not newer formats like FP8 or OCP-MXFP. Additionally, the rapid development of LLMs often outpaces hardware upgrades, leaving many new data formats unsupported and complicating deployment.<\/p>\n\n\n\n<p>Additionally, while hardware accelerators may lack direct support for custom data types, their memory systems can convert these types into fixed-width data blocks that store any data format. For instance, NF4 tensors can be converted into FP16 or FP32 for floating-point operations.<\/p>\n\n\n\n<p>Building on these insights, we developed the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/ladder-enabling-efficient-low-precision-deep-learning-computing-through-hardware-aware-tensor-transformation\/\" target=\"_blank\" rel=\"noreferrer noopener\">Ladder<\/a> data type compiler, a method to separate data storage from computation, enabling broader support for custom data types. It bridges the gap between emerging custom data formats with the precision types supported by current hardware.<\/p>\n\n\n\n<p>Ladder offers a flexible system for converting between algorithm-specific and hardware-supported data types without data loss. For low-bit applications, it optimizes performance by translating low-bit data into the most efficient formats for the hardware being used. As shown in Figure 1, this includes mapping low-bit computations to supported instructions and efficiently managing data storage across the memory hierarchy.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"465\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure1.png\" alt=\"Figure 1: A diagram illustrating the Ladder architecture. At the top, the tTile-Graph shows a computational flow where inputs in NF4 and FP16 formats feed into a matrix multiplication (MatMul) operation, which outputs in FP16. This output, along with another FP16 input, proceeds to an addition (Add) operation, also in FP16. Below, the tTile-Device schematic depicts a hierarchical memory structure with L2\/Global Memory, L1\/Shared Memory, and L0\/Register, organized under 'Core.' Transformations occur in the loading and storing stages around computation, with arrows indicating data flow. The scheduling mechanism assigns operations to different layers of the memory hierarchy to optimize performance.\" class=\"wp-image-1123677\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure1.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure1-300x136.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure1-768x349.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure1-240x109.png 240w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Figure 1: The Ladder architecture<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"evaluating-ladder\">Evaluating Ladder<\/h3>\n\n\n\n<p>Evaluations of Ladder on NVIDIA and AMD GPUs show that it outperforms existing deep neural network (DNN) compilers for natively supported data types. It also handles custom data types not supported by GPUs, achieving speedups of up to 14.6 times.&nbsp;<\/p>\n\n\n\n<p>As the first system to support custom low-precision data types for running DNNs on modern hardware accelerators, Ladder provides researchers with flexibility in optimizing data types. It also enables hardware developers to support a wider range of data types without requiring hardware modifications.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"t-mac-table-lookup-for-mpgemm-without-multiplication\">T-MAC: Table-lookup for mpGEMM without multiplication<\/h2>\n\n\n\n<p>Deploying low-bit quantized LLMs on edge devices often requires dequantizing models to ensure hardware compatibility. However, this approach has two major drawbacks:&nbsp;<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Performance:<\/strong> Dequantization overhead can result in poor performance, negating the benefits of low-bit quantization.<\/li>\n\n\n\n<li><strong>Development:<\/strong> Developers must redesign data layouts and kernels for different mixed precisions.<\/li>\n<\/ol>\n\n\n\n<p>To address these challenges, we introduce <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/t-mac-cpu-renaissance-via-table-lookup-for-low-bit-llm-deployment-on-edge\/\" target=\"_blank\" rel=\"noreferrer noopener\">T-MAC<\/a>, a novel LUT-based method that enables mpGEMM without dequantization or multiplication.&nbsp;<\/p>\n\n\n\n<p>T-MAC replaces traditional multiplication operations with bit-wise table lookups, offering a unified and scalable solution for mpGEMM. It incorporates techniques to reduce the size of tables and store them directly on the chip, minimizing the overhead of accessing data from memory. By eliminating dequantization and lowering computational costs, T-MAC enables efficient inference of low-bit LLMs on resource-constrained edge devices. Figure 2 illustrates T-MAC\u2019s architecture.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1268\" height=\"719\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure2.png\" alt=\"Figure 2: A diagram showing offline and online processes for bit-serial computation. Offline: integer weights are decomposed into 1-bit indices and permuted into tiles. Online: activations are precomputed with 1-bit patterns, processed via a lookup table (LUT), and aggregated using weighted summation in bit-serial aggregation.\" class=\"wp-image-1123683\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure2.png 1268w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure2-300x170.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure2-1024x581.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure2-768x435.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure2-240x136.png 240w\" sizes=\"auto, (max-width: 1268px) 100vw, 1268px\" \/><figcaption class=\"wp-element-caption\">Figure 2. Overview of the T-MAC system<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"evaluating-t-mac\">Evaluating T-MAC<\/h3>\n\n\n\n<p>Performance evaluations of T-MAC on low-bit models demonstrated substantial benefits in efficiency and speed. On the Surface Laptop 7 with the Qualcomm Snapdragon X Elite chipset, T-MAC achieved:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>48 tokens per second for the 3B BitNet-b1.58 model&nbsp;<\/li>\n\n\n\n<li>30 tokens per second for the 2-bit 7B Llama model&nbsp;<\/li>\n\n\n\n<li>20 tokens per second for the 4-bit 7B Llama model<\/li>\n<\/ul>\n\n\n\n<p>These speeds far exceed average human reading rates, outperforming llama.cpp by 4\u20135 times and doubling the speed of a dedicated NPU accelerator.&nbsp;Even on lower-end devices like the Raspberry Pi 5, T-MAC made it possible for the 3B BitNet-b1.58 model to generate 11 tokens per second. It also proved highly power-efficient, matching llama.cpp&#8217;s generation rate while using only <sup>1<\/sup>\/<sub>4<\/sub> to <sup>1<\/sup>\/<sub>6<\/sub> of the CPU cores.<\/p>\n\n\n\n<p>These results establish T-MAC as a practical solution for deploying LLMs on edge devices with standard CPUs, without relying on GPUs or NPUs. T-MAC allows LLMs to run efficiently on resource-constrained devices, expanding their applicability across a wider range of scenarios.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"lut-tensor-core-driving-hardware-for-mpgemm\">LUT Tensor Core: Driving hardware for mpGEMM<\/h2>\n\n\n\n<p>While T-MAC and Ladder optimize mpGEMM on existing CPU and GPU architectures, improving computational efficiency, they cannot match the performance of dedicated hardware accelerators with built-in LUT support. Achieving significant improvements in performance, power, and area (PPA) requires overcoming four key challenges:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Table precompute and storage:<\/strong> Precomputing and storing LUTs add overhead, increasing area usage, latency, and storage requirements, which can reduce overall efficiency gains.<\/li>\n\n\n\n<li><strong>Bit-width flexibility:<\/strong> Hardware must support various precision levels, such as int4\/2\/1 for weights and FP16\/8 or int8 for activations, along with their combinations. This flexibility is crucial for accommodating diverse model architectures and use cases.<\/li>\n\n\n\n<li><strong>LUT tiling shape: <\/strong>Inefficient tiling shapes can raise storage costs and limit reuse opportunities, adversely affecting performance and efficiency.<\/li>\n\n\n\n<li><strong>Instruction and compilation: <\/strong>LUT-based mpGEMM requires a new instruction set. Existing compilation stacks, designed for standard GEMM hardware, may not optimally map and schedule these instructions, complicating integration with LLM inference software.<\/li>\n<\/ol>\n\n\n\n<p>In response, we developed <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/lut-tensor-core-lookup-table-enables-efficient-low-bit-llm-inference-acceleration\/\">LUT Tensor Core<\/a>, a software-hardware codesign for low-bit LLM inference. To address precomputation overhead in conventional LUT-based methods, we introduce techniques like software-based DFG transformation, operator fusion, and table symmetrization to optimize table precomputation and storage. Additionally, we propose a hardware design with an elongated tiling shape to support table reuse and a bit-serial design to handle various precision combinations in mpGEMM.<\/p>\n\n\n\n<p>To integrate with existing GPU microarchitectures and software stacks, we extended the MMA instruction set, added new LMMA instructions, and developed a cuBLAS-like software stack for easy integration into existing DNN frameworks. We also created a compiler for end-to-end execution planning on GPUs with LUT Tensor Core. This design and workflow, illustrated in Figure 3, enabled the quick and seamless adoption of LUT Tensor Core.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1266\" height=\"533\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure3.png\" alt=\"Figure 3: Diagram of the LUT Tensor Core workflow. The left side shows operator fusion, where 'Norm' produces activations for pre-computation, and 'Weight Reinterpretation' processes low-bit weights. Both feed into LUT-mpGEMM, utilizing an activation LUT table and reinterpreted weights. The right side illustrates the LUT Tensor Core, comprising a LUT table for precomputed values, low-bit weights, and multiplexers (MUX) for computation.\" class=\"wp-image-1123680\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure3.png 1266w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure3-300x126.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure3-1024x431.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure3-768x323.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-bit_figure3-240x101.png 240w\" sizes=\"auto, (max-width: 1266px) 100vw, 1266px\" \/><figcaption class=\"wp-element-caption\">Figure 3. The LUT Tensor Core workflow<\/figcaption><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"evaluating-lut-tensor-core\">Evaluating LUT Tensor Core<\/h3>\n\n\n\n<p>Testing LUT Tensor Core on low-bit LLMs, such as BitNet and Llama, showed significant performance gains, achieving 6.93 times the inference speed while using just 38.3% of the area of a traditional Tensor Core. With nearly identical model accuracy, this results in a 20.9-fold increase in computational density and an 11.2-fold boost in energy efficiency. As AI models grow in scale and complexity, LUT Tensor Core enables low-bit LLMs to be applied in new and diverse scenarios.<\/p>\n\n\n\n<p>We believe the LUT technique could drive a paradigm shift in AI model inference. Traditional methods rely on multiplication and accumulation operations, whereas LUT implementations provide higher transistor density, greater throughput per chip area, lower energy costs, and better scalability. As large models adopt low-bit quantization, the LUT method could become the standard for system and hardware design, advancing the next generation of AI hardware innovation.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"unlocking-new-possibilities-for-embodied-ai\">Unlocking new possibilities for embodied AI<\/h2>\n\n\n\n<p>Low-bit quantization improves the efficiency of running large models on edge devices while also enabling model scaling by reducing the bits used to represent each parameter. This scaling enhances model capabilities, generality, and expressiveness, as shown by the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/the-era-of-1-bit-llms-all-large-language-models-are-in-1-58-bits\/\">BitNet model<\/a>, which starts with a low-bit configuration and expands.<\/p>\n\n\n\n<p>Technologies like T-MAC, Ladder, and LUT Tensor Core provide solutions for running low-bit quantized LLMs, supporting efficient operation across edge devices and encouraging researchers to design and optimize LLMs using low-bit quantization. By reducing memory and computational demands, low-bit LLMs could power embodied AI systems, such as robots, enabling dynamic perception and real-time environmental interaction.<\/p>\n\n\n\n<p><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/T-MAC\" target=\"_blank\" rel=\"noreferrer noopener\">T-MAC<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/BitBLAS\" target=\"_blank\" rel=\"noreferrer noopener\">Ladder<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> are open source and available on GitHub. We invite you to test and explore these innovations in AI technology with Microsoft Research.<\/p>\n\n\n\n\t<div class=\"border-bottom border-top border-gray-300 mt-5 mb-5 msr-promo text-center text-md-left alignwide\" data-bi-aN=\"promo\" data-bi-id=\"1085508\">\n\t\t\n\n\t\t<p class=\"msr-promo__label text-gray-800 text-center text-uppercase\">\n\t\t<span class=\"px-4 bg-white display-inline-block font-weight-semibold small\">Spotlight: Blog post<\/span>\n\t<\/p>\n\t\n\t<div class=\"row pt-3 pb-4 align-items-center\">\n\t\t\t\t\t\t<div class=\"msr-promo__media col-12 col-md-5\">\n\t\t\t\t<a class=\"bg-gray-300\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/medfuzz-exploring-the-robustness-of-llms-on-medical-challenge-problems\/\" aria-label=\"MedFuzz: Exploring the robustness of LLMs on medical challenge problems\" data-bi-cN=\"MedFuzz: Exploring the robustness of LLMs on medical challenge problems\" target=\"_blank\">\n\t\t\t\t\t<img decoding=\"async\" class=\"w-100 display-block\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2024\/09\/MedFuzz-BlogHeroFeature-1400x788-1.jpg\" alt=\"MedFuzz blog hero (decorative)\" \/>\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t<div class=\"msr-promo__content p-3 px-5 col-12 col-md\">\n\n\t\t\t\t\t\t\t\t\t<h2 class=\"h4\">MedFuzz: Exploring the robustness of LLMs on medical challenge problems<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<p class=\"large\">Medfuzz tests LLMs by breaking benchmark assumptions, exposing vulnerabilities to bolster real-world accuracy.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<div class=\"wp-block-buttons justify-content-center justify-content-md-start\">\n\t\t\t\t\t<div class=\"wp-block-button\">\n\t\t\t\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/medfuzz-exploring-the-robustness-of-llms-on-medical-challenge-problems\/\" class=\"btn btn-brand glyph-append glyph-append-chevron-right\" aria-label=\"Read more\" data-bi-cN=\"MedFuzz: Exploring the robustness of LLMs on medical challenge problems\" target=\"_blank\">\n\t\t\t\t\t\t\tRead more\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div><!--\/.msr-promo__content-->\n\t<\/div><!--\/.msr-promo__inner-wrap-->\n\t<\/div><!--\/.msr-promo-->\n\t","protected":false},"excerpt":{"rendered":"<p>Advances in low-bit quantization techniques enable efficient operation of LLMs on resource-constrained edge devices. Discover how innovations like T-MAC, Ladder, and LUT Tensor Core improve computational efficiency and enhance hardware compatibility.<\/p>\n","protected":false},"author":38004,"featured_media":1124472,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[1],"tags":[],"research-area":[13556,13547],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[269148,243984,269142],"msr-impact-theme":[264846],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-1123686","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-systems-and-networking","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-blog-homepage-featured","msr-post-option-include-in-river"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199560],"msr_impact_theme":["Computing foundations"],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[920469],"related-projects":[],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Shijie Cao","user_id":40633,"display_name":"Shijie Cao","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/shijiecao\/\" aria-label=\"Visit the profile page for Shijie Cao\">Shijie Cao<\/a>","is_active":false,"last_first":"Cao, Shijie","people_section":0,"alias":"shijiecao"},{"type":"user_nicename","value":"Lingxiao Ma","user_id":39769,"display_name":"Lingxiao Ma","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/lingm\/\" aria-label=\"Visit the profile page for Lingxiao Ma\">Lingxiao Ma<\/a>","is_active":false,"last_first":"Ma, Lingxiao","people_section":0,"alias":"lingm"},{"type":"user_nicename","value":"Ting Cao","user_id":37446,"display_name":"Ting Cao","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/ticao\/\" aria-label=\"Visit the profile page for Ting Cao\">Ting Cao<\/a>","is_active":false,"last_first":"Cao, Ting","people_section":0,"alias":"ticao"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-960x540.jpg\" class=\"img-object-cover\" alt=\"graphical user interface, application, icon\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/01\/Low-Bit-BlogHeroFeature-1400x788-1.jpg 1400w\" sizes=\"auto, (max-width: 960px) 100vw, 960px\" \/>","byline":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/shijiecao\/\" title=\"Go to researcher profile for Shijie Cao\" aria-label=\"Go to researcher profile for Shijie Cao\" data-bi-type=\"byline author\" data-bi-cN=\"Shijie Cao\">Shijie Cao<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/lingm\/\" title=\"Go to researcher profile for Lingxiao Ma\" aria-label=\"Go to researcher profile for Lingxiao Ma\" data-bi-type=\"byline author\" data-bi-cN=\"Lingxiao Ma\">Lingxiao Ma<\/a>, and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/ticao\/\" title=\"Go to researcher profile for Ting Cao\" aria-label=\"Go to researcher profile for Ting Cao\" data-bi-type=\"byline author\" data-bi-cN=\"Ting Cao\">Ting Cao<\/a>","formattedDate":"February 5, 2025","formattedExcerpt":"Advances in low-bit quantization techniques enable efficient operation of LLMs on resource-constrained edge devices. Discover how innovations like T-MAC, Ladder, and LUT Tensor Core improve computational efficiency and enhance hardware compatibility.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1123686","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38004"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1123686"}],"version-history":[{"count":59,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1123686\/revisions"}],"predecessor-version":[{"id":1126740,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1123686\/revisions\/1126740"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1124472"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1123686"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1123686"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1123686"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1123686"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1123686"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1123686"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1123686"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1123686"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1123686"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1123686"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1123686"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}