{"id":600474,"date":"2020-02-23T21:23:12","date_gmt":"2019-07-30T11:58:14","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&#038;p=600474"},"modified":"2020-02-23T21:23:13","modified_gmt":"2020-02-24T05:23:13","slug":"micro-codesign","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/micro-codesign\/","title":{"rendered":"Micro Co-design"},"content":{"rendered":"<p>The massive scale of cloud infrastructure services enables, and often necessitates, vertical co-design of the infrastructure stack by cloud providers. <strong>Micro<\/strong>\u00a0co-design is a\u00a0<strong>Mi<\/strong>nimally invasive,\u00a0<strong>C<\/strong>heap, and ret<strong>ro<\/strong>-fittable approach to co-design that extracts efficiency out of existing software infrastructure layers. It does this by making lightweight changes to generic software interfaces. We are exploring micro co-design in the context of four systems: Instalytics, Astra, Gandiva, and Quiver, aimed at improving the efficiency and functionality of key infrastructure for big data analytics and AI.<\/p>\n<ul>\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/instalytics-storage-for-big-data\/\"><u>Instalytics<\/u><\/a> achieves significant cost and performance improvements to big data analytics processing over large data sets, by co-design of the distributed file system layer and the compute layer. Instalytics customizes the 3-way replication that happens at the store, by keeping each copy partitioned by a different dimension, while still ensuring efficient recovery.\u00a0 With an end-to-end implementation and evaluation of Instalytics on a cluster of 500 machines, we have demonstrated significant query performance and cost savings on micro benchmarks and representative production workloads.<\/li>\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/astra-custom-wired-dnns\/\"><u>Astra<\/u><\/a> is a compiler and execution engine for speeding up execution of a deep learning training job. Astra employs co-design by customizing the compilation model to fit the unique properties of DNN training \u2013 in particular, the remarkable predictability across mini-batch iterations. Thus, instead of using a static performance model to reason about which optimizations to use, Astra defers the choice of optimizations to runtime, and uses each mini-batch iteration as an opportunity to explore the optimization space. It uses various techniques such as dynamic fine-grained profiling and barriers \u00a0to prune the large state space of optimizations. We have implemented Astra in two popular deep learning frameworks \u2013 pyTorch and Tensorflow. Compared to native pyTorch and Tensorflow, Astra improves performance of \u201clong tail\u201d models (models that use non-standard operators and hence don\u2019t fit hand-optimized libraries such as cuDNN) by 2.5 to 3x.\u00a0 Compared to Tensorflow XLA which is a static optimizer, Astra achieves up to 70% speedup.<\/li>\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/gandiva-scheduler-for-dnns\/\"><u>Gandiva<\/u><\/a> is a co-designed cluster scheduler for running deep learning training jobs. Similar to Astra, Gandiva leverages the predictability of DNN training jobs to make the scheduler \u201ccontinuous\u201d and \u201cintrospective\u201d. For example, it performs efficient time-slicing by aligning suspension of a job with the end of a mini-batch, thus reducing the amount of data to checkpoint by 50x. It uses application-aware profiling (mini-batch time) to decide when to migrate jobs, and when to pack multiple jobs on the same GPU. Gandiva achieves better cluster utilization and job latency due to the tight integration between the job scheduler and the deep learning framework (Tensorflow and pyTorch). We have implemented Gandiva as a scheduler in Kubernetes, along with changes to two frameworks \u2013 pyTorch and Tensorflow. We see that Gandiva improves cluster utilization by 26%, and improves time to initial feedback (latency) \u00a0by 77%.\u00a0 The time-slicing in Gandiva also improves hyper-parameter exploration\/ AutoML by nearly 6x.<\/li>\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/quiver-an-informed-storage-cache-for-deep-learning\/\">Quiver<\/a> is an informed storage cache for deep learning training (DLT) jobs in a cluster of GPUs. Quiver employs domain-specific intelligence within the caching layer, to achieve much higher efficiency compared to a generic storage cache. First, Quiver uses a secure hash-based addressing to transparently reuse cached data across multiple jobs and even multiple users operating on the same dataset. Second, by co-designing with the deep learning framework (e.g. PyTorch), Quiver employs a technique of <em>substitutable cache hits<\/em> to get more value from the existing contents of the cache, thus avoiding cache thrashing when cache capacity is much smaller than the working set. Third, Quiver dynamically prioritizes cache allocation to jobs that benefit the most from the caching.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>The massive scale of cloud infrastructure services enables, and often necessitates, vertical co-design of the infrastructure stack by cloud providers. Micro\u00a0co-design is a\u00a0Minimally invasive,\u00a0Cheap, and retro-fittable approach to co-design that extracts efficiency out of existing software infrastructure layers. It does this by making lightweight changes to generic software interfaces. We are exploring micro co-design in [&hellip;]<\/p>\n","protected":false},"featured_media":395198,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"research-area":[13556,13547],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-600474","msr-project","type-msr-project","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-research-area-systems-and-networking","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[566493,571077,559971,638817],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Kaushik Rajan","user_id":32574,"people_section":"Section name 1","alias":"krajan"},{"type":"user_nicename","display_name":"Ramachandran Ramjee","user_id":33337,"people_section":"Section name 1","alias":"ramjee"},{"type":"user_nicename","display_name":"Nipun Kwatra","user_id":37634,"people_section":"Section name 1","alias":"nkwatra"},{"type":"user_nicename","display_name":"Lidong Zhou","user_id":32673,"people_section":"Section name 1","alias":"lidongz"}],"msr_research_lab":[199562],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/600474"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":13,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/600474\/revisions"}],"predecessor-version":[{"id":638823,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/600474\/revisions\/638823"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/395198"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=600474"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=600474"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=600474"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=600474"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=600474"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}