{"id":956640,"date":"2023-07-27T09:00:00","date_gmt":"2023-07-27T16:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=956640"},"modified":"2023-07-26T13:59:30","modified_gmt":"2023-07-26T20:59:30","slug":"a-fail-in-place-approach-for-sustainable-server-operations","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/a-fail-in-place-approach-for-sustainable-server-operations\/","title":{"rendered":"A fail-in-place approach for sustainable server operations"},"content":{"rendered":"\n

This research paper was presented at the <\/em>17th USENIX Symposium on Operating Systems Design and Implementation (opens in new tab)<\/span><\/a> (OSDI), a premier forum for discussing the design, implementation, and implications of systems software.<\/em><\/p>\n\n\n\n

\"\"Microsoft<\/figure>\n\n\n\n

Cloud platforms aim to provide a seamless user experience, alleviating the challenge and complexity of managing physical servers in datacenters. Hardware failures are one such challenge, as individual server components can fail independently, and while failures affecting individual customers are rare, cloud platforms encounter a substantial volume of server failures. Currently, when a single component in a server fails, the entire server needs to be serviced by a technician. This all-or-nothing operating model is increasingly becoming a hindrance to achieving cloud sustainability goals.<\/p>\n\n\n\n

Finding a sustainable server repair solution<\/h2>\n\n\n\n

A sustainable cloud platform (opens in new tab)<\/span><\/a> should be water-positive and carbon-negative. Water consumption in datacenters primarily arises from the need for cooling, and liquid cooling (opens in new tab)<\/span><\/a> has emerged as a potential solution for waterless cooling (opens in new tab)<\/span><\/a>. Paradoxically, liquid cooling also increases the complexity and time required to repair servers. Therefore, reducing the demand for repairs becomes essential to achieving water-positive status.<\/p>\n\n\n\n

To become carbon-negative, Microsoft has been procuring renewable energy for its datacenters<\/a> since 2016. Currently, Azure\u2019s carbon emissions largely arise during server manufacturing, as indicated in Microsoft\u2019s carbon emission report<\/a>. Extending the lifetime of servers, which Microsoft has recently done<\/a> to a minimum of six years, is a key strategy to reduce server-related carbon emissions. However, longer server lifetimes highlight the importance of server repairs, which not only contribute significantly to costs but also to carbon emissions. Moreover, sourcing replacement components can sometimes pose challenges. Consequently, finding ways to minimize the need for repairs becomes crucial.<\/p>\n\n\n\n

<\/div>\n\n\n\n\t
\n\t\t\n\n\t\t

\n\t\tMicrosoft research podcast<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Illustrated\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

NeurIPS 2024: The co-evolution of AI and systems with Lidong Zhou<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Just after his NeurIPS 2024 keynote on the co-evolution of systems and AI, Microsoft CVP Lidong Zhou joins the podcast to discuss how rapidly advancing AI impacts the systems supporting it and the opportunities to use AI to enhance systems engineering itself.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tListen now\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

Reducing server repairs by 60% with Hyrax<\/h2>\n\n\n\n

To support Microsoft sustainability goals, our paper, \u201cHyrax: Fail-in-Place Server Operation in Cloud<\/a>,\u201d proposes that cloud platforms adopt a fail-in-place<\/em> paradigm where servers with faulty components continue to host virtual machines (VMs) without the need for immediate repairs. With this approach, cloud platforms could significantly reduce repair requirements, decreasing costs and carbon emissions at the same time. However, implementing fail-in-place in practice poses several challenges.<\/p>\n\n\n\n

First, we want to ensure graceful degradation, where faulty components are identified and deactivated in a controlled manner. Second, deactivating common components like dual in-line memory modules (DIMMs) can significantly impact server performance due to reduced memory interleaving. It is crucial to prevent VM customers from experiencing loss in performance resulting from these deactivations. Finally, the cloud platform must be capable of using the capacity of servers with deactivated components, necessitating algorithmic changes in VM scheduling and structural adjustments in the cloud control plane.<\/p>\n\n\n\n

To address these challenges, our paper<\/a> introduces Hyrax, the first implementation of fail-in-place for cloud compute servers. Through a multi-year study of component failures across five server generations, we found that existing servers possess sufficient redundancy to overcome the most common types of server component failures. We propose effective mechanisms for component deactivation that can mitigate a wide range of possibilities, including issues like corroded connectors or chip failures. Additionally, Hyrax introduces a degraded server state and scheduling optimizations to the production control plane, enabling effective utilization of servers with deactivated components, as illustrated in Figure 1.<\/p>\n\n\n\n

\"Figure
Figure 1. Compared with an all-or-nothing operation, Hyrax adds an additional online server state and two additional steps in the offline state transitions.<\/figcaption><\/figure>\n\n\n\n

Our results demonstrate that Hyrax achieves a 60 percent reduction in repair demand without compromising datacenter capacity, as shown in Figure 2. This reduction in repairs leads to a 5 percent decrease in embodied carbon emissions over a typical six-year deployment period, as fewer replacement components are needed. In a subsequent study<\/a>, we show that Hyrax enables servers to run for 30 percent longer, resulting in a proportional reduction in embodied carbon. We also demonstrate that Hyrax does not impact VM performance.<\/p>\n\n\n\n

\"Figure
Figure 2. Hyrax effectively reduces the need for repairs across multiples configuration points without compromising datacenter capacity.<\/figcaption><\/figure>\n\n\n\n

Deactivating memory modules without impacting performance<\/h2>\n\n\n\n

One of Hyrax\u2019s key technical challenges is the need to deactivate components at the firmware level, as software-based deactivations prove to be insufficient. This requirement requires addressing previously unexplored performance implications.<\/p>\n\n\n\n

A good example is the deactivation of a memory module, specifically a DIMM. To understand DIMM deactivation, it is important to consider how CPUs access memory, which is usually hidden from software. This occurs at the granularity of a cache line, which is 64 bytes and resides on a single DIMM. Larger data is divided into cache lines and distributed among all DIMMs connected to a CPU in a round-robin fashion. This interleaving mechanism ensures that while one DIMM is handling cache line N, another DIMM serves cache line N+1. From a software standpoint, memory is typically presented as a uniform address space that encompasses all cache lines across all the DIMMs attached to the CPU. Accessing any portion of this address space is equally fast in terms of memory bandwidth. Figure 3 shows an example of a server with six memory channels populated with two 32-GB DIMMs each. From the software perspective, the entire 384 GB of address space appears indistinguishable and offers a consistent 120 GB\/sec bandwidth.<\/p>\n\n\n\n

However, deactivating a DIMM causes the interleaving policy to reconfigure in unexpected ways. Figure 3 demonstrates this scenario, where the second DIMM on channel B (B2) has been identified as faulty and subsequently deactivated. Consequently, three different parts of the address space exhibit different characteristics: 120 GB\/sec (six-way interleaving), 80 GB\/sec (four-way interleaving), and 20 GB\/sec (one-way interleaving). These performance differences are invisible to software and naively scheduling VMs on such a server can lead to variable performance, a suboptimal outcome.<\/p>\n\n\n\n

\"Figure
Figure 3. A healthy server (top) offers the same memory bandwidth throughout its address space. A server that is degraded due to the deactivation of the second DIMM on channel B (bottom), offers three different bandwidths regions. Hyrax effectively manages this bandwidth heterogeneity.<\/figcaption><\/figure>\n\n\n\n

Hyrax enables cloud platforms to work around this issue by scheduling VMs on only the parts of the address space that offer sufficient performance for that VM\u2019s requirements. Our paper<\/a> discusses how this works in more detail.<\/p>\n\n\n\n

Implications and looking forward<\/h2>\n\n\n\n

Hyrax is the first fail-in-place system for cloud computing servers, paving the way for future improvements. One potential enhancement involves reconsidering the approach to memory regions with 20 GB\/sec memory bandwidth. Instead of using them only for small VMs, we could potentially allocate these regions to accommodate large data structures, such as by adding buffers for input-output devices that require more than 20 GB\/sec of bandwidth.<\/p>\n\n\n\n

Failing-in-place offers significant flexibility when it comes to repairs. For example, instead of conducting daily repair trips to individual servers scattered throughout a datacenter, we are exploring the concept of batching repairs, where technicians would visit a row of server racks once every few weeks to address issues across multiple servers simultaneously. By doing so, we can save valuable time and resources while creating new research avenues for optimizing repair schedules that intelligently balance capacity loss and repair efforts.<\/p>\n\n\n\n

Achieving sustainability goals demands collective efforts across society. In this context, we introduce fail-in-place as a research direction for both datacenter hardware and software systems, directly tied to water and carbon efficiency. Beyond refining the fail-in-place concept itself and exploring new server designs, this new paradigm also opens up new pathways for improving maintenance processes using an environmentally friendly approach.<\/p>\n","protected":false},"excerpt":{"rendered":"

Managing server failures at the scale of a cloud platform is challenging. The Hyrax fail-in-place approach reduces the need for immediate repairs and creates a path toward lowering water consumption and carbon emissions in cloud datacenters. <\/p>\n","protected":false},"author":42183,"featured_media":956691,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[1],"tags":[],"research-area":[13547],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-956640","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-systems-and-networking","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[282170],"related-projects":[757045],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Daniel S. Berger","user_id":38892,"display_name":"Daniel S. Berger","author_link":"Daniel S. Berger<\/a>","is_active":false,"last_first":"Berger, Daniel S.","people_section":0,"alias":"daberg"},{"type":"guest","value":"marisa-you","user_id":"956952","display_name":"Marisa You","author_link":"Marisa You<\/a>","is_active":true,"last_first":"You, Marisa","people_section":0,"alias":"marisa-you"},{"type":"user_nicename","value":"Celine Irvene","user_id":40636,"display_name":"Celine Irvene","author_link":"Celine Irvene<\/a>","is_active":false,"last_first":"Irvene, Celine","people_section":0,"alias":"celineirvene"},{"type":"guest","value":"mark-jung","user_id":"956958","display_name":"Mark Jung","author_link":"Mark Jung<\/a>","is_active":true,"last_first":"Jung, Mark","people_section":0,"alias":"mark-jung"},{"type":"guest","value":"tyler-narmore","user_id":"956964","display_name":"Tyler Narmore","author_link":"Tyler Narmore<\/a>","is_active":true,"last_first":"Narmore, Tyler","people_section":0,"alias":"tyler-narmore"},{"type":"guest","value":"jacob-shapiro","user_id":"956970","display_name":"Jacob Shapiro","author_link":"Jacob Shapiro<\/a>","is_active":true,"last_first":"Shapiro, Jacob","people_section":0,"alias":"jacob-shapiro"},{"type":"user_nicename","value":"Luke Marshall","user_id":37386,"display_name":"Luke Marshall","author_link":"Luke Marshall<\/a>","is_active":false,"last_first":"Marshall, Luke","people_section":0,"alias":"lumarsha"},{"type":"guest","value":"savyasachi-samal","user_id":"956976","display_name":"Savyasachi Samal","author_link":"Savyasachi Samal<\/a>","is_active":true,"last_first":"Samal, Savyasachi","people_section":0,"alias":"savyasachi-samal"},{"type":"guest","value":"preetha-subbarayalu","user_id":"956982","display_name":"Preetha Subbarayalu","author_link":"Preetha Subbarayalu<\/a>","is_active":true,"last_first":"Subbarayalu, Preetha","people_section":0,"alias":"preetha-subbarayalu"},{"type":"guest","value":"ashish-raniwala","user_id":"956988","display_name":"Ashish Raniwala","author_link":"Ashish Raniwala<\/a>","is_active":true,"last_first":"Raniwala, Ashish","people_section":0,"alias":"ashish-raniwala"},{"type":"guest","value":"brijesh-warrier","user_id":"956994","display_name":"Brijesh Warrier","author_link":"Brijesh Warrier<\/a>","is_active":true,"last_first":"Warrier, Brijesh","people_section":0,"alias":"brijesh-warrier"},{"type":"user_nicename","value":"Ricardo Bianchini","user_id":33393,"display_name":"Ricardo Bianchini","author_link":"Ricardo Bianchini<\/a>","is_active":false,"last_first":"Bianchini, Ricardo","people_section":0,"alias":"ricardob"}],"msr_type":"Post","featured_image_thumbnail":"\""Microsoft","byline":"","formattedDate":"July 27, 2023","formattedExcerpt":"Managing server failures at the scale of a cloud platform is challenging. The Hyrax fail-in-place approach reduces the need for immediate repairs and creates a path toward lowering water consumption and carbon emissions in cloud datacenters.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/956640","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/42183"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=956640"}],"version-history":[{"count":22,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/956640\/revisions"}],"predecessor-version":[{"id":956949,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/956640\/revisions\/956949"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/956691"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=956640"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=956640"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=956640"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=956640"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=956640"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=956640"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=956640"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=956640"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=956640"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=956640"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=956640"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}