{"id":382118,"date":"2017-05-18T08:38:07","date_gmt":"2017-05-18T15:38:07","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&#038;p=382118"},"modified":"2018-11-05T09:39:32","modified_gmt":"2018-11-05T17:39:32","slug":"project-pelican","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/project-pelican\/","title":{"rendered":"Project Pelican"},"content":{"rendered":"<h2>Pelican: A building block for exascale cold data storage<\/h2>\n<p>Pelican aims to store infrequently accessed (cold) data as inexpensively as possible.<\/p>\n<p>The amount of data stored is growing at a huge rate, but not all of it is \u201chot,\u201d i.e. frequently accessed.\u00a0 There is little reason to store cold data in the same high-performance, high-cost systems as hot data. Our goal was to design a storage system\u2014called Pelican\u2014specifically to take advantage of the needs of cold data workload.<\/p>\n<h3>Resource constraints for storing large amounts of data:<\/h3>\n<ul>\n<li>Accessibility<\/li>\n<li>Throughput<\/li>\n<li>Latency<\/li>\n<li>Hardware cost<\/li>\n<li>Power cost<\/li>\n<\/ul>\n<p>The design of Pelican, a rack-scale hard disk based storage unit, is to optimize for lower total cost of ownership by trading off access latency; only 8% of its drives can spin concurrently.\u00a0 Because cold storage by definition means it is rarely accessed, this trade-off makes sense.\u00a0 In effect, Pelican is designed to \u201cright provision\u201d its storage\u2014its server, power, cooling and interconnect bandwidth resources are designed to support cold data workloads.<\/p>\n<p>The challenge with this approach is that constraining the number of drives that can spin is a complex resource management problem, which it solves with a unique data layout and IO scheduling scheme.<\/p>\n<p>In this <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/pelican-a-building-block-for-exascale-cold-data-storage\/\" target=\"_blank\" rel=\"noopener noreferrer\">project<\/a>, we evaluate the performance of a prototype Pelican, and compare it against a traditional storage rack using a cross-validated simulator. We show that compared to an over-provisioned storage rack, Pelican performs well for cold workloads, providing high throughput with access latency and drive failure rates.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-383444 size-large\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/pelican-illustration-full-1-1024x480.jpg\" alt=\"Project Pelican\" width=\"1024\" height=\"480\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/pelican-illustration-full-1-1024x480.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/pelican-illustration-full-1-300x141.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/pelican-illustration-full-1-768x360.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/pelican-illustration-full-1.jpg 1920w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/p>\n<h3>\u00a0Pelican: Rack-scale co-design<\/h3>\n<ul>\n<li>Hardware & software co-designed:\n<ul>\n<li>Power, Cooling, Mechanical, HDDs & Software.<\/li>\n<li>Trade latency for lower cost.<\/li>\n<\/ul>\n<\/li>\n<li>Massive density, low per-drive overhead.<\/li>\n<li>1152 3.5\u201d HDDs per 52U.<\/li>\n<li>2 servers, PCIe bus stretched rack-wide.\n<ul>\n<li>4x 10G links out of rack.<\/li>\n<\/ul>\n<\/li>\n<li>Only 8% of disks can spin.<\/li>\n<\/ul>\n<p>Once updates to the rack design are complete, we expect to deploy this system in Windows Azure Storage datacenters worldwide.<\/p>\n\t\t\t<div class=\"ms-grid \">\n\t\t\t<div class=\"ms-row\">\n\t\t\t\t\t<div  class=\"m-col-12-24\" >\n\t\t<img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-382496\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/Pelican1-2.jpg\" alt=\"Project Pelican\" width=\"594\" height=\"400\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/Pelican1-2.jpg 737w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/Pelican1-2-300x202.jpg 300w\" sizes=\"auto, (max-width: 594px) 100vw, 594px\" \/>\t<\/div>\n\t \t<div  class=\"m-col-12-24\" >\n\t\t<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-382478\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/Pelican2.png\" alt=\"Project Pelican\" width=\"594\" height=\"400\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/Pelican2.png 737w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/05\/Pelican2-300x202.png 300w\" sizes=\"auto, (max-width: 594px) 100vw, 594px\" \/>\t<\/div>\n\t<\/p>\t\t\t<\/div>\n\t\t<\/div>\n\t\t\n","protected":false},"excerpt":{"rendered":"<p>Pelican: A building block for exascale cold data storage Pelican aims to store infrequently accessed (cold) data as inexpensively as possible. The amount of data stored is growing at a huge rate, but not all of it is \u201chot,\u201d i.e. frequently accessed.\u00a0 There is little reason to store cold data in the same high-performance, high-cost [&hellip;]<\/p>\n","protected":false},"featured_media":382502,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13547],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-382118","msr-project","type-msr-project","status-publish","has-post-thumbnail","hentry","msr-research-area-systems-and-networking","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"2012-11-01","related-publications":[167585,292682,382274],"related-downloads":[],"related-videos":[382535],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"user_nicename","display_name":"Shobana Balakrishnan","user_id":33627,"people_section":"Group 1","alias":"shobanab"},{"type":"user_nicename","display_name":"Richard Black","user_id":33417,"people_section":"Group 1","alias":"rjblack"},{"type":"user_nicename","display_name":"Austin Donnelly","user_id":31148,"people_section":"Group 1","alias":"austind"},{"type":"user_nicename","display_name":"Sergey Legtchenko","user_id":33580,"people_section":"Group 1","alias":"serleg"},{"type":"user_nicename","display_name":"Eric Peterson","user_id":31753,"people_section":"Group 1","alias":"ericpete"},{"type":"user_nicename","display_name":"Ant Rowstron","user_id":31061,"people_section":"Group 1","alias":"antr"}],"msr_research_lab":[199561,199562],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/382118","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":4,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/382118\/revisions"}],"predecessor-version":[{"id":547782,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/382118\/revisions\/547782"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/382502"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=382118"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=382118"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=382118"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=382118"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=382118"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}