{"id":474279,"date":"2018-03-16T15:02:38","date_gmt":"2018-03-16T22:02:38","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&#038;p=474279"},"modified":"2018-03-16T15:21:41","modified_gmt":"2018-03-16T22:21:41","slug":"cloudviews-materialized-views-big-data-workloads","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/cloudviews-materialized-views-big-data-workloads\/","title":{"rendered":"CloudViews: Materialized Views for Big Data Workloads"},"content":{"rendered":"<p>Analytics workloads in Microsoft&#8217;s clusters are generated by users submitting SCOPE scripts, which are compiled down to jobs, which are then executed on the clusters. \u00a0An analysis of these workloads suggests\u00a0that there is significant overlap between the sub-graphs of the different jobs. \u00a0That is, the sub-graphs of different jobs compute the <em>same<\/em> result. \u00a0 This suggests that we can build upon view materialization to improve performance. \u00a0However, doing view materialization at a massive scale, where the system processes hundreds of thousands of jobs per day, poses interesting challenges. \u00a0This project has two parts:<\/p>\n<ul>\n<li>An <em>algorithmic<\/em> component where we develop algorithms for automatically identifying which views to materialize.<\/li>\n<li>A <em>systems-building<\/em>\u00a0component where we implement our algorithms into the SCOPE runtime system and deploy them in Microsoft&#8217;s clusters.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Analytics workloads in Microsoft&#8217;s clusters are generated by users submitting SCOPE scripts, which are compiled down to jobs, which are then executed on the clusters. \u00a0An analysis of these workloads suggests\u00a0that there is significant overlap between the sub-graphs of the different jobs. \u00a0That is, the sub-graphs of different jobs compute the same result. \u00a0 This [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13547],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-474279","msr-project","type-msr-project","status-publish","hentry","msr-research-area-systems-and-networking","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"2016-05-02","related-publications":[723658,723682],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"guest","display_name":"Alekh Jindal","user_id":474288,"people_section":"Section name 1","alias":""}],"msr_research_lab":[],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/474279"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":1,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/474279\/revisions"}],"predecessor-version":[{"id":474282,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/474279\/revisions\/474282"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=474279"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=474279"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=474279"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=474279"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=474279"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}