{"id":235469,"date":"2015-01-27T08:00:00","date_gmt":"2015-01-27T08:00:00","guid":{"rendered":"https:\/\/blogs.technet.microsoft.com\/inside_microsoft_research\/2015\/01\/27\/trill-moves-big-data-faster-by-orders-of-magnitude\/"},"modified":"2016-07-20T07:29:31","modified_gmt":"2016-07-20T14:29:31","slug":"trill-moves-big-data-faster-by-orders-of-magnitude","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/trill-moves-big-data-faster-by-orders-of-magnitude\/","title":{"rendered":"Trill Moves Big Data Faster, by Orders of Magnitude"},"content":{"rendered":"<p class=\"posted-by\">Posted by <span class=\"author\">George Thomas Jr.<\/span><\/p>\n<p><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/msdnshared.blob.core.windows.net\/media\/TNBlogsFS\/prod.evol.blogs.technet.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/00\/90\/35\/trill-throughput-vs-spe-x_source.JPG\" target=\"_blank\"><img decoding=\"async\" style=\"float: right; margin: 7px 5px;\" title=\"Trill's throughput compared to SPE-X - click to see full graph\" src=\"https:\/\/msdnshared.blob.core.windows.net\/media\/TNBlogsFS\/prod.evol.blogs.technet.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/00\/90\/35\/trill-throughput-vs-spe-x_275.png\" alt=\"Trill's throughput compared to SPE-X\" \/><span class=\"sr-only\"> (opens in new tab)<\/span><\/a>In today\u2019s high-productivity computing environments that process dizzying amounts of data each millisecond, a research project named for \u201ca trillion events per day\u201d may seem relatively ordinary.<\/p>\n<p>But when you understand that <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Trill project page\" href=\"http:\/\/research.microsoft.com\/en-us\/projects\/trill\/\" target=\"_blank\">Trill<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, a new high-performance streaming analytics engine developed by Microsoft researchers, can process data at two to four orders of magnitude faster than today\u2019s streaming engines, well, now you\u2019re getting into &#8220;wow&#8221; territory, especially considering Trill is just a .NET library:<\/p>\n<ul>\n<li>As a single-node engine library, any .NET application, service, or platform can easily include it and start processing queries;<\/li>\n<li>A temporal query language allows users to express complex queries over real-time and\/or offline data sets; and,<\/li>\n<li>Trill\u2019s high performance across intended usage scenarios means users can get results significantly faster than before.<\/li>\n<\/ul>\n<p>&#8220;Prior systems have only achieved subsets of these benefits, but Trill provides all of these advantages in one package, so to speak.&#8221; says <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Badrish Chandramouli\" href=\"http:\/\/research.microsoft.com\/en-us\/people\/badrishc\/\" target=\"_blank\">Badrish Chandramouli<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, one of the Microsoft researchers who developed Trill.<\/p>\n<p>Its secret? Trill incorporates new techniques and algorithms that process events in batches, with the data within those batches organized in new ways that enable queries to execute much more efficiently than before, but to users it&#8217;s the same as working with a .NET library &#8212; no need to leave the .NET environment.<\/p>\n<p>Bing Ads customers, in fact, already are enjoying the paradigm shift, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Bings Ads customers get results in less than an hour\" href=\"http:\/\/advertise.bingads.microsoft.com\/en-us\/blogpost\/129607\" target=\"_blank\">seeing results in less than an hour of launching Bing ad campaigns<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n<p>And it doesn&#8217;t end there.<\/p>\n<p>&#8220;While it can be integrated into today\u2019s distribution fabrics such as SCOPE (in Bing ads) and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Project Orleans\" href=\"http:\/\/research.microsoft.com\/en-us\/projects\/orleans\/\" target=\"_blank\">Orleans<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> (in <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Halo\" href=\"https:\/\/www.halowaypoint.com\/en-us\">Halo<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>) to achieve scale-out, we are currently looking at developing new techniques to achieve even better performance in distributed computing and <a title=\"Internet of Things\" href=\"http:\/\/www.microsoft.com\/en-us\/server-cloud\/internet-of-things.aspx\" target=\"_blank\">Internet-of-Things<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> scenarios,&#8221; Chandramouli says.<\/p>\n<p>Started in early 2012 by Chandramouli and fellow researcher Jonathan Goldstein, and detailed in <em><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Trill: A High-Performance Incremental Query Processor for Diverse Analytics\" href=\"http:\/\/research.microsoft.com\/pubs\/231690\/trill-vldb2015.pdf\" target=\"_blank\">Trill: A High-Performance Incremental Query Processor for Diverse Analytics<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/em> (1.5 MB .pdf), its roots can be traced to earlier research in Complex Event Detection and Response algebra (CEDR), dating back to 2007, and published in <em><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Consistent Streaming Through Time: A Vision for Event Stream Processing\" href=\"http:\/\/research.microsoft.com\/pubs\/156569\/cidr07p42.pdf\" target=\"_blank\">Consistent Streaming Through Time: A Vision for Event Stream Processing<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/em> (660 KB .pdf). And in the interim, a successive paper that introduced the idea of using a single language and engine to handle real-time and offline datasets, <em><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Temporal Analytics on Big Data for Web Advertising\" href=\"http:\/\/research.microsoft.com\/apps\/pubs\/default.aspx?id=155806\" target=\"_blank\">Temporal Analytics on Big Data for Web Advertising<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/em>, won Best-paper at IDCE 2012.<\/p>\n<p>\u201cFrom CEDR to Trill to multiple Microsoft products: This body of work is a great example of how within Microsoft Research we evolve from science to technology to business impact,\u201d says <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Jeannette Wing, Corporate Vice President, Microsoft Research\" href=\"http:\/\/research.microsoft.com\/en-us\/press\/jeannette-wing.aspx\" target=\"_blank\">Jeannette Wing<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, Corporate Vice President, Microsoft Research. \u201cIt also shows the nature and value of long-term research, where patience and persistence really pay off.\u201d<\/p>\n<p>While not directly available to the public, Trill also is being used elsewhere at Microsoft, as a query processor within the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Azure Stream Analytics\" href=\"http:\/\/azure.microsoft.com\/en-us\/services\/stream-analytics\/\" target=\"_blank\">Azure Stream Analytics<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> service, currently under public preview. Additional collaborators on Trill include: <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Mike Barnett\" href=\"http:\/\/research.microsoft.com\/en-us\/people\/mbarnett\/\" target=\"_blank\">Mike Barnett<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Rob DeLine\" href=\"http:\/\/research.microsoft.com\/en-us\/people\/rdeline\/\" target=\"_blank\">Rob DeLine<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"Danyel Fisher\" href=\"http:\/\/research.microsoft.com\/en-us\/people\/danyelf\/\" target=\"_blank\">Danyel Fisher<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, John Platt, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" title=\"James Terwilliger\" href=\"http:\/\/research.microsoft.com\/en-us\/people\/jamest\/\" target=\"_blank\">James Terwilliger<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, and John Wernsing.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Posted by George Thomas Jr. In today\u2019s high-productivity computing environments that process dizzying amounts of data each millisecond, a research project named for \u201ca trillion events per day\u201d may seem relatively ordinary. But when you understand that Trill, a new high-performance streaming analytics engine developed by Microsoft researchers, can process data at two to four [&hellip;]<\/p>\n","protected":false},"author":30766,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[194470,194460,194463],"tags":[186587,194758,200595,200677,201063,201201,202137,202249,202815,197037,204821,204063,204343],"research-area":[13555,13547],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-235469","post","type-post","status-publish","format-standard","hentry","category-computer-architecture","category-search-and-information-retrieval","category-systems","tag-net","tag-azure-stream-analytics","tag-badrish-chandramouli","tag-bing-ads","tag-computer-systems-and-networking","tag-danyel-fisher","tag-james-terwilliger","tag-john-wernsing","tag-mike-barnett","tag-rob-deline","tag-search-and-information-retrieval","tag-streaming-analytics","tag-trill","msr-research-area-search-information-retrieval","msr-research-area-systems-and-networking","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[171207],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"January 27, 2015","formattedExcerpt":"Posted by George Thomas Jr. In today\u2019s high-productivity computing environments that process dizzying amounts of data each millisecond, a research project named for \u201ca trillion events per day\u201d may seem relatively ordinary. But when you understand that Trill, a new high-performance streaming analytics engine developed&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/235469"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/30766"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=235469"}],"version-history":[{"count":1,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/235469\/revisions"}],"predecessor-version":[{"id":235625,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/235469\/revisions\/235625"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=235469"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=235469"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=235469"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=235469"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=235469"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=235469"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=235469"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=235469"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=235469"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=235469"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=235469"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}