\n\n\n\n\n\n

Automated Learning and Intelligence for Causation and Economics<\/h2>\n\n\n\n
Alice is a project to direct Artificial Intelligence towards economic decision making. We are building tools that combine state-of-the-art machine learning with econometrics \u2013 the measurement of economic systems — in order to bring automation to economic decision making. The heart of this project is a striving to measure causation: if you want to understand or make policy decisions in a complex economy, you need to know why the system moves the way it does.<\/p>\n\n\n\n
Microsoft has a long tradition of work at the intersection of Economics and Computer Science, and this team brings together researchers from across Social Science, AI, and ML. The Alice project allows us to dramatically scale the success that we\u2019ve had in adapting existing ML technology for economic applications and in developing new deep learning architectures for causal inference. All of our research is tied to concrete policy-relevant applications, including work on demand estimation and price optimization, on measuring effectiveness of advertising and sales strategies, and on the design of incentives for desirable healthcare and education outcomes. The goal is to use AI to improve and democratize economic research while taking from economic theory to push the frontier of AI.<\/p>\n\n\n\n
\n
Check out our code<\/a><\/div>\n<\/div>\n\n\n\n\n\n
\n
A Microsoft & TripAdvisor Case Study<\/h3>\n
Causal AI for Customer Segmentation<\/h2>\n
$\"Asian$ <\/p>\n
\n\t\n\t\tView case study PDF\t<\/a>\n\n\t<\/div>\n
TripAdvisor<\/strong> is an online travel research company that empowers people around the world to plan and enjoy the ideal trip. TripAdvisor\u2019s mission is to give travelers a platform to share their experiences to promote transparency in the travel industry and enable informed consumer choices.<\/p>\n
The ALICE<\/strong><\/em> project at Microsoft Research New England^{1<\/sup> applies Artificial Intelligence concepts to economic decision making. To make policy decisions in a complex economy, you need to know why the system moves the way it does. The ALICE team\u2019s innovative tools make this kind of cause and effect analysis more reliable, scalable, and accessible for data scientists without extensive economic training. These tools are collected in the open source, production level quality EconML<\/strong> <\/em>library.^{2<\/sup><\/p>\n}}
TripAdvisor\u2019s ability to provide users with accurate information hinges on other travelers sharing their experiences on the platform. TripAdvisor and MSR joined forces to understand whether a membership model would improve user engagement on the website.<\/p>\n
The Challenge<\/h3>\n
On a cold New England day, a chance encounter between two data scientists working on causal problems, one at MSR and one at TripAdvisor, uncovered an exciting opportunity for collaboration: MSR could better understand the causal questions of interest to business decision makers and build intelligent, flexible and reusable solutions, while TripAdvisor would learn about state-of-the-art tools that they could immediately apply to inform business decisions.<\/p>\n
The two teams agreed to work together on a problem of mutual interest: leveraging an existing A\/B test for user retention policies to understand the effect of a membership model on user engagement. The ALICE team was perfectly positioned to contribute research and build new machine learning tools to tackle this problem.<\/p>\n
Strategy<\/h3>\n
TripAdvisor wanted to know whether promoting membership on their platform could drive engagement and bookings. They couldn\u2019t just look directly at the existing data, comparing members and non-members, because the customers who had chosen to become members were exactly the most engaged ones. Nor could they run a direct A\/B test because they could not force users to sign up for membership. Fortunately, TripAdvisor had just run an experiment to explore user retention by offering a random subset of customers an easier sign-up process for membership.<\/p>\n
$\"the$ <\/p>\n
$\"arrow$ <\/p>\n
\u201dDeveloping a deep understanding of our travelers so we can create truly relevant experiences is at the core of what we do at TripAdvisor. Our partnership with Microsoft Research has allowed us to unlock critical insights that inform how to improve those experiences in a 1:1 manner.\u201d <\/em><\/div>\n
\u2014Matthew Dacey,<\/strong> Vice President Membership and Growth at TripAdvisor<\/div>\n
Results<\/h3>\n
The ALICE team developed a new statistical method to exploit this indirect nudge towards membership to measure the direct effects of membership on engagement. Their approach expands on the traditional economic technique of instrumental variables, which adjusts for imperfect compliance: some customers who were offered the easier sign-up still declined membership while some customers without the easier process signed up anyway.<\/p>\n
TripAdvisor found that membership does have a positive effect on user engagement, but this effect varies widely across users. Some major drivers of this variation are the platform from which the user accesses the TripAdvisor platform and the pages that the user had visited before the experiment.<\/p>\n
$\"graph$ <\/p>\n
The key innovation that revealed this variation is the development of a Machine Learning (ML) based method for estimating heterogeneous causal effects in A\/B tests with non-compliance. Traditional instrumental variable methods do not allow for complex individual-level differences in either compliance with the experiment or the effect of the intervention.<\/p>\n
$\"Male$ <\/p>\n
MSR\u2019s novelty was to reduce the problem to a sequence of standard Machine Learning steps and take advantage of the many recent algorithmic and statistical advances in machine learning on non-linear and high-dimensional modelling. The ML algorithms allow data scientists to automatically segment the market in a datadriven manner as opposed to predefining a small set of key user features based on prior, perhaps incorrect, beliefs.<\/p>\n
The methodology has been implemented in our production level quality EconML library<\/strong><\/em>.^{2<\/sup> The technical details of this methodology were published at the 33rd Conference on Neural Information Processing Systems.^{3<\/sup><\/p>\n}}
Moving Forward<\/h3>\n
Thanks to their collaboration with Microsoft Research, TripAdvisor can understand and differentiate their customers in new ways. TripAdvisor is now more focused on providing membership opportunities to iPhone users and customers who visit vacation rental pages.<\/p>\n
The EconML software package includes a suite of cutting-edge statistical tools to answer causal questions. In addition to this kind of market segmentation, the EconML package is well suited to estimating price sensitivities and designing personalized pricing plans or attributing growth in revenue across multiple drivers, such as customer engagements or advertising campaigns. For more information visit the ALICE team<\/strong><\/em> website^{1<\/sup> or the EconML website<\/strong><\/em>.^{2<\/sup><\/p>\n<\/blockquote>\n}}
\n
^1<\/sup>https:\/\/www.microsoft.com\/en-us\/research\/project\/alice\/ (opens in new tab)<\/span><\/a>
\n^2<\/sup>https:\/\/econml.azurewebsites.net\/ (opens in new tab)<\/span><\/a>
\n^{3<\/sup> “Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments”(NeurIPS\u201919)
\n}https:\/\/arxiv.org\/abs\/1905.10176 (opens in new tab)<\/span><\/a><\/p>\n<\/blockquote>\n\n\n","protected":false},"excerpt":{"rendered":"
Alice is a project to direct Artificial Intelligence towards economic decision making. We are building tools that combine state-of-the-art machine learning with econometrics \u2013 the measurement of economic systems — in order to bring automation to economic decision making. The heart of this project is a striving to measure causation: if you want to understand or make policy decisions in a complex economy, you need to know why the system moves the way it does.<\/p>\n","protected":false},"featured_media":418151,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr_group_start":"","footnotes":""},"research-area":[13556,13548],"msr-group-type":[243694],"msr-locale":[268875],"msr-impact-theme":[],"class_list":["post-656316","msr-group","type-msr-group","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-research-area-economics","msr-group-type-group","msr-locale-en_us"],"msr_group_start":"","msr_detailed_description":"","msr_further_details":"","msr_hero_images":[],"msr_research_lab":[199563],"related-researchers":[{"type":"user_nicename","display_name":"Eleanor Dillon","user_id":38467,"people_section":"Leads","alias":"eldillon"},{"type":"user_nicename","display_name":"Keith Battocchi","user_id":36548,"people_section":"Team","alias":"kebatt"},{"type":"user_nicename","display_name":"Friederike Niedtner","user_id":39919,"people_section":"Team","alias":"fniedtner"},{"type":"user_nicename","display_name":"Fabio Vera","user_id":41398,"people_section":"Team","alias":"fabiovera"},{"type":"user_nicename","display_name":"Sida Peng","user_id":37008,"people_section":"Affiliates","alias":"sidpeng"},{"type":"user_nicename","display_name":"Markus Mobius","user_id":32980,"people_section":"Affiliates","alias":"mobius"},{"type":"user_nicename","display_name":"Nicole Immorlica","user_id":33086,"people_section":"Affiliates","alias":"nicimm"},{"type":"user_nicename","display_name":"Brendan Lucier","user_id":31303,"people_section":"Affiliates","alias":"brlucier"},{"type":"user_nicename","display_name":"Emre Kiciman","user_id":31739,"people_section":"Affiliates","alias":"emrek"},{"type":"user_nicename","display_name":"Amit Sharma","user_id":30997,"people_section":"Affiliates","alias":"amshar"},{"type":"user_nicename","display_name":"Robert Osazuwa Ness","user_id":41009,"people_section":"Affiliates","alias":"robertness"}],"related-publications":[471336,471504,471510,475107,475113,493301,493316,626067,626082,626151,626160],"related-downloads":[494567,598717],"related-videos":[],"related-projects":[656325,596605],"related-events":[],"related-opportunities":[],"related-posts":[],"tab-content":[{"id":0,"name":"Case Study","content":"
\r\n
A Microsoft & TripAdvisor Case Study<\/h3>\r\n
Causal AI for Customer Segmentation<\/h2>\r\n $\"Asian$ \r\n
[msr-button text=\"View case study PDF\" url=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2020\/04\/MSR_ALICE_casestudy_2020.pdf\" ]<\/div>\r\nTripAdvisor<\/strong> is an online travel research company that empowers people around the world to plan and enjoy the ideal trip. TripAdvisor\u2019s mission is to give travelers a platform to share their experiences to promote transparency in the travel industry and enable informed consumer choices.\r\n\r\nThe ALICE<\/strong><\/em> project at Microsoft Research New England^{1<\/sup> applies Artificial Intelligence concepts to economic decision making. To make policy decisions in a complex economy, you need to know why the system moves the way it does. The ALICE team\u2019s innovative tools make this kind of cause and effect analysis more reliable, scalable, and accessible for data scientists without extensive economic training. These tools are collected in the open source, production level quality EconML<\/strong> <\/em>library.^{2<\/sup>\r\n\r\nTripAdvisor\u2019s ability to provide users with accurate information hinges on other travelers sharing their experiences on the platform. TripAdvisor and MSR joined forces to understand whether a membership model would improve user engagement on the website.\r\nThe Challenge<\/h3>\r\nOn a cold New England day, a chance encounter between two data scientists working on causal problems, one at MSR and one at TripAdvisor, uncovered an exciting opportunity for collaboration: MSR could better understand the causal questions of interest to business decision makers and build intelligent, flexible and reusable solutions, while TripAdvisor would learn about state-of-the-art tools that they could immediately apply to inform business decisions.\r\n\r\nThe two teams agreed to work together on a problem of mutual interest: leveraging an existing A\/B test for user retention policies to understand the effect of a membership model on user engagement. The ALICE team was perfectly positioned to contribute research and build new machine learning tools to tackle this problem.\r\n}}
Strategy<\/h3>\r\nTripAdvisor wanted to know whether promoting membership on their platform could drive engagement and bookings. They couldn\u2019t just look directly at the existing data, comparing members and non-members, because the customers who had chosen to become members were exactly the most engaged ones. Nor could they run a direct A\/B test because they could not force users to sign up for membership. Fortunately, TripAdvisor had just run an experiment to explore user retention by offering a random subset of customers an easier sign-up process for membership.\r\n\r\n $\"the$ \r\n\r\n $\"arrow$ \r\n
\u201dDeveloping a deep understanding of our travelers so we can create truly relevant experiences is at the core of what we do at TripAdvisor. Our partnership with Microsoft Research has allowed us to unlock critical insights that inform how to improve those experiences in a 1:1 manner.\u201d <\/em><\/div>\r\n
\u2014Matthew Dacey,<\/strong> Vice President Membership and Growth at TripAdvisor<\/div>\r\n
Results<\/h3>\r\nThe ALICE team developed a new statistical method to exploit this indirect nudge towards membership to measure the direct effects of membership on engagement. Their approach expands on the traditional economic technique of instrumental variables, which adjusts for imperfect compliance: some customers who were offered the easier sign-up still declined membership while some customers without the easier process signed up anyway.\r\n\r\nTripAdvisor found that membership does have a positive effect on user engagement, but this effect varies widely across users. Some major drivers of this variation are the platform from which the user accesses the TripAdvisor platform and the pages that the user had visited before the experiment.\r\n\r\n $\"graph$ \r\n\r\nThe key innovation that revealed this variation is the development of a Machine Learning (ML) based method for estimating heterogeneous causal effects in A\/B tests with non-compliance. Traditional instrumental variable methods do not allow for complex individual-level differences in either compliance with the experiment or the effect of the intervention.\r\n\r\n $\"Male$ \r\n\r\nMSR\u2019s novelty was to reduce the problem to a sequence of standard Machine Learning steps and take advantage of the many recent algorithmic and statistical advances in machine learning on non-linear and high-dimensional modelling. The ML algorithms allow data scientists to automatically segment the market in a datadriven manner as opposed to predefining a small set of key user features based on prior, perhaps incorrect, beliefs.\r\n\r\nThe methodology has been implemented in our production level quality EconML library<\/strong><\/em>.^{2<\/sup> The technical details of this methodology were published at the 33rd Conference on Neural Information Processing Systems.^3<\/sup>\r\n}Moving Forward<\/h3>\r\nThanks to their collaboration with Microsoft Research, TripAdvisor can understand and differentiate their customers in new ways. TripAdvisor is now more focused on providing membership opportunities to iPhone users and customers who visit vacation rental pages.\r\n\r\nThe EconML software package includes a suite of cutting-edge statistical tools to answer causal questions. In addition to this kind of market segmentation, the EconML package is well suited to estimating price sensitivities and designing personalized pricing plans or attributing growth in revenue across multiple drivers, such as customer engagements or advertising campaigns. For more information visit the ALICE team<\/strong><\/em> website^{1<\/sup> or the EconML website<\/strong><\/em>.^{2<\/sup><\/blockquote>\r\n}}\r\n
^1<\/sup>https:\/\/www.microsoft.com\/en-us\/research\/project\/alice\/<\/a>\r\n^2<\/sup>https:\/\/econml.azurewebsites.net\/<\/a>\r\n^{3<\/sup> \"Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments\"(NeurIPS\u201919)\r\n}https:\/\/arxiv.org\/abs\/1905.10176<\/a><\/p>\r\n<\/blockquote>"}],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-group\/656316","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-group"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-group"}],"version-history":[{"count":11,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-group\/656316\/revisions"}],"predecessor-version":[{"id":934026,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-group\/656316\/revisions\/934026"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/418151"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=656316"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=656316"},{"taxonomy":"msr-group-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-group-type?post=656316"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=656316"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=656316"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}