Experimentation Platform
Articles
Microsoft’s Experimentation Platform (ExP) provides a platform used by product teams across Microsoft to run 1,000s of A/B tests every month. From a product perspective this means that we have a big responsibility both as a steward for data-driven decision-making…
At Microsoft, the trustworthiness of A/B test results is essential for data-driven decisions. In earlier parts of our trustworthy experimentation series, we shared patterns and some recommendations to increase trustworthiness in both pre- and during- experiment stages. Now we tackle…
Introduction Data plays a vital role in the lifecycle of every single product in the technology industry. With data, we can generate insights to improve products and provide a superior customer experience. Yet insights are only actionable if they are derived from trustworthy data. For example, a navigation…
Microsoft Teams is a communication platform [1]. It integrates meet, chat, call and collaborate in one place. The application updates multiple times a month [2], with additional new features and iterative improvements to existing features. To ensure high quality user…
We are all familiar with terabytes and petabytes. But have you heard about zettabytes (1000 petabytes)[1]? Worldwide data volume is expected to hit 163 zettabytes by 2025, 10 times the data in 2017. Your product will contribute to the surge,…
This is a common story among experimenters: you have a hypothesis to test, you code the change you want to deploy, and you design an A/B test to properly measure the impact of the change on the user. After the…
At Microsoft, we continuously improve products by developing new features for them. To facilitate data-driven decision-making in software development, product teams across Microsoft run tens of thousands of A/B tests each year. While the primary purpose of A/B testing is…
Trustworthy data and analyses are key to making sound business decisions, particularly when it comes to A/B testing. Ignoring data quality issues or biases introduced through design and interpretations risks leading to incorrect conclusions that could hurt your product. In…
Written by Aleksander Fabijan (Microsoft), Benjamin Arai (Microsoft), Pavel Dmitriev (Outreach.io), and Lukas Vermeer (Booking.com) At the time when we published the Experimentation Evolution Model in 2017 [1], we never expected such an interest in it. Practitioners from hundreds of…