Experimentation Platform
Articles
How does Microsoft ExP team manage metrics computation for A/B tests at scale, providing trustworthy analyses of thousands of metrics for customers using different underlying compute infrastructures? In this article we give an overview of the metric computation pipeline at…
A/B Testing Across Products
Imagine a purely hypothetical scenario where the Bing team wants to implement an icon that allows customers to view content in Microsoft News. Imagine that the Bing team, not knowing how this would affect user experience, wants to A/B test…
When we run A/B tests, we choose the randomization unit[1]. For example, if we want to evaluate whether a product feature increases user engagement, we will run an A/B test and randomize by user. Typically, A/B tests on websites, mobile…
An A/A test is an A/B test where versions A and B are identical; there is no real change to the product. An A/A test is one of the most powerful tools for testing an A/B testing platform end-to-end. When…
During World War II, Abraham Wald who was a statistician at Columbia University arrived at a very counterintuitive solution. He was tasked by the military to determine where to place armor on airplanes to increase their chances of surviving the…
Trustworthy data and analyses are key to making sound business decisions, particularly when it comes to AB testing. Ignoring data quality issues or biases introduced through design and interpretations risks leading to incorrect conclusions that could hurt your product. In…
It’s critical to make cautious and informed decisions about changes that impact users– especially during times of extra stress. Prompted by the Covid-19 crisis, Microsoft’s ExP team shares their thinking about effective use of A/B testing to mitigate uncertainty during…