{"id":965463,"date":"2023-09-05T13:36:07","date_gmt":"2023-09-05T20:36:07","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-blog-post&p=965463"},"modified":"2023-09-05T13:36:39","modified_gmt":"2023-09-05T20:36:39","slug":"experimentation-and-the-north-star-metric","status":"publish","type":"msr-blog-post","link":"https:\/\/www.microsoft.com\/en-us\/research\/articles\/experimentation-and-the-north-star-metric\/","title":{"rendered":"Experimentation and the North Star Metric"},"content":{"rendered":"\n
Ram Hariharan and Will Dubyak<\/em><\/p>\n\n\n\n We must measure user impact to continue enhancing Copilot User Experience. <\/p>\n\n\n\n This post addresses application of A\/B testing and the North Star metric to this question. It uses an actual example to demonstrate test set up, interpretation of results, and sound decision making. It shows the power of these tools and highlights the hazards of too-rapid interpretation. <\/p>\n\n\n\n There are many technical improvements deriving from thoughtful application of A\/B Experimentation, but this paper should be viewed from the perspective of enhancing customer experience; our focus is always doing what we must to make customers more successful. Given the increasing embrace of Copilot across the range of our products, we see a tremendous opportunity to use experimentation to make Copilot more impactful on the end-to-end experience. <\/p>\n\n\n\n This post is not a recipe; there are volumes written about testing and metrics. Nor is it a comprehensive overview of the example use case. It is meant to illustrate how A\/B testing and metrics in real life can be applied and show how misinterpretation or misuse can lead to weaker decision making. <\/p>\n\n\n\n Two key ideas: <\/p>\n\n\n\n Microsoft Power Automate is a low-code tool to create a flow to streamline automating repetitive processes. Power Automate Copilot makes creating flows easy, saving user time and effort. Users simply describe the automated workflow they want in everyday language and Copilot transforms the words into a flow, jumpstarting the authoring process. For example, a text input \u201cSend me a notification when I get a high importance email from my manager\u201d generates this flow: <\/p>\n\n\n\nFor thousands of years mariners have understood the value of the North Star as a beacon to help navigate a journey.\u00a0 It is a trusted source of truth; reference to it is the basis of life and death navigational decisions.\u00a0 If they adhere to its message, they find their way home.\u00a0 They ignore it at their peril.<\/em>\u00a0<\/h3>\n\n\n\n
\n
\n
The use case<\/em> <\/h3>\n\n\n\n