{"id":17190,"date":"2024-10-21T08:00:00","date_gmt":"2024-10-21T15:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?p=17190"},"modified":"2024-10-31T09:12:29","modified_gmt":"2024-10-31T16:12:29","slug":"measuring-the-success-of-our-microsoft-365-copilot-rollout-at-microsoft","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/insidetrack\/blog\/measuring-the-success-of-our-microsoft-365-copilot-rollout-at-microsoft\/","title":{"rendered":"Measuring the success of our Microsoft 365 Copilot rollout at Microsoft"},"content":{"rendered":"\n
\"Microsoft<\/figure>\n\n\n\n

Microsoft 365 Copilot is a first-of-its-kind technology and demands a first-of-its-kind adoption process. We\u2019re using new approaches to learning and development, community engagement and, of course, measuring success.<\/p>\n\n\n\n

Microsoft Digital, the company\u2019s IT organization, has been leading the charge on Copilot adoption; and measuring the value Copilot is driving for internal users has been an essential part of our efforts. Follow along to see how our strategy for understanding the adoption success, trace the evolution of our approach and metrics, and get pointers for tracking your own Copilot rollout.<\/p>\n\n\n\n

Learn how AI is already changing work\u2014including here at Microsoft.<\/a><\/em><\/p>\n\n\n\n

Microsoft 365 Copilot: More than just a tool<\/h2>\n\n\n\n
\"Images
Peter Varey, Tom Heath, and Tara Suan are part of our efforts to augment our Microsoft 365 Copilot adoption efforts with data and insights.<\/figcaption><\/figure>\n\n\n\n

Several aspects of Microsoft 365 Copilot make it an entirely new opportunity for adoption leaders. To start, it isn\u2019t just a single platform or app accessed through one interface, but an intelligent assistant that manifests in different facets of the Microsoft technology stack. Users engage with Copilot using natural language queries and prompts, a new habit for most people.<\/p>\n\n\n\n

\n

\u201cCopilot is like nothing we\u2019ve landed before, so we have to think differently,\u201d says Peter Varey, director of employee insights on the Employee Experience Success team in Microsoft Digital. \u201cInstead of just promoting a new tool, we\u2019re changing the fundamentals of how we work.\u201d<\/p>\n<\/blockquote>\n\n\n\n

A different way of interacting with technology isn\u2019t the only thing we had to consider. Depending on an employee\u2019s role, the value Copilot provides varies widely. Capturing and reinforcing that value is the core of an effective measurement strategy.<\/p>\n\n\n\n

Strategizing for insight<\/strong><\/strong><\/h2>\n\n\n\n

It\u2019s tempting to track usage alone and leave it at that. But we knew that to really support Microsoft employees in their Microsoft 365 Copilot adoption journey, we needed to go deeper.<\/p>\n\n\n\n

\n

\u201cPeople only maintain habits when they become part of their identity,\u201d says Tom Heath, a senior business program manager driving AI transformation in Microsoft Digital. \u201cSo, a lot of our adoption strategies are based around \u2018sticky metrics\u2019 that demonstrate consistent, habitual usage.\u201d<\/p>\n<\/blockquote>\n\n\n\n

The core objectives of Microsoft 365 Copilot adoption telemetry<\/h3>\n\n\n\n
\n
\n
\"\"<\/figure>\n\n\n\n

Drive clarity on usage<\/strong>, the reporting options available, and the intended audience for each.<\/p>\n<\/div>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n

Ensure the access and visibility<\/strong> we need to support other adoption
workstreams effectively.<\/p>\n<\/div>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n

Provide valuable Customer Zero feedback<\/strong> on the telemetry sources and
dashboards available to customers.<\/p>\n<\/div>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n

Use our reporting to
capture key insights<\/strong> that would help us further fine-tune our adoption
approach.<\/p>\n<\/div>\n<\/div>\n\n\n\n

Microsoft Digital established our Microsoft 365 Copilot adoption telemetry workstream according to four main objectives.<\/em><\/p>\n\n\n\n

App telemetry is only part of the equation. It\u2019s also important to collect qualitative data through listening campaigns that track satisfaction, adoption enablers, and product feedback.<\/p>\n\n\n\n

Microsoft\u2019s user experience (UX) experts played a key role in planning and implementing this aspect of our measurement strategy. By grounding our listening in the core tenets of effective UX and emerging guidelines for AI-human interactions, we could ensure we were asking the right questions\u2014an essential piece of harvesting helpful feedback.<\/p>\n\n\n\n

\n

\u201cIf we can understand what\u2019s happening for the user, we\u2019re in a better position to help them,\u201d says Tara Suan, a senior UX research lead with Microsoft Digital. \u201cIt\u2019s less about identifying issues or problems and more molding adoption to their experience.\u201d<\/p>\n<\/blockquote>\n\n\n\n

Metrics that move the needle<\/h2>\n\n\n\n

One of the lessons we\u2019ve learned during our ongoing Microsoft 365 Copilot adoption measurement efforts is that useful metrics change as a rollout progresses.<\/p>\n\n\n\n

\u201cWe\u2019re evolving in this journey and building toward a more thorough understanding of what success looks like,\u201d Varey says. \u201cSo, we started with high-level OKRs (Objectives and Key Results) around usage, and now we\u2019ve progressed toward more granular metrics.\u201d<\/p>\n\n\n\n

In the beginning, the goal was just to ensure that as many Microsoft employees as possible were trying Copilot and experimenting with AI assistance. To do that, we initially focused on monthly active usage (MAU). <\/p>\n\n\n\n

But maturity is about more than just accessing a tool once a month. As adoption has progressed, we\u2019ve been getting more granular. That\u2019s possible because of how we structure our objective and key results (OKR) lifecycles: Each measurement cycle lasts six months before we institute new key metrics, and we\u2019re always looking ahead to the next cycle to evaluate what data will be most useful for our next adoption phase.<\/p>\n\n\n\n

We capture these metrics through a combination of internal measurement tools and commercially available solutions that include  Copilot Dashboard in Viva Insights<\/a>, the Copilot Adoption Report in the Analyst Workbench<\/a>, and the Experience Insights Dashboard in the Microsoft 365 admin center<\/a>. Using these tools, we can account for weekly and even daily usage, as well as slicing and dicing the data by role, internal organization, and region. We\u2019re also exploring Copilot distribution by app penetration to understand who\u2019s accessing Copilot through Word, Excel, Teams, or other vectors.<\/p>\n\n\n\n

On the qualitative side, we use several options for employee listening. Adoption surveys and in-app sentiment checks provide always-on qualitative data. At the same time, we run listening campaigns through collaborative bug bashes, satisfaction surveys, community outreach through Viva Engage, focus groups, events like \u201cprompt-a-thons,\u201d and more.<\/p>\n\n\n\n

These feedback channels help us understand different dimensions of how our employees think about Copilot:<\/p>\n\n\n\n