Measuring the success of our Microsoft 365 Copilot rollout at Microsoft

Microsoft 365 Copilot is a first-of-its-kind technology and demands a first-of-its-kind adoption process. We’re using new approaches to learning and development, community engagement and, of course, measuring success.

Microsoft Digital, the company’s IT organization, has been leading the charge on Copilot adoption; and measuring the value Copilot is driving for internal users has been an essential part of our efforts. Follow along to see how our strategy for understanding the adoption success, trace the evolution of our approach and metrics, and get pointers for tracking your own Copilot rollout.

Learn how AI is already changing work—including here at Microsoft.

Microsoft 365 Copilot: More than just a tool

Several aspects of Microsoft 365 Copilot make it an entirely new opportunity for adoption leaders. To start, it isn’t just a single platform or app accessed through one interface, but an intelligent assistant that manifests in different facets of the Microsoft technology stack. Users engage with Copilot using natural language queries and prompts, a new habit for most people.

“Copilot is like nothing we’ve landed before, so we have to think differently,” says Peter Varey, director of employee insights on the Employee Experience Success team in Microsoft Digital. “Instead of just promoting a new tool, we’re changing the fundamentals of how we work.”

A different way of interacting with technology isn’t the only thing we had to consider. Depending on an employee’s role, the value Copilot provides varies widely. Capturing and reinforcing that value is the core of an effective measurement strategy.

Strategizing for insight

It’s tempting to track usage alone and leave it at that. But we knew that to really support Microsoft employees in their Microsoft 365 Copilot adoption journey, we needed to go deeper.

“People only maintain habits when they become part of their identity,” says Tom Heath, a senior business program manager driving AI transformation in Microsoft Digital. “So, a lot of our adoption strategies are based around ‘sticky metrics’ that demonstrate consistent, habitual usage.”

The core objectives of Microsoft 365 Copilot adoption telemetry

Microsoft Digital established our Microsoft 365 Copilot adoption telemetry workstream according to four main objectives.

App telemetry is only part of the equation. It’s also important to collect qualitative data through listening campaigns that track satisfaction, adoption enablers, and product feedback.

Microsoft’s user experience (UX) experts played a key role in planning and implementing this aspect of our measurement strategy. By grounding our listening in the core tenets of effective UX and emerging guidelines for AI-human interactions, we could ensure we were asking the right questions—an essential piece of harvesting helpful feedback.

“If we can understand what’s happening for the user, we’re in a better position to help them,” says Tara Suan, a senior UX research lead with Microsoft Digital. “It’s less about identifying issues or problems and more molding adoption to their experience.”

Metrics that move the needle

One of the lessons we’ve learned during our ongoing Microsoft 365 Copilot adoption measurement efforts is that useful metrics change as a rollout progresses.

“We’re evolving in this journey and building toward a more thorough understanding of what success looks like,” Varey says. “So, we started with high-level OKRs (Objectives and Key Results) around usage, and now we’ve progressed toward more granular metrics.”

In the beginning, the goal was just to ensure that as many Microsoft employees as possible were trying Copilot and experimenting with AI assistance. To do that, we initially focused on monthly active usage (MAU).

But maturity is about more than just accessing a tool once a month. As adoption has progressed, we’ve been getting more granular. That’s possible because of how we structure our objective and key results (OKR) lifecycles: Each measurement cycle lasts six months before we institute new key metrics, and we’re always looking ahead to the next cycle to evaluate what data will be most useful for our next adoption phase.

We capture these metrics through a combination of internal measurement tools and commercially available solutions that include  Copilot Dashboard in Viva Insights, the Copilot Adoption Report in the Analyst Workbench, and the Experience Insights Dashboard in the Microsoft 365 admin center. Using these tools, we can account for weekly and even daily usage, as well as slicing and dicing the data by role, internal organization, and region. We’re also exploring Copilot distribution by app penetration to understand who’s accessing Copilot through Word, Excel, Teams, or other vectors.

On the qualitative side, we use several options for employee listening. Adoption surveys and in-app sentiment checks provide always-on qualitative data. At the same time, we run listening campaigns through collaborative bug bashes, satisfaction surveys, community outreach through Viva Engage, focus groups, events like “prompt-a-thons,” and more.

These feedback channels help us understand different dimensions of how our employees think about Copilot:

• Net satisfaction: The overall positive or negative experience with Copilot.
• Favorability: Whether Copilot makes employees more productive or faster at their work.
• AI-assisted hours: A measure of time saved through Copilot usage.

In the end, these parallel efforts mean we can collate robust datasets to track both quantitative usage data and qualitative feedback around the employee experience.

Results that inform adoption

Microsoft 365 Copilot adoption metrics are already providing helpful insights that help us ask the right questions to shape further usage across the company.

For example, app telemetry tells us that Microsoft employees use Copilot summarization more than any other feature. It also tells us that most of our users access Copilot through Teams instead of more comprehensive pathways like Graph-grounded Chat.

Our telemetry team extrapolates that the high visibility of in-app prompts—such as the ten-minute Teams meeting summarization reminder—leads to more intuitive access to Copilot capabilities within the flow of work. We can use that information to report to the product group that visible Copilot prompts guide users into beneficial behaviors. We can also signal the adoption team that they might need to supplement less visible Copilot onramps with learning initiatives. That adoption guidance has only become possible as our measurement methods have matured.

From a qualitative perspective, Microsoft employees are deeply satisfied with Copilot. 76% report feeling satisfied with the tool, and 85% are using the tool regularly—more than any other single Microsoft solution.

But listening provides more in-depth information. Patterns of adoption are more illuminating.

“A lot of times, people will share their feelings with you as a researcher,” Suan says. “It’s up to us to peel the layers back and contextualize them to understand what’s really going on, then collaborate with our adoption partners to meet users’ needs.”

One example stands out. With early-career employees, we noticed a distinct arc in Copilot adoption behavior: initial delight and experimentation in the first three weeks of usage, a dip in enthusiasm from weeks three through 10, and then more consistent usage around the 11-week mark. Thanks to this insight, we can provide a consistent, predictable picture of how employees adopt Copilot over time and introduce change management interventions like skilling and reminders that can potentially address that midstream dip.

Finally, it’s important to examine these trends data carefully. If the current adoption for Copilot in Teams is 75%, we can ask why the other 25% of potential users are less engaged. Is it because they don’t use Teams regularly? Is there a language issue? Is it simply a lack of knowledge about how to use the tool?

When employees say they don’t see the value of Copilot for their work, we have a deeper conversation with those respondents to understand their challenges. In many of these follow-ups, we’ve discovered that the difficulty isn’t so much about the tool’s capabilities, but about not having the knowledge and education they need to realize its value.

“A key learning was that a certain percent of folks said they struggled to find time to learn,” Heath says. “So, the change strategy needs to not only understand how to land copilot but also create a better environment for learning and encourage leaders to do so. Learning days, group learning, and gamification are all tactics you can use for this.”

This kind of information has been extremely helpful for our change leaders, who use it to tailor their adoption efforts.

Like technology adoption itself, measuring change never stops maturing. As we continue to track our internal Copilot adoption journey and usage, we’ll keep digging down into new data and asking more directive, deeper questions.

“This is a fundamental way for us to stay in front of adoption and shape our strategy,” says Stephan Kerametlian, director of employee experience in Microsoft Digital. “When we see what the data’s telling us, the trends that emerge, and how our employees feel about Copilot, it puts us in a position to shape both our technology and how we implement it—with people at the center.”