Microsoft 365 Copilot is a first-of-its-kind technology and demands a first-of-its-kind adoption process. We’re using new approaches to learning and development, community engagement and, of course, measuring success.

Microsoft Digital, the company’s IT organization, has been leading the charge on Copilot adoption; and measuring the value Copilot is driving for internal users has been an essential part of our efforts. Follow along to see how our strategy for understanding the adoption success, trace the evolution of our approach and metrics, and get pointers for tracking your own Copilot rollout.

Learn how AI is already changing work—including here at Microsoft.

Microsoft 365 Copilot: More than just a tool

Several aspects of Microsoft 365 Copilot make it an entirely new opportunity for adoption leaders. To start, it isn’t just a single platform or app accessed through one interface, but an intelligent assistant that manifests in different facets of the Microsoft technology stack. Users engage with Copilot using natural language queries and prompts, a new habit for most people.

“Copilot is like nothing we’ve landed before, so we have to think differently,” says Peter Varey, director of employee insights on the Employee Experience Success team in Microsoft Digital. “Instead of just promoting a new tool, we’re changing the fundamentals of how we work.”

A different way of interacting with technology isn’t the only thing we had to consider. Depending on an employee’s role, the value Copilot provides varies widely. Capturing and reinforcing that value is the core of an effective measurement strategy.

Strategizing for insight

It’s tempting to track usage alone and leave it at that. But we knew that to really support Microsoft employees in their Microsoft 365 Copilot adoption journey, we needed to go deeper.

“People only maintain habits when they become part of their identity,” says Tom Heath, a senior business program manager driving AI transformation in Microsoft Digital. “So, a lot of our adoption strategies are based around ‘sticky metrics’ that demonstrate consistent, habitual usage.”

The core objectives of Microsoft 365 Copilot adoption telemetry

Drive clarity on usage, the reporting options available, and the intended audience for each.

Ensure the access and visibility we need to support other adoption
workstreams effectively.

Provide valuable Customer Zero feedback on the telemetry sources and
dashboards available to customers.

Use our reporting to
capture key insights that would help us further fine-tune our adoption
approach.

Microsoft Digital established our Microsoft 365 Copilot adoption telemetry workstream according to four main objectives.

App telemetry is only part of the equation. It’s also important to collect qualitative data through listening campaigns that track satisfaction, adoption enablers, and product feedback.

Microsoft’s user experience (UX) experts played a key role in planning and implementing this aspect of our measurement strategy. By grounding our listening in the core tenets of effective UX and emerging guidelines for AI-human interactions, we could ensure we were asking the right questions—an essential piece of harvesting helpful feedback.

“If we can understand what’s happening for the user, we’re in a better position to help them,” says Tara Suan, a senior UX research lead with Microsoft Digital. “It’s less about identifying issues or problems and more molding adoption to their experience.”

Metrics that move the needle

One of the lessons we’ve learned during our ongoing Microsoft 365 Copilot adoption measurement efforts is that useful metrics change as a rollout progresses.

“We’re evolving in this journey and building toward a more thorough understanding of what success looks like,” Varey says. “So, we started with high-level OKRs (Objectives and Key Results) around usage, and now we’ve progressed toward more granular metrics.”

In the beginning, the goal was just to ensure that as many Microsoft employees as possible were trying Copilot and experimenting with AI assistance. To do that, we initially focused on monthly active usage (MAU).

But maturity is about more than just accessing a tool once a month. As adoption has progressed, we’ve been getting more granular. That’s possible because of how we structure our objective and key results (OKR) lifecycles: Each measurement cycle lasts six months before we institute new key metrics, and we’re always looking ahead to the next cycle to evaluate what data will be most useful for our next adoption phase.

We capture these metrics through a combination of internal measurement tools and commercially available solutions that include  Copilot Dashboard in Viva Insights, the Copilot Adoption Report in the Analyst Workbench, and the Experience Insights Dashboard in the Microsoft 365 admin center. Using these tools, we can account for weekly and even daily usage, as well as slicing and dicing the data by role, internal organization, and region. We’re also exploring Copilot distribution by app penetration to understand who’s accessing Copilot through Word, Excel, Teams, or other vectors.

On the qualitative side, we use several options for employee listening. Adoption surveys and in-app sentiment checks provide always-on qualitative data. At the same time, we run listening campaigns through collaborative bug bashes, satisfaction surveys, community outreach through Viva Engage, focus groups, events like “prompt-a-thons,” and more.

These feedback channels help us understand different dimensions of how our employees think about Copilot:

• Net satisfaction: The overall positive or negative experience with Copilot.
• Favorability: Whether Copilot makes employees more productive or faster at their work.
• AI-assisted hours: A measure of time saved through Copilot usage.

In the end, these parallel efforts mean we can collate robust datasets to track both quantitative usage data and qualitative feedback around the employee experience.

Results that inform adoption

Microsoft 365 Copilot adoption metrics are already providing helpful insights that help us ask the right questions to shape further usage across the company.

For example, app telemetry tells us that Microsoft employees use Copilot summarization more than any other feature. It also tells us that most of our users access Copilot through Teams instead of more comprehensive pathways like Graph-grounded Chat.

Our telemetry team extrapolates that the high visibility of in-app prompts—such as the ten-minute Teams meeting summarization reminder—leads to more intuitive access to Copilot capabilities within the flow of work. We can use that information to report to the product group that visible Copilot prompts guide users into beneficial behaviors. We can also signal the adoption team that they might need to supplement less visible Copilot onramps with learning initiatives. That adoption guidance has only become possible as our measurement methods have matured.

From a qualitative perspective, Microsoft employees are deeply satisfied with Copilot. 76% report feeling satisfied with the tool, and 85% are using the tool regularly—more than any other single Microsoft solution.

But listening provides more in-depth information. Patterns of adoption are more illuminating.

“A lot of times, people will share their feelings with you as a researcher,” Suan says. “It’s up to us to peel the layers back and contextualize them to understand what’s really going on, then collaborate with our adoption partners to meet users’ needs.”

One example stands out. With early-career employees, we noticed a distinct arc in Copilot adoption behavior: initial delight and experimentation in the first three weeks of usage, a dip in enthusiasm from weeks three through 10, and then more consistent usage around the 11-week mark. Thanks to this insight, we can provide a consistent, predictable picture of how employees adopt Copilot over time and introduce change management interventions like skilling and reminders that can potentially address that midstream dip.

Finally, it’s important to examine these trends data carefully. If the current adoption for Copilot in Teams is 75%, we can ask why the other 25% of potential users are less engaged. Is it because they don’t use Teams regularly? Is there a language issue? Is it simply a lack of knowledge about how to use the tool?

When employees say they don’t see the value of Copilot for their work, we have a deeper conversation with those respondents to understand their challenges. In many of these follow-ups, we’ve discovered that the difficulty isn’t so much about the tool’s capabilities, but about not having the knowledge and education they need to realize its value.

“A key learning was that a certain percent of folks said they struggled to find time to learn,” Heath says. “So, the change strategy needs to not only understand how to land copilot but also create a better environment for learning and encourage leaders to do so. Learning days, group learning, and gamification are all tactics you can use for this.”

This kind of information has been extremely helpful for our change leaders, who use it to tailor their adoption efforts.

Like technology adoption itself, measuring change never stops maturing. As we continue to track our internal Copilot adoption journey and usage, we’ll keep digging down into new data and asking more directive, deeper questions.

“This is a fundamental way for us to stay in front of adoption and shape our strategy,” says Stephan Kerametlian, director of employee experience in Microsoft Digital. “When we see what the data’s telling us, the trends that emerge, and how our employees feel about Copilot, it puts us in a position to shape both our technology and how we implement it—with people at the center.”

Here are some suggestions for measuring the impact of Copilot at your company:

• Don’t let your Objectives and Key Results lifecycles run too long because things will evolve fast. Use an OKR while it’s useful, and then develop more nuanced metrics for the next cycle.
• Get a firm understanding of how employees are using core Microsoft products first, for example Word and Teams. This will establish product-level benchmarks, and then you can layer Copilot telemetry on top.
• Don’t be discouraged when certain paths of inquiry don’t turn out. Negative results are part of the research process.
• Make sure employee metadata is well structured so you can partition and measure your workforce effectively.
• Let the data lead your questions. Examples include use versus non-use, where the usage occurs, and employee satisfaction. Establish the data, then ask why it looks the way it does.
• Focus efforts on the outcomes you want. Identify benefits employees demonstrably crave and what’s holding them back, then build adoption efforts around those.

Ready to help your employees accomplish more with Microsoft 365 Copilot? Get started here.

• Learn how AI is already changing work—including here at Microsoft.
• Read the Microsoft 365 Copilot deployment and adoption guide we put together based on our internal experience here at Microsoft.
• Check out our Microsoft 365 Copilot deployment and adoption guide for executives.
• Learn how we’re powering a generational shift in IT at Microsoft with AI.

• Want more information? Email us and include a link to this story and we’ll get back to you.
• Please share your feedback with us—take our survey and let us know what kind of content is most useful to you.

The post Measuring the success of our Microsoft 365 Copilot rollout at Microsoft appeared first on Inside Track Blog.

Nobody helps employees adopt a new tool like their peers—the people who know their responsibilities, their organizational culture, and their day-to-day tasks.

In the initial stages of our Microsoft 365 Copilot adoption, a technology that brings many new possibilities and ways of working to the table, we knew we needed an unconventional approach to change management. At Microsoft Digital, the company’s IT organization, we understand that peer-to-peer support is one of the most powerful levers for driving excitement, engagement, education, and action for employees.

We turned to a grassroots approach to adoption support: The Copilot Champs Community.

New technology needs a new approach to adoption

As we began our Microsoft 365 Copilot rollout, it quickly became apparent that a community-based approach would be particularly powerful for this technology. That’s because individual people use Copilot in ways that are highly specific to their work and organization.

So, although Microsoft Digital has an extremely robust team of change management professionals, we could never hope to address the needs and culture of every organization across the company. We also recognized that excitement for Copilot was leading many of our employees to dive in headfirst and become early super-users when it came to applying this tool to their roles.

“The way to get people to really adopt a change is to make it relevant to their roles, and you achieve that by showing them the value of that change,” says Lisa Fryc, director of business program management on Microsoft Digital’s readiness team. “Peer-to-peer adoption efforts create a trusted environment where people can get the best information from their own community members.”

From a centralized change management perspective, our role at Microsoft Digital becomes providing the “one front door” to the assets and resources employees need, then relying on community support to direct employees to the material most relevant to their work. The result is a hub-and-spoke model that involves both core and local change management teams working to empower the Copilot Champs Community.

The internal benefits of our Copilot Champs Community

“People learn better peer-to-peer,” says Cadie Kneip, readiness business program manager responsible for the Copilot Champs Community on Microsoft’s Digital team. “Everyone has their own learning style, so our job is to develop training material and package it in a way that peer leaders can grab hold of and customize for their specific audience.”

Building a community that drives change

Although Microsoft has always encouraged culturally driven change initiatives, we’ve never run a champs program of quite this scale or sophistication. Microsoft 365 Copilot called for all-in peer-to-peer adoption efforts with a structured, targeted approach.

“We have an internal culture of peer-to-peer knowledge generosity,” Kneip says. “You can always find someone in a specific area of the world or organization who can touch on their colleagues’ concerns—it’s just a matter of bringing them on board.”

Microsoft Digital’s Employee Experience Success team developed a strategy for building our Copilot Champs Community, then nurturing champs as adoption leaders. It started with establishing the community, a process that continues as the team grows.

• Recruiting: We seek out and reward enthusiasm and provide a relatively open door to our Champs Community. That way, we can find enthusiastic participants who can appeal to all audiences.
• Welcome communications: We reach out to our prospective champs through Teams or email with a message welcoming them to the team, links to key resources including our baseline Copilot training, and opportunities to engage immediately.
• Branding: After our champs complete their required training and onboarding, they receive a Copilot Champs Community badge that serves two functions: signaling to their peers that they’re a source of guidance for Copilot and motivating the champs themselves through recognition.
• Kick-off call: Calls for new members help set expectations, foster excitement, and familiarize champs with their Microsoft Digital point of contact.
• Baseline poll: We try to establish a baseline for our champs’ existing Copilot familiarity, their eagerness to participate, and the assets that will be most useful for them.

These elements provided a steady influx of enthusiastic champs. From there, it was up to community leadership to make sure they were ready to support their peers.

“There’s more to managing a champs community than sending out a newsletter,” says Chad Hawthorne, director of business program management for Digital Experience, Readiness, and Change Management. “We recognized that, despite their deep knowledge, our Copilot super-users aren’t professional trainers, so we had to develop ways to give them assets and coaching to deliver effective peer-to-peer adoption.”

Our Champs Community nurture path includes the following elements:

• Viva Engage: A Viva Engage community created specifically for champs is our forum for posting announcements, encouraging discussions, and showcasing wins.
• Teams channel: We manage housekeeping, access to group files, contact with subject matter experts, and private discussions through an exclusive Teams channel.
• Train-the-trainer sessions: These live training sessions are crucial for enabling champs to host adoption activations such as lunch-and-learn or power hour trainings for their own teams.
• Community calls: A regular cadence of community calls lets our core adoption team showcase new features, go over housekeeping items, celebrate wins, share best practices, and generally keep the excitement and engagement rolling.
• Shout-outs: We use corporate communications channels to highlight champ stories that we can share as blog posts or leaders can use to give kudos to teams.

Setting champs free to lead

We discovered that a properly established and nurtured Copilot Champs Community quickly produces highly effective peer leaders.

Yen Anderson and Brian Shaw are two of our almost 4,500 champs and counting. Both started their Microsoft 365 Copilot journey as eager early adopters, and the community has given them a forum to share their expertise.

“As soon as I knew Copilot would become a broadly available enterprise tool, I knew the future of work would be changing,” says Anderson, Copilot champ and senior customer success account manager for Azure and AI. “And because I ramped up early, I was already demoing it to my team so they could see its value.”

Much of our champs’ activities take place in demonstration sessions. Shaw typically conducts 1–2 of these per week. Champs are also active in localized or organizational Viva Engage groups, which provide the perfect forum for ongoing conversations, showcasing learning, and community building.

“I love seeing people just light up,” says Shaw, Copilot champ and principal customer success account manager for RCG. “The technology is so impactful that people understand how it can change their work as soon as they see a demonstration.”

After a champ-led session, there’s a valuable opportunity for follow-up and relationship building. That only reinforces our champs’ position as peer leaders and makes them ideal support for their colleagues, further unburdening our centralized change management professionals.

Anderson and Shaw’s experiences are characteristic of champs across the community. Their expertise and innovation are as diverse as their roles.

Role
Senior customer success account manager, Azure and AI

Copilot expertise
Yen is a master of the art of the prompt. She’s honed her skills to develop prompts that save time on organizational tasks, boost productivity, and enhance creativity.

An innovation she’s shared
Yen specializes in helping peers craft the perfect prompts to meet their needs. With a keen interest in the mechanics of prompting, she helps her colleagues get on the path to advanced AI assistance by starting small, then introducing greater and greater specificity to their Copilot instructions.

“I love showing people how transformative generative AI tools can be,” Anderson says.

Role
Senior product manager, Microsoft Digital

Copilot expertise
Arias is committed to the ethical use of Copilot while using it actively in his own product management work. He takes an academic stance toward Copilot, with its capabilities at the center of his ongoing doctoral thesis.

An innovation he’s shared
Arias demonstrates ways to refine prompts in tandem with the principles of product management as expressed through the PMI framework. He encourages peers to use persona-based, goal-oriented prompts to put Copilot in the human-like headspace it needs to garner the results they want.

“Copilot helps me craft well-structured sentences for my communications, ensuring clarity and efficacy,” Arias says.

Roles
Business program manager, Greater China Region

Copilot expertise
As a business program manager, Cheng is responsible for driving adoption. She specializes in providing trainings from the beginner to advanced levels on using Copilot in Excel and Word.

An innovation she’s shared
Cheng is both an expert in using Copilot and a change management professional, so she’s pioneered ways to demonstrate the value of Copilot to her audience. Key strategies include real-time co-navigation of processes and using Copilot in Bing to drive self-learning.

“Find real business scenarios that your audience is interested in to attract users to try Copilot and keep using it consistently,” Cheng says.

Roles
Members of the Engineering, Data, Operations, and Tools (EDOT) Business Partners and Planning team

Copilot expertise
Harris and Wilson’s work often requires collaboration with remote team members, which they’ve learned to automate through Copilot.

An innovation they’ve shared
Together, Harris and Wilson have developed a simple IP kit that lets peers run Copilot-fueled collaboration sessions to create plans that include action items and owners.

“We love showing people how Copilot can supercharge collaboration and help unleash collective potential, empowering teams to achieve more together,” Wilson says.

For them, it’s all about enabling our employees to collaborate in more powerful ways.

“There’s a ton of potential for employees to use Copilot to work better together,” Harris says.

He called out how they’re already using it this way in many Microsoft 365 applications.

“It’s accelerating joint brainstorming, idea affinization, and action planning through tools like Microsoft Whiteboard and Loop, which foster creativity and efficiency,” he says.

Role
Principal customer success account manager, RCG

Copilot expertise
As someone with an engineering background, Shaw is intimately familiar with the ways Copilot in Microsoft Power Platform can drastically increase efficiency for developers through low-code/no-code app creation. In fact, he helped develop the automated onboarding process for the growing Copilot Champs Community by using Power Automate.

An innovation he’s shared
Shaw has become an expert in helping users navigate the multiple modalities of accessing Copilot across their day-to-day workflows: App-specific, Copilot in Bing, and interfacial elements that emerge in the flow of work. He opens the black box of behind-the-scenes AI operations to help people bridge what they’re trying to accomplish with how to access it via Copilot.

“Copilot is one of the only tools out there that will teach you how to use it. You just have to find the right questions to ask,” Shaw says.

Empowering leaders, enabling employees

This approach to driving adoption has produced outstanding results so far. With almost 4,500 champs and counting, the community has become a self-perpetuating engine of growth and empowerment. The Champs Community is also a valuable vehicle for sharing product feedback and helping the Microsoft 365 Copilot product group improve the tool itself.

“Change management is a critical role within large enterprises as many companies are realizing more and more when attempting to improve or evolve business processes and IT systems that directly benefit users,” says Andrew Osten, general manager of business operations and programs for Employee Experience Success. “What our team has done is invest time and talent into crafting an operationalized adoption program with many proven techniques that can be leveraged to accelerate both end user and company ROI from business and IT investments.”

The Copilot Champs Community is continually developing new and ingenious workflows for using Copilot in everyday work. They include building finance reports, creating videos and decks by combining textual and visual generative AI elements, ideation assistance, and other role-based use cases—all of which blossom out into greater productivity and creativity for employees across Microsoft.

“In every champ session our community runs, it takes just one magical use case to turn a skeptic into a champion or power user,” Kneip says. “It’s that easy to flip the switch, and it’s that moment we capture with a well-structured Champs Community.”

If you want to implement a Copilot Champs Community in your organization, use this resource to plan and implement your peer-to-peer adoption efforts.

Here are some tips for getting started with a champs community at your company:

• Use immediate incentives to help people work toward mastery. Start with small wins that progress toward greater proficiency.
• Let champs tailor the experience for their own peers. That includes subdividing champ groups according to organizational needs.
• Start with surveys about utilization. Find the people who are most engaged and give them opportunities.
• When you’re cultivating early adopters, look for the generalists—people who know a little about a lot of different areas.
• Start with community channels like Teams and Viva Engage. Viva Engage is especially valuable because it encourages ongoing conversations, engagement, and communities of practice.
• Encourage communities to keep track of time savings and other ROI metrics.

New to Microsoft 365 Copilot? Try it out at your organization today.

• Learn how we’re deploying Microsoft 365 Copilot with the help of—you guessed it—Copilot.
• Check out how we’re deploying Microsoft 365 Copilot internally at Microsoft.
• Find out how we’re using Microsoft 365 to enable our employees to collaborate in better ways.

• Want more information? Email us and include a link to this story and we’ll get back to you.
• Please share your feedback with us—take our survey and let us know what kind of content is most useful to you.

The post Driving Microsoft 365 Copilot adoption with our Copilot Champs Community appeared first on Inside Track Blog.

Our digital transformation has been a twofold journey.

First, we upgraded our core processes, providing efficient and effective self-service portals for our employees and powerful tools for our HR team using SAP SuccessFactors. Those processes include the nuts-and-bolts applications associated with human capital management (HCM): the employee portal, rewards, payroll, and other essential HR functions.

With the core processes in place, our Microsoft Digital Employee Experience (MDEE) team had everything they needed to revolutionize the data at the center of HR.

The architecture we chose? Microsoft Azure Data Lake.

[Explore all the ways that AI is driving Microsoft’s digital transformation. Learn how Microsoft is creating the digital workplace.]

Building a modernized HR data estate

When data is scattered across disparate systems, it’s difficult to provide agility, insights, and advanced analytics through AI. In today’s world of big data and predictive intelligence, these capabilities aren’t just a luxury. They drive talent conversations, workforce planning, and an improved employee experience that affects business outcomes.

But when an enterprise’s data is siloed or fragmented, those outcomes are out of reach.

“What happens when you don’t have a modern data architecture?” asks Harsh Raj Singh Thakur, principal software engineering manager on the MDEE HR Data and Insights team. “You have a tedious and drawn-out process before you can retrieve your metrics. It’s a cumbersome task, it’s expensive, it’s not easy to maintain, and there’s a lot of cost to get it all done.”

To make HR insights more accessible and insightful, MDEE first had to assemble a unified and accessible data estate. Our SAP SuccessFactors implementation for core HR processes helped lay the groundwork by streamlining external and operational data to make them more organized and available for processing.

With modern core processes in place, MDEE engineers could turn their attention to data.

The journey to data transformation

Like all large-scale transformations, this one involved a great deal of complexity and multiple touchpoints. Microsoft Azure Data Lake provided the modern analytics platform that would not only enable the team to ingest, store, transform, and analyze the data, but also deliver simpler data discoverability, maintain data security, and ensure compliance.

Unifying the data

Considering the wide array of HR systems at Microsoft, it was important to bring all the data together to give HR an end-to-end view of the employee lifecycle and the moments that matter in an employees’ journey. At the same time, the team took efforts to reduce redundant data copies across the enterprise.

The ease of use from actually having everything collocated in an Azure Data Lake makes it easy to build out connected insights. It’s the foundation of our modernization journey.

—Harsh Raj Singh Thakur, principal software engineering manager, Microsoft Digital Employee Experience HR Data and Insights

“Enabling connected insights which are trusted and secure through a modern data platform in Azure Cloud was a key goal as we set out to drive the digital data transformation in the HR ecosystem,” says Johnson Samuel, principal group engineering manager for MDEE’s HR Data and Insights team.

Multiple systems make up the HR ecosystem: Employee Central for core HR, iCIMS for applicant tracking, listening systems, rewards, CRM, employee learning, and more. While each of these systems serves an important purpose, the potential to unlock insights by unifying all of their data is immense.

“The ease of use from actually having everything collocated in an Azure Data Lake makes it easy to build out connected insights.” Raj Singh Thakur says. “It’s the foundation of our modernization journey.”

Azure Data Lake Storage Gen2 serves as the common storage layer, which ingests data through Azure Data Factory, messaging systems, and other sources. By properly defining storage structures and models, the team had made the first step toward a more modern data platform.

Expanding the data footprint with new metrics and scorecards

Ever-increasing volumes of data illustrated the need for advanced analytics. They were no longer a choice—they were a necessity.

“There are many lines of businesses within HR, like Global Talent Acquisition, Talent and Learning, and HR Services who manage HR operations,” Samuel says. “We’ve enabled new capabilities for each of these different HR functions.”

Key metrics across the ecosystem include the recruiting funnel, workforce, headcount, employee engagement, learning and development, and other functions across HR. The analytics apparatus uses a combination of Azure Synapse Analytics, Azure Analysis Services, and Power BI Shared Datasets, while Microsoft Power BI is responsible for visualization.

This powerful combination of technologies helped build complex analytics and drove consistency across teams. It also unlocked the ability to bring disparate metrics together to help determine correlation and causation between different factors.

Data governance

Next, the team needed to ensure that engineers and end users could access data in the lake safely and securely. Good governance keeps data access compliant because users can only request information that’s relevant to their roles. Driven by the HR Privacy team and enabled by a home-grown security and governance platform, MDEE established column-level security (CLS) on the Data Lake.

“When an HR team requests data, they get access to only the specific data set,” Raj Singh Thakur says. “So if you’re looking for an employee’s name and alias but your role doesn’t require you to know their salary, gender, or other aspects of their identity, you won’t get access.”

This approach makes sure we respect our employees’ privacy and that we comply with local laws that regulate how we use our data. Data governance also includes data discoverability, quality, and lineage functionality, which the team established through Microsoft Purview and in-house solutions to support more complex scenarios.

Modern engineering

MDEE also developed key platform capabilities that ensure high-quality and trustworthy data across the estate and drive engineering efficiency.

Whether the metric is headcount, performance management, employee learning, or any other area, each of them follows the architectural pattern of a Data Lakehouse, a system where all information resides in the Data Lake, without the need to build separate data marts. It allows our engineers to scale storage and compute independently for greater efficiency.

Between telemetry dashboards that help engineers understand system health and continuous optimization across code and infrastructure, this new architecture has helped save significant Azure costs—a reduction of around 50% over 2 years. Meanwhile, enabling agile development and DevOps is helping the team deliver iteratively and realize business value faster.

But the real value lies in the insights that unified, normalized data empowers.

“We’ve normalized the data by leveraging a company-wide taxonomy that we can use across other projects very easily,” says Mithun Manganahalli Goud, principal software engineer on MDEE’s HR Data and Insights team. “So from a data-delivery service standpoint, we can provide information to a wide range of downstream systems and data consumers.”

Building a platform for the future

While the new architecture is actively meeting current reporting needs, MDEE also looked toward the future.

We’ve created a rich content system where we can manage emerging requirements with the current data and metadata, so it’s future-ready. We already have the process in place, so we won’t have to go back and reinvent the wheel.

—Mithun Manganahalli Goud, principal software engineer, Microsoft Digital Employee Experience Data and People Analytics

The platform is capable of enabling deep insights that leverage machine learning. While today’s focus is on descriptive and diagnostic functions, the team is working toward predictive and prescriptive analytics through AI and machine learning.

“We’ve created a rich content system where we can manage emerging requirements with the current data and metadata, so it’s future-ready,” Manganahalli Goud says. “We already have the process in place, so we won’t have to go back and reinvent the wheel.”

When our HR team takes the next step into AI-driven insights, the foundations will already be in place.

Driving human-centered innovation with Microsoft Azure Data Lake

Our modernized data architecture has enhanced the HR teams’ capabilities. Better data immediacy means data pulls that used to take 24 hours now get done in a fraction of the time—around four to six hours. Similarly, the time it takes to enable self-service access for bring-your-own-compute data processing is rapidly falling.

One of the most unique and forward-thinking outcomes is that we’ve been able to combine qualitative with quantitative data. We’re able to create data models with our survey information as well as more quantitative data like attrition and diversity, then combine them in an aggregated, de-identified way to understand broad insights.

—Dawn Klinghoffer, vice president, People Analytics

But the most powerful outcomes are the cross-category, cross-disciplinary insights that unified and accessible data provides for HR leaders.

“One of the most unique and forward-thinking outcomes is that we’ve been able to combine qualitative with quantitative data,” says Dawn Klinghoffer, vice president of People Analytics at Microsoft. “We’re able to create data models with our survey information as well as more quantitative data like attrition and diversity, then combine them in an aggregated, de-identified way to understand broad insights.”

The more people interact with the data, the more it will lead to deeper questions and better insights to drive their business or Microsoft as a whole.

—Patrice Pelland, partner group engineering director, Microsoft Digital Employee Experience

For example, by combining sentiment data with de-identified calendar and email metadata, we’ve been able to quantify the impact of blocking focus time on employees’ perception of work-life balance.

“Making data available to all people in a self-service, consumable way gives them the opportunity to ask the questions they don’t even know they have,” says Patrice Pelland, partner group engineering director for MDEE. “The more people interact with the data, the more it will lead to deeper questions and better insights to drive their business or Microsoft as a whole.”

Those questions and insights have already led to human-centered improvements and innovations. One example is the wide adoption of team agreements that empower employees to collectively self-determine the work modes that serve them best. HR’s work has even informed some of the “nudge” product features for employee experience tools like Microsoft Viva, for instance, recommending focus blocks to improve productivity and overall work-life balance—a metric that’s currently on the rise across Microsoft.

Ultimately, the more people who have access to high-quality, trustworthy data, the more we can provide a world-class experience for all employees.

“There’s a lot of envisioning based on the services that we’ve been building that people didn’t even think could exist,” Pelland says. “We’re building the foundational layers to offer things that will be truly transformational for the HR business. Whatever size your organization is, and whichever HCM you use, with Azure, you can do what we’re doing right now.”

• The gold standard should be unity between transactional tools and data tools.
• Start from an understanding that it’s about people and ground your work in that.
• Think big but think holistically; start with a goal and work toward it iteratively.
• Consider the experiences that will delight your end users.
• Start from how you’re going to use the data, then work backward.
• Collaborate early and often. Otherwise, preconceived notions can creep in.

• Explore all the ways that AI is driving Microsoft’s digital transformation.
• Learn how Microsoft is creating the digital workplace.

Want more information? Email us and include a link to this story and we’ll get back to you.

Please share your feedback with us—take our survey and let us know what kind of content is most useful to you.

The post Enabling advanced HR analytics and AI with Microsoft Azure Data Lake appeared first on Inside Track Blog.

The pandemic has changed the way we work—and with it, the way we eat. Consumer expectations are evolving as people, including employees at Microsoft, get used to seamless, on-demand ordering through mobile applications and direct-to-you deliveries.

With Microsoft employees returning to the office after two years of working mostly remote, the Dining at Microsoft Operations team for the Puget Sound campus knew they needed to help them feel comfortable, fuel up, and be productive at work. They wanted to integrate new programs and ordering capabilities that would respect changing expectations and incorporate the intuitive nature of the mobile apps employees use at home.

Between Dining Operations and their partners on the Microsoft Digital team, plans were already in place to streamline the mobile ordering experience. The imminent return to work and the transition to a flexible, hybrid environment confirmed the urgency of the transformation. What began as an initiative to provide premium value for employees has become a necessary service consideration.

[See how Microsoft employees navigate their campuses with IoT tech and indoor mapping. Learn about the ways that Microsoft is modernizing the support experience with ServiceNow. Find out about reinventing the employee experience at Microsoft.]

A campus-scale challenge

With more than 70 dining locations on the Puget Sound campus and thousands of frequently changing menu items, Dining Operations’ challenge was to provide an experience that would feel intuitive for users and meet the needs of tens of thousands of diners each day. It would need to provide seamless food ordering and reflect the unfolding reality of hybrid work.

We recognize that from an employee experience standpoint, Dining at Microsoft needs to reflect our cultural principles, whether that’s around sustainability, accessibility, diversity and inclusion, or digital transformation. We have a job to do, and that’s to make sure we create a great dining experience for our employees every day and have our food and programming reflect who we are as an organization.

—Jodi Westwater, senior services manager of Dining Operations for the Puget Sound Campus

While online ordering was already available to employees through a browser-based point-of-sale (POS) platform, Dining Operations wanted a modern, intuitive, mobile-first experience to streamline how people browse menus and purchase items. They also wanted to integrate it into the digital environment employees use every day.

It was imperative that any solution should embody Microsoft’s priorities and culture.

“We recognize that from an employee experience standpoint, Dining at Microsoft needs to reflect our cultural principles, whether that’s around sustainability, accessibility, diversity and inclusion, or digital transformation,” says Jodi Westwater, senior services manager of Dining Operations for the Puget Sound Campus. “We have a job to do, and that’s to make sure we create a great dining experience for our employees every day and have our food and programming reflect who we are as an organization.”

Dining transformation, tailored to employees

The teams incorporated a mobile menu and ordering interface into an internal app that employee use to access transportation, explore their benefits, and manage other elements of their day-to-day roles. Incorporating dining into the app would mean that employees could order food in the mobile-friendly, full-service environment they already use.

To make the integration work, the team needed to bridge the gap between the internal mobile app and Dining Operations’ existing POS and menu tool. Since the POS system was originally intended as a standalone touch-screen service, the team used Microsoft Azure API to create the connective tissue between the platforms.

“One of the key focuses early on for building this integration was not only that the information be accurate for Microsoft end users,” says Thomas Po, a product manager on the Microsoft Digital team. “It also had to be relatively easy to use on the back end to minimize room for error and stay in sync with the operations side.”

POS integration was only part of the challenge. To meet Microsoft’s commitment to accessibility, the team worked closely with internal stakeholders to review and implement the Microsoft Accessibility Standards (MAS). They conducted user-group testing with employee resource groups, individuals, and Microsoft Digital’s internal accessibility experts. As a result, the app features inclusive elements like high-visibility contrast settings and read-along technology.

Since the app would be handling financial transactions in conjunction with third-party tools, it needed to be highly secure. So Microsoft Digital worked closely with the Finance Security team to ensure that the app met the strict data-capture and retention requirements built into all Microsoft technology.

Throughout the process, they leveraged tools throughout the Azure stack, including Azure API for integration with the dining POS system and Cosmos DB as a data repository, as well as other third-party tools hosted on Microsoft Azure.

The new ordering experience rolled out as a pilot in April of 2021 for use by essential employees working onsite, and it’s now in place across the entire Puget Sound Campus. The app allows employees to browse menus that feature images of the food at any dining location. They can order their food, pay digitally, and pick it up at the café, food hall, or espresso location of their choice.

From an experience standpoint, everything we do, design, and ideate must be user-centric, which for us means employee-led. What do employees need? What do we anticipate their habits to be? How will preferences change in a hybrid workplace? And how do we meet and exceed those ever-changing expectations?

—Jodi Westwater, senior services manager of Dining Operations for the Puget Sound Campus

The app automatically finds the nearest dining location based on an employee’s current whereabouts. In addition, iOS users can complete their transactions through Apple Pay, adding an extra layer of seamlessness to the mobile experience.

Employees can even browse the week’s menu ahead of time. With an increasing emphasis on hybrid workplaces and flexible in-person attendance, they might decide to make the trip to the office when their favorite food is available!

The mobile app integration doesn’t just reflect the intuitive experience of mobile food ordering that employees have embraced during the pandemic. It provides a way for workers who are understandably anxious about public eating spaces with the opportunity to retrieve their food quickly and eat on their own terms. It’s also a quick and easy solution for employees who have back-to-back meetings and may only have a few moments to grab food.

“From an experience standpoint, everything we do, design, and ideate must be customer-centric, which for us means employee-led,” Westwater says. “What do employees need? What do we anticipate their habits to be? How will preferences change in a hybrid workplace? And how do we meet and exceed those ever-changing expectations?”

The future of fueling up at work

Online ordering has more than tripled since before the pandemic. Previously, employees placed less than two percent of orders at the Puget Sound campus online. Now, approximately ten to twelve percent are placed digitally—at least a quarter of those via the mobile app. To make the feature even more accessible, the team will make dining order-ahead capabilities available on Microsoft Viva Connections, which will enable employees to order food on their mobile or desktop, using the same Microsoft Teams interface that they use throughout their day.

For diners who prefer the in-person experience or who might be anxious about crowding as more people return to work, Dining Operations is exploring a system that provides employees with more information about which cafés are busiest and when. The tool will use a mix of colors and graphics to indicate dining location traffic and occupancy so people can decide where they’d like to eat. This new functionality will also give staff valuable insights into usage patterns so they can use data to accommodate the ebb and flow of diners throughout the day and reduce food waste by ordering stock to reflect usage patterns accurately.

“Everything we’re doing is designed to create the most convenient and intuitive experience for Microsoft employees, visitors, and guests,” Westwater says. “We’re not just making sure we offer great food onsite, but that the ordering and dining process is accessible, that it makes sense, and that it’s easy to access.”

• Meet users where they’re at: There’s no such thing as one-size-fits-all.
• Build the app around the behavior: The app won’t change how users want to interact, so think about how they would use it.
• Put on your user hat: Consider everything from the customer perspective.
• Leverage user-testing: Identify your critical misses.
• Start small: Work with pilots and see what sticks.
• Nothing is sacred: Embrace reprioritization, pivot, and adapt.

• Learn how Microsoft is reinventing its employee experience for a hybrid world.
• Read how Microsoft is modernizing its support experience with ServiceNow.
• Find out how Microsoft is using new IoT technology and indoor mapping to navigate its campuses.

The post Dining transformation at Microsoft eases the transition as employees return to work appeared first on Inside Track Blog.

That was the situation here at Microsoft until our IT teams started using Microsoft Azure Firewall Manager.

This platform helped streamline and centralize their control over our large firewall ecosystems. It also cleared the way for insights into the thousands of policies spread out across the company. We just needed the right tool to make it a reality.

To accomplish that goal, our Microsoft Azure Firewall product team developed the new Policy Analytics feature for Microsoft Azure Firewall Manager, now available to all customers in public preview. With Policy Analytics, cloud network engineers can identify and remedy rule-based vulnerabilities before they become liabilities.

[Learn about the ways that Microsoft Azure AD MFA enhances remote security at Microsoft. See how next-generation connectivity is transforming our enterprise network.]
For a transcript, please view the video on YouTube: https://www.youtube.com/watch?v=4EnLgGEstJw, select the “More actions” button (three dots icon) below the video, and then select “Show transcript.”

Streamlining and consolidating with Microsoft Azure Firewall Manager

This journey began roughly five years ago. The Microsoft Azure team recognized that maintaining multiple firewalls in isolation can be a headache.

First, there’s the need to manually update numerous individual firewalls. At the same time, threat definitions and newly discovered software weak points demand constant attention. That kind of maintenance requires immense effort from engineers.

The solution?

Virtualizing traditional hardware firewalls into a SaaS environment with Microsoft Azure Firewall and consolidating control over the entire ecosystem with Microsoft Azure Firewall Manager.

Being able to manage firewalls as a fleet and have the operational optics into the health and welfare of the firewalls, along with the ability to manage things from a holistic perspective, has tremendous benefits.

—Tom McCleery, principal engineering manager, Microsoft Digital Employee Experience

Since going live in 2019, Microsoft Azure Firewall Manager has drastically improved corporate network management both internally at Microsoft and for our enterprise customers.

There are several benefits for IT practitioners. First, it offers a central location where system engineers can update threat definitions then rapidly deploy them across the cloud. At the same time, a unified ecosystem allows for mass creation and distribution of rules and policies.

“Being able to manage firewalls as a fleet and have the operational optics into the health and welfare of the firewalls, along with the ability to manage things from a holistic perspective, has tremendous benefits,” says Tom McCleery, principal engineering manager with Microsoft Digital Employee Experience (MDEE).

Teams can also customize firewall rules for specific regions by defining policies that inherit characteristics from a baseline. This helps enforce a centrally defined policy while providing flexibility for administrators to customize firewall rules.

“The main benefit for me as an engineer is time,” says Beth Garrison, principal cloud network engineer for MDEE. “I spend less time making these small changes and more time focusing on improving our network service. That’s more time focusing on complex network problems instead of managing individual firewall updates.”

The need for an all-up policy analytics solution

Microsoft Azure Firewall Manager’s successful internal implementation significantly improved day-to-day operations for our cloud network engineers. But an organization’s network security policies constantly evolve to keep pace with workloads. Over time, network and application rules change and can lose their efficacy, impacting the firewall’s performance and security.

For example, applications might migrate to a new network, but rules referencing the former network remain. Or teams could unknowingly duplicate rules throughout a policy hierarchy.

At an organization like Microsoft with more than 80 firewalls in operation, those kinds of problems can scale rapidly. As a result, MDEE engineers need to keep a close eye on policies.

Policy analytics was one of the most sought-after features in Microsoft Azure Firewall Manager. What out-of-the-box experience could give us a sense of what happens with our policies or how we’re using rules? How can I improve my security posture or the performance of the firewall?

—Mark Gakman, senior product manager, Microsoft Azure Firewall product team

“Policy management is a very process-heavy operation in general,” McCleery says. “Across the several thousand virtual networks we oversee, managing rules is our top volume.”

To streamline MDEE’s efforts and combat vulnerabilities, the logical next step was providing visibility into policy management over time to generate actionable insights.

“Policy analytics was one of the most sought-after features in Microsoft Azure Firewall Manager,” says Mark Gakman, senior product manager on the Azure Firewall product team. “What out-of-the-box experience could give us a sense of what happens with our policies or how we’re using rules? How can I improve my security posture or the performance of the firewall?”

A collaboration between Microsoft Azure engineers and MDEE led to Policy Analytics for Microsoft Azure Firewall Manager.

Policy Analytics with Microsoft Azure Firewall Manager

Our Policy Analytics feature focuses on providing oversight on all rules in operation across an enterprise’s entire firewall ecosystem.

Four key Policy Analytics features deliver insights for network engineers:

• Firewall flow logs display the traffic flowing through Microsoft Azure Firewall, hit rates, and network and application rule matches. This view helps identify top flows across all rules, filtered by specific sources, destinations, ports, and protocols.
• Rule analytics present traffic flows mapped to destination network address translation (DNAT), network, and application rules. This provides enhanced visibility into all flows matching a rule over time. As a result, users can analyze rules across both parent and child policies.
• The policy insight panel aggregates insights and highlights recommendations to optimize Microsoft Azure Firewall policies.
• Single-rule analysis analyzes traffic flows matching the selected rule, then recommends optimizations based on intelligent insights.

These features help MDEE cloud network engineers identify patterns associated with different kinds of vulnerabilities including fat flows, top talkers, underutilized rules, and duplicate policies.

For the MDEE engineers supporting our internal corporate network, the top priority was eliminating duplicate rules. These introduce risk into networks by creating backdoor entry points, which complicate rule management and slow firewall performance among other problems.

We get weekly tickets saying users can’t connect from a particular source to a particular destination. But we just type in a source IP and we can see what’s happening at a pretty high level—very quickly. So from an on-call perspective for direct-response individuals like me, it’s been a huge help.

—Beth Garrison, principal cloud network engineer, Microsoft Digital Employee Experience

With the added visibility that Policy Analytics provides, the team managed to discover 1,400 duplicate rules and eliminate more than 1,200 of them. Clearing these duplicates has both significantly improved our security posture and paved the way for automated ACL updates to run more smoothly.

An improved security posture is a massive win for our corporate networks as a whole. But for Garrison and her team of cloud network engineers, the biggest impact has been time savings, especially in their troubleshooting work.

“We get weekly tickets saying users can’t connect from a particular source to a particular destination,” Garrison says. “But we just type in a source IP and we can see what’s happening at a pretty high level very quickly. So from an on-call perspective for direct-response individuals like me, it’s been a huge help.”

By Garrison’s estimate, analytics queries that used to take five or 10 minutes now clock in at around 30 seconds. Those time savings translate to better service and more flexibility for her team.

The emerging possibilities of Policy Analytics

Policy Analytics for Microsoft Azure Firewall Manager is currently in public preview. Even at this early stage, the response from customers has been incredible.

“This solves a big pain point for large organizations with tens or hundreds of firewall deployments,” Gakman says. “After only six months in preview, more than 1,000 enterprise customers have activated Policy Analytics. From the conversations I’m having, the demand for these capabilities is strong.”

The team continues to add more analysis and features as Policy Analytics matures. One of the most exciting developments is the ongoing growth of intelligent recommendations for single-rule analysis.

That kind of support is especially helpful for organizations who don’t have cloud network engineers in their IT organizations. By following AI-driven, automated recommendations when a user zooms in on a particular rule, even teams who lack network expertise will be able to increase their security posture.

For our support teams and our customers’ IT professionals, Policy Analytics for Microsoft Azure Firewall Manager is one more step toward a truly cloud-driven business world.

“It’s a feature, but it’s really the underpinning for a whole discipline within my team,” McCleery says. “Our biggest goal is helping people and processes work at the pace of the cloud.”

• The tech is the easy part: Focus on people and process as you’re developing solutions.
• If it’s not measured, it’s not valued. Do everything you can to get the data on the table.
• Deploy to your most underutilized firewalls first to build confidence and comfort.
• Get in the habit of looking into your rules periodically and adjustments will become simpler over time.

• Watch how Microsoft’s IT teams manage our corporate firewalls with Microsoft Azure Firewall Manager.
• Learn about the ways that Microsoft Azure AD MFA enhances remote security at Microsoft.
• See how next-generation connectivity is transforming our enterprise network.

The post Enhancing Microsoft’s security posture with Microsoft Azure Firewall Manager appeared first on Inside Track Blog.

Our Software Licensing Service (SLS) is the Microsoft Digital Employee Experience team responsible for software asset management.

“When you think about software asset management, you want to track a license’s lifecycle from requisition through allocation and deployment,” says Patrick Graff, the senior service engineer leading the SLS team. “Then you need to maintain it until you can reclaim it at the end-of-life stage.”

If individual departments and employees manage their own software licenses, organizations are open to all kinds of inefficiencies and risks—not to mention a subpar experience for employees who need access to third-party software. Manually tracking the number of licenses enterprise-wide, total spend, and overall software entitlements is incredibly labor-intensive.

Under those fragmented circumstances, how can a procurement department know the total entitlements from a particular vendor? How do they ensure their company maintains the proper enterprise licensing position at scale? Do they know how many unallocated licenses are available? And how do they manage reclamation, reallocation, and rightsizing during renewal and true-up cycles?

SLS wanted an enterprise-level platform that would both streamline the employee experience and optimize license management. So they partnered with Microsoft Procurement and Infrastructure Engineering Services (IES) to implement a unified software catalog.

[Learn about streamlining vendor assessment with ServiceNow VRM at Microsoft. See how Microsoft is Modernizing the support experience with ServiceNow. Read about instrumenting ServiceNow with Azure Monitor.]

A strategic partnership drives software licensing excellence

Since 2015, we’ve used ServiceNow to automate our helpdesk support process. When they introduced their Software Asset Management (SAM) module, it was a natural fit for implementing a centralized software catalog.

“The scope of work and the objective resonated with SLS because they understood the pain points of using disconnected tools,” says Sherif Mazhar, principal product manager for the IES team partnered with SLS. “They were interested in consolidating those tools and gaining the ability to track license usage accurately.”

We started seeing better control, better insights, better reporting, and also better visibility into unused licenses and how we could reassign them.

—Sherif Mazhar, principal product manager, Infrastructure Engineering Services

Thanks to collaboration with ServiceNow engineering teams, ServiceNow features well-tuned, out-of-the-box compatibility with Microsoft technologies. They also maintain an enterprise roadmap to streamline large integrations.

The teams started small with a single license portfolio: Adobe Creative Cloud. Tools like Adobe Photoshop and Premiere Pro are essential for creatives, but many groups had purchased their own licenses on an ad-hoc basis.

Once the Adobe licenses were consolidated into ServiceNow SAM, the team saw rapid results. “The value realization was quick with Adobe,” Mazhar says. “We started seeing better control, better insights, better reporting, and also better visibility into unused licenses and how we could reassign them.”

Within 12 months, the team had cut back excess licenses across Microsoft, resulting in significant savings. With such a successful pilot already showing results, SLS decided to move forward with a more universal ServiceNow SAM implementation.

Implementing ServiceNow Software Asset Management across Microsoft

ServiceNow features several out-of-the box enterprise integrations, but the work of developing one process for license management required extensive collaboration between SLS and IES.

Several different technologies needed to come together to facilitate a unified experience:

• Microsoft System Center Configuration Manager (SCCM) connectors provide one-directional imports into ServiceNow, bringing relevant data into the ServiceNow instance from an SQL Server database and mapping it to ServiceNow’s SAM database.
• Microsoft SharePoint grants automatically provisioned access to relevant download files once the software is allocated to the end user.
• Microsoft Azure Active Directory (AAD) handles identity and access management for software acquisition, enabling single-sign-on (SSO) and multi-factor authentication (MFA) capabilities for cloud-based and SaaS tools.
• A Microsoft Teams integration for the ServiceNow Virtual Agent helps employees troubleshoot and seek support via chat within a Teams App.

Once the ServiceNow implementation was complete, the team needed to loop the whole project into the existing employee workflow by connecting it with internal procurement and IT portals. SLS ensured that employees felt at home in the new experience by unifying the catalog’s color coding and UI with the portals employees already know how to navigate.

The result is a streamlined experience for employees and a management environment that delivers optimization and compliance.

A transformative third-party software licensing experience

When one of our employees wants access to third-party software, they log in to the IT or procurement portal of their choice and navigate to the Unified Software Catalog in ServiceNow. From there, they simply find the software tool they need and submit a request.

If a piece of software requires no extra permissions, the employee can simply requisition it. Otherwise, they fill out a request form, which initiates an automated workflow that manages permissions, their device’s operating system, relevant purchase orders and cost centers, and our entitlements within that software portfolio.

The real power of the tool is that we can set up configurable workflows for different types of products.

—Tony Bouker, senior product manager, Infrastructure Engineering Services

When the license allocation is complete, the end user gets an email with installation instructions. They can then proceed to an automatically provisioned SharePoint folder to download and install the software.

For SaaS tools and cloud-based suites like Adobe Creative Cloud, the team has created another way to access their software. The system adds the employee’s alias to an internal identity group, which grants access through SSO powered by AAD.

“The real power of the tool is that we can set up configurable workflows for different types of products,” says Tony Bouker, senior product manager with IES.

Efficiency, optimization, and compliance

Microsoft SLS has integrated the software requisitioning process into the Bing search engine. Now, employees can search software titles through Bing, which then points to the Unified Software Catalog.

Employees no longer have to conduct manual, online searches for third-party software or send emails asking for requisitions. Now they simply search in Bing or head directly into the Unfied Software Catalog and initiate an automated requisition workflow.

For SLS, the outcomes are about data-driven insights and license consolidation. The team can track Microsoft’s overall licensing position across all third-party software without the need for time-consuming detective work or manual uploads. When the time comes for renewals and true-ups, that visibility is essential.

It also mitigates risk through robust governance and policy by reducing vulnerabilities, data breaches, and license compliance violations.

On a more strategic level, the tool helps SLS optimize our software licensing frameworks for individual providers. For example, if one employee uses several tools within a provider’s toolkit, the team has the data it needs to decide whether it’s more efficient to allocate those licenses individually or as part of an “all-apps” subscription.

On the macro level, it gives us the ability to negotiate volume licenses more accurately, at exactly the level that fulfils our organizational needs. Good data drives informed decision making.

As those optimizations scale, Graff estimates that we’re saving an average of 10 percent across all of our enterprise license positions. For an organization the size of Microsoft, that represents cost savings in the millions.

Beyond Microsoft, this implementation is laying the groundwork for a wide-ranging change in how enterprises manage their third-party software. “If you look at the Microsoft presence in the market, every single customer who’s using our technologies leverages our endpoint management tools for asset management and license tracking,” Mazhar says. “So this will open the door for a lot of opportunity for Microsoft, for ServiceNow, and for our customers.”

• Start small with a targeted publisher and gain early wins to build confidence with stakeholders.
• Have a close relationship with your partner teams so you can recognize needs and grab opportunities.
• Build out your key process areas first, and identify workflow patterns you can reuse to scale your software asset management program.
• Establish policies to make sure the changes you put into effect have teeth.
• When working with a third-party partner, make sure you have the right connections to ensure you can provide feedback at the right level.
• Ensure leadership understands your priorities so they can manage those relationships at the highest level.

• Learn about streamlining vendor assessment with ServiceNow VRM at Microsoft.
• See how Microsoft is Modernizing the support experience with ServiceNow.
• Read about instrumenting ServiceNow with Azure Monitor.

The post Rethinking software licensing at Microsoft with ServiceNow Software Asset Management appeared first on Inside Track Blog.

That’s a lot of transacting.

Through smart data assembly, natural language processing, and a custom Microsoft Teams bot, the engineers on Microsoft’s Royalties Team have created Royalties Assurance as a Service (RaaS), the company’s new internal royalty transaction platform.

Our royalties system is dynamic and complicated—it processes multiple millions of transactions per day, dealing with hundreds of thousands of products, processing and calculating earned royalties according to the specific contracts in an accurate, timely, and compliant manner.

—Jagannathan Venkatesan, principal group engineering manager, Microsoft Global Payments and Cash

“It’s one thing to build an extensible and scalable system processing vast quantities of data spread across multiple dimensions and business rules to meet the accuracy, timeliness, and compliance needs of a multi-billion dollar business,” says Jagannathan Venkatesan, principal group engineering manager in Microsoft’s Global Payments and Cash organization. “It is an entirely different challenge to be able to reduce that complexity through building a fully connected data system to bring it to a single canvas that is easy to interact with. With RaaS, our Royalties team was able to do just that with the help of RaaS APIs that enable system-to-system integration—including Microsoft Teams integration—and human augmented exploratory analytics.”

Enterprise royalties, complex connections

Like many companies, Microsoft manages complex royalties relationships with other organizations. For example, the Microsoft Store sells Xbox games that leverage intellectual property from third party vendors. Or partners sell services through Microsoft that necessitate royalties based on consumption. In each of these cases, once an incoming transaction occurs and Microsoft has obtained the funds, a complex orchestration of calculations must take place to ensure each party receives the proper payments.

“Royalties payouts are a critical aspect of Microsoft business, enabling our global digital content partners to realize the value of the products they have onboarded onto the Microsoft ecosystem,” Venkatesan says. “Our royalties system is dynamic and complicated—it processes multiple millions of transactions per day, dealing with hundreds of thousands of products, processing and calculating earned royalties according to the specific contracts in an accurate, timely, and compliant manner.”

Gaining clarity throughout that entire system of relationships is essential for accuracy in accounting and payouts, and it is an integral part of generating organization-wide insights.

“When we generate a statement for a partner for a particular period, our system must be able to enable the business to walk back from the statement to products to transactions along with bringing appropriate contact and partner data including historical information,” Venkatesan says.

To achieve that level of clarity and trust in the system, the royalties team needed to aggregate the datasets underlying all of Microsoft’s royalties relationships and transactions, then make the results available in one easily accessible place.

“The challenge is the complexity around calculating payouts and retrieving that information,” says Ehsan Mehrabi, a senior finance manager on the Royalties team. “We need to make sure everything is correct before payments go out the door or transactions get their accounting treatment.”

The engineering team took up the challenge.

[Learn how Microsoft’s finance team uses anomaly detection and automation to transform royalty statements processing. Find out how Microsoft designed a modern data catalog to enable business insights. Explore how AI and chatbots simplify finance tools at Microsoft.]

Transforming Microsoft’s royalties ecosystem with a connected data system

Unifying the 300 to 400 million financial data points that flow in and out of the company each month was an enormous undertaking for the royalties engineering team.

It encapsulated three main challenges.

The first and most complicated task was defining and canonicalizing the millions of data points associated with the royalties business. The engineering team needed to use automation to identify distinct entities that could be assembled and visualized as a graph of connected data-points.

Defining words such as “contract” and “product” seems like a simple job, but it presents challenges when it comes to automating data in complex business relationships. The data definitions needed to reflect the royalties team’s business needs and be simultaneously consumable by data processing systems.

Sundeep Ratnagiri, engineering manager for Microsoft Royalties, outlines how the team defines these terms, explaining what the word “contract” means when it comes to managing Microsoft’s royalties system.

“For a businessperson, a contract is a legally binding document that defines business terms,” Ratnagiri says. “For an engineer, it is a set of parameters codified in the system to function the way the legal document is written. Similarly, a product is an asset that is transacted upon, with rich attributes that can be referenced in a contract.”

From the start, engineers partnered with their peers across the royalties business and engineering landscape, including the accounting, business, and partner engineering teams. They spoke to a wide array of stakeholders to ensure they could assemble the system’s 300-400 million data point connections per month in ways that would support everyday usage. The result was a single, connected data output with analytical (like aggregation, for example) capabilities powered by the team’s different processing calculation systems (also for example).

The second major task was to represent the different data sets in a connected graph exposed with a single API set, enabling team members to navigate from any point of the royalties system to anywhere else. The engineers utilized Apache Spark for the data modeling pipeline, then modelled it as a graph of connected entities using Microsoft Azure Cosmos DB. The result was a trustworthy, independently validated source for all canonical data that was ready for access and interpretation.

“The natural connective tissue across all these platforms exists,” Ratnagiri says. “Some are straightforward connections. Others are inferred connections. When we link them up, it opens a plethora of analytics.”

The data wouldn’t be helpful to anyone if it wasn’t available for queries, so the team’s third task was enabling access through an API layer. The business users wanted the system to output expressive, incremental information when they submit queries, so they included natural language support in the API.

Like any search tool, the API’s query terms needed to seem natural enough to be intuitive to users but sufficiently rich to accomplish the full range of possible queries. So, the engineering team interviewed stakeholders to define the most relevant search activities and build a series of canonical queries. Each of these queries sets the API off on a traversal through the entire Microsoft Azure Cosmos DB graph to locate and assemble the relevant data for the user.

When customers start to look at connecting multiple data sets, it is important to spend an appropriate amount of time early in the project on entity modeling and relationship curation across these entities. On the storage side, it is particularly important to pick the right partition key on the Cosmos DB side. This can have a significant impact on the latencies of queries in terms of defining in-edges and out-edges.

—Abhijit Mandal, senior software engineer, Microsoft Royalties

To maximize accessibility, the team built access to the API layer into a Microsoft Teams bot. Together, the team calls this end-to-end data solution Royalties as a Service (RaaS). Despite the system’s complexity, the outcomes are all about simplicity and empowerment.

Additionally, the API layer enforces security and confidentiality perimeters depending on who is using the system and what permissions they have.

Query execution

A user simply navigates to RaaS within Microsoft Teams and submits a natural language query like “payee 100010 drilldown” or “contract <abccdd> assurance.” This query passes through several different stages of execution in the pipeline before results are assembled and shown to the user on the Teams bot UI canvas.

These stages in order are:

1. Entity resolution:
   The natural language query is parsed to extract entities, sub-entities, and values. This is done using Azure cognitive service—Language Understanding Intelligence service (LUIS). Related entities are extracted as relationships and used in graph traversals. For the query “payee 100010 drilldown,” the entities and entity values extracted are “Payee”:“100010”
2. Intent formation:
   Intents are formed from the LUIS layer as well. Along with parsed entities, the user-intended action is added to form the intent object.
3. Dynamic Gremlin query generation:
   The intent object is passed through a query generation layer. The layer converts an intent object to a gremlin query that can be executed against a Cosmos graph DB instance. This is an example of a dynamic gremlin query:

   Example one:
   
   g.V().hasLabel('payee').has('payeeid','100010').range(0,1000).as('ct')
   
   .select('ct')
   
   .local(properties('column1','column2','column3','column4').group().by(key()).by(value())).dedup()

   Example two: The query below applies a contract ID filter on a contract node and traverses from the contract node over to product across connected edges, selecting the products associated with the contract.

   g.V().hasLabel('contract').has('contractid','1000010').as('contract')
   .outE('contract_to_product').inV().as('product')
   .select('product')
   .local(properties('column1', 'column2', 'column2', 'column3', 'column4',
   'column5').group().by(key()).by(value())).dedup()

4. Gremlin query execution:
   The final stage in the query layer is the execution of the dynamic gremlin query and converting the response to JObject of the relevant entities being selected.

The intent of sharing what a query looks like is to give customers an example of how they could tackle something similar in related efforts.

“When customers start to look at connecting multiple data sets, it is important to spend an appropriate amount of time early in the project on entity modeling and relationship curation across these entities,” says Abhijit Mandal, a senior software engineer working on the platform. “On the storage side, it is particularly important to pick the right partition key on the Cosmos DB side. This can have a significant impact on the latencies of queries in terms of defining in-edges and out-edges.”

The RaaS system today serves queries within sub-second latencies over a graph of 32 million entities connected through 110 million relationships. It’s been a long, important journey to launch RaaS, one that brought together disconnected tools that Microsoft uses to manage the agreements and relationships that define the company’s underlying royalties.

Aggregation and insights powered by connected data

Previously, users pulled data directly from several different sources, assembled it into meaningful formats, and validated the information through several layers of manual cross-checking. This was onerous for the engineering team—they had to understand each request, craft appropriate queries and mechanisms to harvest the data, and collate and aggregate the queries so they would be available to the business for further handling.

Sometimes, that process had a multi-day cycle time.

I’ve always thought of RaaS as a data-quality tool. This knowledge is now baked into a system. We’re getting a reliable answer through a unified process because it’s been structured properly.

—Chris Roozen, senior project manager, Microsoft Royalties

“With our RaaS system, retrieval and presentation of relevant information is automatic and driven by the end user with no time lost on the engineering and business sides, with the additional advantage of eliminating human error,” Venkatesan says.

That means it’s more difficult for errors to be entered into the system and that access is improved, which boosts accuracy and improves user satisfaction.

“I’ve always thought of RaaS as a data-quality tool,” says Chris Roozen, senior project manager on the Royalties team. “This knowledge is now baked into a system. We’re getting a reliable answer through a unified process because it’s been structured properly.”

The easier it is to get a clear picture of individual data pools, the simpler it is to look at the big picture and gain business-wide insights.

Opening up the connected data landscape

For now, RaaS is a relatively new capability on the Royalties team. As the internal experts on the query tool, engineers are RaaS’ primary frontline users, handling queries for the rest of the royalties business to help validate their data. In the future, they hope to simplify the search process with intelligent and predictive searches so it’s more user-friendly for non-engineers. In that scenario, anyone on the team will be able to submit queries and navigate the aggregated data independently.

Because team members will source their information through RaaS queries, fewer people will need access to the original data sourcing utilities. Limiting access to those tools helps decrease compliance risk within a large organization like Microsoft.

Similarly, as natural turnover occurs on the team, administrators won’t have to juggle access and training for multiple complicated data tools. Instead, RaaS will help all team members spend their time where it’s most valuable: validating data and building business insights.

There’s even the possibility of an outward-facing portal that customers, partners, and vendors can access to benefit from the ease and transparency that RaaS provides for Microsoft. But for now, RaaS is already demonstrating its value by saving time, eliminating error, and providing holistic insights.

• Collaborate from the start: Partner with your business, finance, and accounting teams to make sure you’re asking the right questions.
• Keep business intelligence front-of-mind: Know the questions your team wants to ask and the kind of answers they expect, then build toward that.
• Know your data and processes: Know the details of your data including its source, meaning, different processing (manual and systems) it powers, to fully capture the extent and strength of the connected data.
• Don’t rush to the API implementation: Make sure you’re seeking out the right information first and spend extra time on data modeling and graph design.
• Make sure you get your user scenarios right: This is cutting-edge work that’s tough for users to understand, so make sure you’re coaching teams on usage.
• Make your APIs very expressive: People are good at digital searches, so adapt your natural language processing to reflect everyday search habits.

Learn how Microsoft’s finance team uses anomaly detection and automation to transform royalty statements processing.

Find out how Microsoft designed a modern data catalog to enable business insights.

Explore how AI and chatbots simplify finance tools at Microsoft.

Learn how Microsoft is turning data into intelligent experiences.

Powering digital transformation at Microsoft with Modern Data Foundations.

Driving Microsoft’s transformation with AI.

The post Simplifying Microsoft’s royalty ecosystem with connected data service appeared first on Inside Track Blog.

Paperless contract lifecycle management (CLM) suddenly became more important during the worldwide COVID-19 pandemic, when schools had to go fully virtual to teach their students.

In 2020, organizations everywhere scrambled to adapt to changing conditions. The effect was especially pronounced for school systems, which needed to manage the transition to remote learning while ensuring educators and students were equipped with the tools and technology to make education at home effective. As a result, there was rapid growth in demand for professional services among educational organizations.

At the outset of the pandemic, Microsoft committed to supporting educators with training and resources to empower the transition to virtual and hybrid learning. Microsoft’s Global Training Partner (GTP) program was a cornerstone of that effort. The program represents a way for school systems around the world to find certified training partners who can support them with Microsoft tools and technology.

Microsoft’s GTP program was already active and robust, but it needed to scale alongside the rapidly expanding, pandemic-driven training needs of school systems worldwide. To accomplish that, the team needed to streamline the partner onboarding process—fast.

Previously, the process relied on paper forms like non-disclosure agreements, terms and conditions documents, and rules for collaboration—paperless CLM wasn’t something most schools were thinking about. Those hand-filled forms primarily relied on distribution by outside vendors, then headed through Microsoft Corporate, External, and Legal Affairs (CELA). The process wasn’t optimal from an efficiency standpoint, and it also exposed documentation efforts to error and duplicated work.

The Modern Workplace education team started looking for an automated, paperless CLM solution.

[Find out how Microsoft is powering digital transformation with Modern Data Foundations. Learn about designing a modern data catalog to enable business insights at Microsoft. Explore how Microsoft’s modern data governance strategy is accelerating digital transformation.]

Building a paperless CLM solution

Early in their discovery process, members of Microsoft’s Modern Workplace education team and owner of the GTP program approached CELA to see if they had a process in place for streamlining partner onboarding documentation. As it happened, CELA had already been working with Microsoft Digital Employee Experience engineers on a system to facilitate paperless CLM.

The intention was to build a headless system so that teams like GTP could come and integrate with us via APIs. Other teams would then snap into our existing engine to create their own separate processes.

—Chau Nguyen, program manager, Microsoft CELA

Over the course of seven months in 2019, Microsoft’s internal engineering team had been developing a solution leveraging icertis CLM capabilities, a leader in the CLM space.

“We worked with business partners in CELA to see what the needs were, then wove in the functionality of the Icertis CLM,” says Bidyadhar Patra, a Microsoft Digital Employee Experience engineering manager partnered with CELA. “We asked what other capabilities were necessary, worked through brainstorming and analysis, then worked on design.”

Patra’s team used tools across the Microsoft Azure technology stack to create a modern, API-based CLM solution with template automation, obligation management, and workflow management capabilities. “The intention was to build a headless system so that teams like GTP could come and integrate with us via APIs,” says Chau Nguyen, program manager on the CELA team. “Other teams would then snap into our existing engine to create their own separate processes.”

Legal is a complex domain. To understand the nuances of legal terms and build a user experience that’s helpful for efficiency and compliance, it’s critical to work closely with all of our internal business partners.

—Bidyadhar Patra, engineering manager, Microsoft Digital

The solution represented an integrated tool for managing the entire contract lifecycle, from creation through signing and output. It also included an API layer to make the system serviceable across different teams within Microsoft. The tool became known internally as Intelligent Contracting as a Service (ICaaS).

Due to the complexities of compliance and contract law, as well as the unique needs of the education space, close collaboration between Microsoft Digital Employee Experience engineers and CELA’s legal professionals was essential throughout the process.

“Legal is a complex domain,” Patra says. “To understand the nuances of legal terms and build a user experience that’s helpful for efficiency and compliance, it’s critical to work closely with all of our internal business partners.”

CELA implemented ICaaS in September or 2019, just in time to support the GTP program’s pandemic-driven need for rapid scaling and onboarding. With the CLM foundation in place, the Modern Workplace education team worked alongside CELA and Microsoft Digital Employee Experience to create a system that would largely automate partner onboarding, from application to document production, distribution, and retrieval.

The GTP onboarding process is a mostly automated workflow supported by the ICaaS CLM system that requires human interaction only at essential junctures:

• Potential GTP candidates apply for the program through an external portal, where an algorithm trims unqualified applicants.
• Eligible candidates enter a customer relationship management (CRM) system for approval and governance.
• The system initiates document creation integrated with Adobe E-Sign for paperless onboarding and passes it to the candidate.
• The tool pushes signed documents through workflows specific to legal and managing teams.
• Once onboarding is complete, the system uploads the partner to the public-facing GTP catalog.
• A Microsoft Power Apps integration automatically ingests them into an exclusive GTP channel on Microsoft Teams for communication and management.
• The training partner’s identity enters a custom Microsoft Power BI dashboard for tracking and monitoring.

The system removes all human intervention except where necessary, freeing team members to spend their time and energy resolving issues or developing strategic projects. This allowed it to become less about tool management and more about program management.

The outcomes of automation

This solution has enormously impacted Microsoft’s ability to provide the professional support necessary for educators to reach students during remote and hybrid learning.

“Because we were able to do all of the back-end cleanup and automation with CRM, we’re now driving greater partner discoverability,” says one of the members of the Modern Workplace education team. “It makes it very easy for the customers to discover who their training partners are, the training topics they need, and just get connected. So we’re able to drive leads and connect customers to partners in a more refined way.”

The system is helping scale the GTP program more quickly and effectively than ever before. From a potential training partner’s application to inception into the program, the time has decreased from three months to just one week. As a result, there are now more than 500 training partners in the GTP catalog.

On average, one partner has contact with around 1,000 educators per year. With those numbers, it’s not surprising that Microsoft Teams, the primary platform for virtual and hybrid learning, has reached 100 million monthly active users in education. The Modern Workplace education team attributes much of that success to the training partners onboarded through the streamlined intake system.

The Modern Workplace education team’s work has even forged a path for other businesses within Microsoft to develop their own automated contract management solutions in conjunction with CELA. There are currently 13 individual teams tapping into the ICaaS CLM engine.

In the education space, the collaboration between Microsoft Digital Employee Experience, CELA, and the GTP program is empowering the technology and training educators need to serve students in any circumstance—whether that’s a typical school day or a global disruption.

• Explore existing solutions before creating your own tool from scratch.
• Collaboration and joint planning are essential for avoiding rework.
• Educate stakeholders about what you’re trying to achieve.
• Stay focused on what drives the best value for customers and partners.
• Tool suites tend to work together well. If you can keep your solution within one toolset, the results will be more manageable.

• Find out how Microsoft is powering digital transformation with Modern Data Foundations.
• Learn about designing a modern data catalog to enable business insights at Microsoft.
• Explore how Microsoft’s modern data governance strategy is accelerating digital transformation.

The post Shifting to paperless contract lifecycle management to help schools navigate COVID-19 appeared first on Inside Track Blog.

Across Europe, workers’ councils are responsible for representing employees and protecting their rights. Elected from a company’s workforce, representatives protect against misuse of employee data and ensure compliance with local employee related law. Germany’s workers’ councils are among the most well-established in Europe, so they’ve become leaders in the conversation around employee data usage.

In the past, when Microsoft Digital, the company’s IT organization, wanted to deploy a new tool or service in Europe, the Microsoft workers’ councils in countries/regions there would undertake a lengthy, product-wide review process to ensure it didn’t violate employee rights. Any features that made personal, behavioral, or performance data usable to the organization triggered extensive reviews that could take up to a year and a half to complete.

The Microsoft Digital team, charged with deploying new features and services, needed a way to avoid delays and rollbacks while meeting regulatory obligations across all of Europe.

We have new functionalities every couple of weeks. We’re moving away from separate applications and toward platform thinking where everything is in one environment, for example in Teams or Dynamics 365. And then we’re adding modules, which means we have to think differently about how we review.

—Anna Kopp, regional director, Microsoft Digital

The solution?

Microsoft Digital, in partnership with Microsoft Human Resources and the company’s legal team, developed a collaboration with the Microsoft workers’ council in Germany that would triage and streamline approvals and then take that framework and use it to inspire similar collaborative efforts across all the countries that have workers’ councils.

[Learn how Microsoft is building an employee-centric experience. Experience the digitally assisted workday at Microsoft. Find out how Microsoft reinvented the employee experience.]
For a transcript, please view the video on YouTube: https://www.youtube.com/watch?v=grMzZ798W30, select the “More actions” button (three dots icon) below the video, and then select “Show transcript.”

Building a process for success

Microsoft Digital, the organization that powers, protects, and transforms the company, started by grappling with how agile software development has changed the nature of deployment—moving toward modular releases for individual features within larger platforms, which dramatically sped deployments up because it avoided lengthy, product-wide reviews.

“We have new functionalities every couple of weeks,” says Anna Kopp, regional director of Microsoft Digital in Germany. “We’re moving away from separate applications and toward platform thinking where everything is in one environment, for example in Teams or Dynamics 365. And then we’re adding modules, which means we have to think differently about how we review.”

The Microsoft Digital team undertook a full inventory of existing tools and categorized them by how they interact with employees’ private information. Features that presented no risk to employee privacy received expedited approval, so the team could move on to modules that share workers’ information in one way or another.

For those features, the team collaborated with the workers’ council to determine what components needed closer scrutiny. Does a tool’s contact card reveal an employee’s org chart? Does it employ gamification or stacked ranking? Are there red, yellow, and green indicators attached to employee names—no matter how benign their intentions?

Each of these elements has implications for how employees’ personal information and behaviors appear or how others might interpret their performance. As a result of their review effort, the team developed a review process for new features Microsoft plans to release.

First, the team must confirm the feature interacts with employees’ information in a limited, private capacity that General Data Protection Regulations (GDPR) are complied with, and that Microsoft commits to not make any performance and behavior control with the data. After that threshold is cleared, Microsoft Digital fills out a single-page intake form asking simple questions like the tool’s name, its audience, its owner, and access rights to the data. Finally, the Microsoft workers’ council expedites a simplified review that takes under four weeks.

Features that share employee information more widely or if Microsoft intends to carry out performance or behavior control with the data enter a second, more extensive process. Microsoft Digital completes a lengthier form that outlines things like how the tool would fit into the organization, how managers might use it, and if the information it presents could inadvertently suggest an employee’s performance to anyone other than really necessary roles or persons (e.g., a direct manager).

Trustworthy codetermination organized in this way—between the workers’ council and Microsoft—gave us speed and structure, and it gave us confidence. Because we’ve established strong relationships with stakeholders, we’ve been able to start from a place of compliance in product development and engineering.

—Peter Albus, chairman, Microsoft Germany Central Workers Council Committee for Employee Data Privacy

That form typically gives the workers’ council everything it needs to approve a feature. But if they still need more information, they initiate a dialogue with other teams; including product, engineering, and human resources.

Trust through collaboration

“What we try to do is balance the benefit with the risks,” says Peter Albus, dedicated workers’ council member and chairman of the Central Workers Council Committee for Employee Data Privacy for Microsoft Germany. “How can we mitigate the risks while enabling the benefit? If the added value is sufficient, we mitigate the risks through technical settings or organizational orders (policies).”

Beyond minimizing the time a feature might take to roll out, increased dialogue between the workers’ council and product teams has helped engineers anticipate regulatory demands and build solutions accordingly.

“Trustworthy codetermination organized in this way—between the workers’ council and Microsoft—gave us speed and structure, and it gave us confidence,” Albus says. “Because we’ve established strong relationships with stakeholders, we’ve been able to start from a place of compliance in product development and engineering.”

The ongoing deployment of Microsoft Viva illustrates how streamlined this process has become. Viva, Microsoft’s Employee Experience Platform (EXP), includes modules that interact with workers’ information differently.

Microsoft Viva Insights supports productivity and well-being by supplying behavioral data and intelligent recommendations directly to employees themselves. Private information doesn’t go beyond their own inboxes, so Microsoft Digital only needed a quick sign-off on the feature.

Microsoft Viva Learning is an AI-powered platform that offers relevant training to employees. In its initial form, Viva Learning included capabilities for tracking progress against course recommendations. Those tracking elements meant that the tool needed further review. But with only minor progress-sharing elements involved to secure approval, the team was able to put the tool into action quickly.

Microsoft Viva Manager Insights provides team leads with data about employees’ work patterns to help assess their workloads. That extra layer of behavioral data necessitates a more in-depth review.

But because the approval of Microsoft Viva Insights and Microsoft Viva Learning was so rapid and approval of Microsoft Viva Goals is coming, the workers’ council can give Viva Manager Insights their full attention. As a result, they’ve substantially reduced the turnaround time for Microsoft Viva compared to previous releases—from one and a half years to a target of around six months.

Cloud, agility, compliance

This more flexible, triage-friendly form of review wouldn’t be possible without cloud-driven agile development, which, for example, enables modular releases for individual features within Microsoft Teams. Agility in product development leads to flexibility in the approval process because entire products aren’t held back by individual feature reviews.

Instead, teams can easily activate or deactivate modules hosted within larger platforms without disrupting the overall experience. Kopp sums up the value of that modularity: “You can bring out the non-critical modules much faster and spend your time reviewing the ones that really need it.”

Meanwhile, increased collaboration between the Microsoft workers’ council in Germany and product teams helps the cross-company collaborative group anticipate and resolve many compliance issues before features reach the approval process. As a result of this multifaceted collaboration, Microsoft Germany is winning back hundreds of hours per year, and they’re sharing those lessons with other country and regional offices in Europe.

A leader in trust

Internal rollouts are only the beginning of the story. The rigorous process developed by the Microsoft workers’ council in Germany is a story worth sharing with partners and customers in hopes that it will help them streamline how they roll out new product features and services at their companies.

Roxana Schupp, account executive for Microsoft Dynamics 365, says it’s become a sales differentiator and a way to build relationships with customers who are trying to solve some of the same challenges at their companies.

“The customers are looking at us as a benchmark,” Schupp says. “The way we deal with data and how we address management or usage of systems in order to comply with regulations in different countries, those are very important for them.”

As a result of this close collaboration between Microsoft Digital and the German workers’ council, an unofficial rule has emerged across Europe. “If it works in Germany, it should work everywhere,” Kopp says.

Here are some tips on how you can better partner with your workers’ councils to streamline how you roll out new features and products at your companies:

• Build a robust triage system: Establish parameters for triggering reviews and criteria for different levels of engagement.
• Understand that you’re allies: Teams focused on compliance aren’t there to be blockers. They want to make sure enablement is compliant.
• Establish trust through dialog: Build internal awareness across teams to bring everyone to the table.
• Engage early: Seeking feedback early in the product development process avoids churn and rework.
• Embrace modularity: Deploying on a feature-by-feature basis empowers effective compliance triage without delaying overall product rollouts. Make the features configurable at the geo/country level to comply with local regulations.
• Ask good questions: What do you need to see? Why is this a concern? What are your fears?
• Compliance and privacy are non-negotiable, so ensure that your technical settings and policies mitigate risk while providing benefits.

• Learn how Microsoft is building an employee-centric experience.
• Experience the digitally assisted workday at Microsoft.
• Find out how Microsoft reinvented the employee experience.

The post Microsoft workers’ council partnerships boost the company’s product and service rollouts appeared first on Inside Track Blog.

We developed an automated contract creation tool to simplify this process for our internal users and to streamline our contract generation overall.

All of our more than 220,000 employees can act as procurement agents for the company. So how can we make that process and that engagement as simple as possible, knowing that these folks don’t have the rich background that our legal experts and the procurement teams have?

—Tom Orrison, director of legal operations, Microsoft Corporate, External, and Legal Affairs

[Learn how Microsoft is shifting to paperless contract lifecycle management to help schools navigate COVID-19. Explore OneExpense, the automated expense reporting backend transforming Microsoft. Find out how audit digitization and machine learning boost Microsoft’s internal audits.]

Maintaining agility while ensuring accuracy

At most enterprise companies, procurement departments manage SOW contracts. At Microsoft, we approach it differently.

“All of our more than 220,000 employees can act as procurement agents for the company,” says Tom Orrison, director of legal operations for Microsoft Corporate, External, and Legal Affairs (CELA). “So how can we make that process and that engagement as simple as possible, knowing that these folks don’t have the rich background that our legal experts and the procurement teams have?”

With Microsoft employees creating 150,000 SOWs per year, CELA needed a way to help users avoid making mistakes that come from manual data entry and from saving contracts locally on individual employee work devices. Our goal was to minimize human error, eliminate compliance risk, and save time and money by reducing the need for hands-on reviews by CELA’s legal professionals.

To solve this, CELA asked the Microsoft Digital Employee Experience team to build a tool to automate SOW contract creation.

Strategizing for simplicity

The teams started with a deep dive into the business problem to make sure the engineers understood the current state of SOWs and end users’ needs.

“We get together and brainstorm when we start any new digitization, and we go through the existing processes,” says Mohit Chand, principal director for the Microsoft Digital Employee Experience team working with CELA. “On the engineering side, we absorb that and then translate it into our strategy.”

Of the 150,000 SOW contracts that Microsoft employees create each year, 35–40 percent are relatively simple, short-form SOWs. Their simplicity made them a good target for automation.

CELA was already using a Microsoft Azure-based, third-party contract lifecycle management tool called Icertis to handle the submission and backend operations of the contracting process. They needed a way to create and upload those contracts that wouldn’t burden end users who don’t have a legal background.

Using a third-party tool based on Microsoft Azure meant that any solution they created could easily integrate with the Azure stack.

“We had already established this platform to build the foundation,” says Bidyadhar Patra, principal engineering manager for Microsoft Digital Employee Experience. “Now it was about onboarding this different business process to the platform so we could reap its benefits.”

Mohit’s team began their work with the objective of creating a web-based contract-creation wizard that used a guided questionnaire with natural language. Close collaboration between CELA and the engineers was essential to their process.

“We have a deep and healthy relationship,” Orrison says. “We view ourselves as the same team, which enables a lot of this work to happen seamlessly.”

Contract wizardry drives efficiency

Working in two-week sprints, Microsoft Digital Employee Experience started building out the tool. Legal is a complex domain, so the engineers needed to rely on the expertise of CELA’s experts to translate the ideas behind legal clauses into simple, user-facing questions.

“The CELA team helped us understand how the template should be configured,” Patra says. “Based on each question, which particular clause should be dynamically added to the contract?”

The result was the Legal Contracting Experience (LCE), a questionnaire-based contract creation wizard that helps users provide all the necessary information to create a short-form SOW, then automatically generates and submits the contract.

It’s more of a conversational or guided experience. Basically, the tool is guiding you through a conversation about how the SOW will be created.

—Bidyadhar Patra, principal engineering manager, Microsoft Digital Employee Experience

The LCE’s natural language questionnaire starts by determining whether a short-form SOW is necessary for a particular project or not. If it is, the tool asks 10 questions that correspond to specific legal triggers. For example, will the supplier handle privacy info? Where is the work being done? Is the SOW tied to an overarching contract from a preexisting vendor relationship?

Based on the user’s responses, the LCE automatically populates the contract with proper legal language drawn from a library of contract clauses. Then it uploads that contract in the appropriate Microsoft Word format directly to Icertis for contract lifecycle management.

“It’s more of a conversational or guided experience,” Patra says. “Basically, the tool is guiding you through a conversation about how the SOW will be created.”

But a library of clauses can’t account for everything in a contract. Each SOW contains a “Description of Services” section unique to that contract. Employees have their own idiosyncrasies for how they like to structure this section, including bulleted and numbered lists, formatted text, and highlighting.

One of the most substantial challenges was integrating rich text into the LCE itself. The first iterations of the tool didn’t include this feature, so users would have to export their contracts to Microsoft Word, fill in the rich-text portions, then re-upload their SOW to the tool.

“There was one sticky part that the legal professionals on our side weren’t happy with, and that’s the rich text formatting that was difficult for the engineering team to move from a web frontend into Word,” Orrison says. “But Bidyadhar’s team did a great job of listening to that feedback and putting in the hard engineering work to come up with a solution.”

The resulting tool incorporates the entire contract creation process into one streamlined, web-based experience that includes dynamic legal clauses, rich-text descriptions of service, and automatic submission to Icertis.

Speed, efficiency, and compliance

The LCE has simplified SOW creation and submission process across the board, from end users to CELA’s legal professionals. The resulting documents are compliant by design, with legal clauses that map themselves to their relevant regional compliance frameworks.

While it used to take around 30 minutes for a Microsoft employee to create a short-form SOW, the LCE brings the process down to an average of five minutes. Meanwhile, a typical CELA contract review used to take from two to five days. But because the LCE draws a contract’s clauses directly from an approved and up-to-date library of legal language, now CELA can accomplish any necessary reviews in under two days.

There were distinct benefits to choosing a third-party CLM solution based on Microsoft Azure. By starting from a place of compatibility, customization was much simpler than it would have been with other tools.

“This whole concept of having a core third-party utility but then adding a layer of extension and facade outside of that to plug it into Microsoft tools has become a blueprint architecture for us,” Chand says.

It also opens up the potential for deeper insights. Out-of-the-box API availability means that the team will be able to use Microsoft Graph API to chart contract-to-clause relationships as a data point, making big-data insights possible over time. Eventually, that will lead to even more intelligent automation, predictively tailoring the SOW creation process to different user profiles and anticipating their needs.

Employees just want to work with a partner to deliver the scope of their job. So the more we can make things seamless for our users, the better the experience, and it gives us the business velocity we’re looking for.

—Tom Orrison, director of legal operations, Microsoft Corporate, External, and Legal Affairs

The team is currently piloting post-assembly scanning for contracts using machine learning with Microsoft Azure Cognitive Services. Soon, the tool will be able to intelligently review the more customizable sections of contracts, even as they’re moving back and forth between Microsoft employees and vendors during the drafting and negotiation phases.

The next steps are integrating LCE into MyOrder, Microsoft Procurement’s internal purchase order environment, and extending it to more expansive forms of contract. That integration will tie the legal side of contract creation directly to the finance side and Microsoft Procurement, streamlining the overall process as the LCE expands its impact to more types of documents.

The engineers even dream of a time when a tool like this could be accessible through Microsoft Word as an integrated contract creation module. The feature would pull legal clauses directly into Word from a company’s preset legal clause pool, facilitate collaboration with suppliers, and submit the contract when it’s complete.

“The more we can integrate with downstream systems, the better off we are,” Orrison says. “Employees just want to work with a partner to deliver the scope of their job. So the more we can make things seamless for our users, the better the experience, and it gives us the business velocity we’re looking for.”

Here are some things to think about as you consider transforming the SOW process at your company:

• Partner closely with your subject matter experts.
• Think through the process from the user’s perspective.
• Recognize that driving change is hard and familiarity is powerful, so users need support.
• Commit to an iterative approach.
• Encourage your partners to be demanding in a good way.
• View your technical and business-side people as the same team.
• Be extremely clear on your must-haves.

• Learn how Microsoft is using Microsoft Dynamics 365 and AI to automate complex internal business processes and transactions.
• See how Microsoft is digitizing contract management with microservices on Azure.
• Beyond CRM: Learn how Microsoft is extending Dynamics 365 to manage its contracts.

The post Transforming how contract creation works at Microsoft with automation appeared first on Inside Track Blog.

[Editor’s note: This content was written to highlight a particular event or moment in time. Although that moment has passed, we’re republishing it here so you can see what our thinking and experience was like at the time.]

Managing financial transactions can be a tricky business.

When Microsoft employees and external payees run into invoice or payment problems, they turn to our expert support agents for help. But how do those agents know what’s working for their customers and what isn’t?

“As a customer-obsessed organization, we simply want to know how well we’re serving our customers,” says Jovalene Teo, senior technical solution manager for the Center of Innovation within Finance Operations. “When you come to us for help, are you getting the help you need?”

Finance Operations knew feedback and user insights were crucial to a great customer experience, so they created opportunities to harvest data at every stage of a user’s support journey. But not everyone has the time or inclination to share their thoughts. For users who don’t provide feedback, the team has developed a way to unlock those insights through machine learning sentiment analysis.

[Find out how Microsoft is reimagining employee support with Azure. Learn how Microsoft rebooted its internal app support with Dynamics 365. See how Microsoft is streamlining its global customer call center system with Azure]

Making it easy for users to share their experience

Our procure-to-pay (P2P) process covers everything from onboarding new suppliers to invoicing and payments. Both internal employees and external suppliers engage with the P2P process, so a robust support structure ensures that people without a finance background can deal with any issues promptly.

Finance Operations wanted to standardize the user feedback experience across every support team that touches the P2P process. That would help our frontline agents serve users better and provide deeper, business-wide insights into pain points and support trends.

“The idea came up that we should build a system where all these teams work through the same platform,” says Dhwani Kamdar, senior product manager for Financial Experience Applications within Finance Operations. “That way, they can talk to each other, and every single ticket or inquiry that comes into Microsoft Finance is tracked.”

The conventional feedback process used external surveys where employees or suppliers initiated a support ticket by contacting the team via email. Once the ticket was complete, users received a link to an external site where they could provide feedback. That extra step was a serious barrier because it asked users to interrupt their workflow by traveling to an entirely different website.

“We thought about how we could integrate feedback into a CRM experience that agents can see and interpret as they’re working on the tickets,” says Rajiv Maheshwari, principal software engineering manager for Financial Experience Applications. “So we went from after-the-fact tracking to more dynamic, interaction-focused insights available to agents in real time.”

Their solution evolved into a new tool that integrated user feedback into the flow of email communications during the support ticket process. As a result, they could incorporate the results into UniFinance, a platform based on Dynamics 365 that 25 Microsoft Finance teams use for support and feedback management.

Integrated, dynamic feedback powers meaningful conversations

Finance Operations worked to implement best practices that would make providing feedback more natural to users. Through collaboration with 25 team leads who manage 850 support agents, the engineers worked to ensure that the tool’s interface was stylish, accessible by design, and smoothly integrated into the email support process.

First, they updated the depth of possible responses. What started as a binary “satisfied” or “dissatisfied” choice evolved into a five-point satisfaction scale. That provided a wider array of feedback and aligned with the highest industry standards.

Next, the team needed to integrate feedback into ongoing support conversations. Fortunately, existing Microsoft Outlook features provided the basis for dynamic, in-email feedback.

The tool collects two different categories of user feedback. The first is known as “active feedback.”

This tool enables the user to provide their feedback more readily and easily and for us to collect those insights. So instead of just post-mortem feedback, we collect insights during the conversation so we can actually work with it to improve our service.

––Jovalene Teo, senior technical solution manager, Center of Innovation, Microsoft Finance Operations

Active feedback provides the opportunity for ongoing sentiment collection from the end user. In the signature line of every support email, users can click on one of five icons—with faces ranging from angry to overjoyed—indicating how they feel about their experience within the flow of communication.

UniFinance routes that user feedback through Microsoft Dynamics 365 CRM, making it available to support agents so they can understand the customer’s level of satisfaction and whether any course corrections are necessary. If a user submits feedback on the lower end of the spectrum, the CRM automatically keeps the ticket open, allowing the agent to offer further support.

“This tool enables the user to provide their feedback more readily and easily and for us to collect those insights,” Teo says. “So instead of just post-mortem feedback, we collect insights during the conversation so we can actually work with it to improve our service.”

When the support ticket is closed, the system requests a more extensive response, known as “closure feedback.” At this stage, users receive a final email featuring an Outlook actionable card that asks for an overall assessment on a five-point scale as well as a field prompting a textual response.

Closure feedback represents the end user’s final sentiment on the ticket’s resolution. Their input contributes to the support team’s key performance metric: customer satisfaction (CSAT).

On top of providing concrete guidance for frontline support agents, aggregating feedback data through Dynamics 365 allows the Finance Operations team to collect, analyze, and surface insights to business owners throughout the group.

Since the implementation of the tool, the participation rate for users providing feedback has soared. Before in-email feedback, the response rate was just 0.01 percent. That rate has now skyrocketed to 8 percent.

Completing the picture with machine learning

But what about the other 92 percent of cases? Finance Operations knew valuable insights were hiding in those interactions as well.

“Actual customer feedback is more common when the experience is on the high end or the low end, so either I’m super happy as a customer and I give you five stars, or I’m very frustrated and I give you one,” Maheshwari says. “But we were missing out on the middle of the spectrum, so we didn’t have the insights to improve that experience.”

Even when the customer doesn’t give feedback, the machine is giving us some insight. So we can still course-correct and make sure we’re doing our best to serve our customers..

––Dhwani Kamdar, senior product manager, Financial Experience Applications, Microsoft Finance Operations

Their solution leveraged machine learning to analyze the sentiment of user emails. Within the tool, Microsoft Azure Cognitive Services compares users’ language against established, extensively trained positive and negative sentiment language models. Then the system returns a positive or negative sentiment score to the support agent.

“Even when the customer doesn’t give feedback, the machine is giving us some insight,” Kamdar says. “So we can still course-correct and make sure we’re doing our best to serve our customers.”

Interpreting language and tone is complicated, so support agents use these sentiment scores cautiously. Even if a message receives a positive score, they still review it carefully. But when the system throws a negative score, agents know they should pay special attention to that message and potentially modify their approach.

Between dynamic user feedback and machine learning sentiment analysis, support agents now have powerful tools in their corner to help them provide excellent service.

Intelligent outcomes in finance

UniFinance has become a powerful engine for improving the finance support experience, providing both frontline and business-level insights.

“Just having sentiment analysis enabled wasn’t the only goal,” Kamdar says. “The goal was to use that data for building Power BIs and executive-level dashboards to see where we are and calculate customer satisfaction. So there are a lot of discussions around the data analytics and insights to create those dashboards and make sure we don’t stop at building the five smileys.”

The team is already piloting ways to use machine learning beyond sentiment analysis. Dynamic categorization is an exciting new feature that engages Azure Cognitive Services to scan user messages for keywords.

It associates those keywords with primary topics like “invoicing” and then breaks them down into increasingly granular subtopics such as “invoice payment failure.” Over time, the goal is to identify trends, build predictability, and even automate responses, saving finance professionals time to focus on more significant challenges.

In the meantime, UniFinance’s email feedback feature continues to improve the user experience through streamlined customer feedback, powerful data insights, and machine learning.

Here are some key lessons to keep in mind if you’re planning to rework the support process at your company:

• Start by reimagining the possibilities: When you work in a traditional field, moving out of the status quo is the goal.
• Think in terms of the customer experience because that’s where you’ll find your innovations.
• Design matters: Never underestimate how aesthetics can drive results.
• Keep an open mind: The more you talk to users and the more demos you run, the better and faster your work will be.
• When you’re working with machine learning and text, start small with specific scenarios, find success, then build on it.
• Having a diverse team is very advantageous because they bring skills, languages, and insights you won’t find elsewhere.

• Learn how Microsoft is integrating ServiceNow with Microsoft Azure Monitor.
• Find out how Microsoft is modernizing its support experience with ServiceNow.
• Learn how Microsoft is enabling a modern support experience.

The post Maximizing Microsoft Finance’s support experience with dynamic feedback and machine learning sentiment analysis appeared first on Inside Track Blog.

Large organizations have a lot of moving parts, and when internal sites experience outages and connectivity problems, those moving parts can grind to a halt.

Getting things up and running again is priority one.

In the past, responding to site outages at Microsoft required immense manual effort from the Experience and Reliability Engineering (ERE) team, which manages the company’s global wide-area networks (WANs). But by working with Microsoft Digital Employee Experience’s AI Operations team, the company has developed a powerful new system for arriving at the root cause of an outage and remedying the issue before it results in serious downtime, disruption, or in worst-case scenarios, lost revenue.

[See how Azure Anomaly Detector is helping Microsoft examine SAP transactions. Learn how Microsoft applied Azure Cognitive Services to automate partner claim validation. Discover ways that Microsoft Dynamics 365 and AI automate complex business processes and transactions.]

A race to the root cause

Site outages occur when buildings lose connectivity and become isolated from the network, essentially cutting workers off from the WAN connection they need to do their jobs. They can arise from any number of issues. An outage might result from a network failure on the part of internet service providers, or there could be a localized power failure. In some cases, a simple physical disruption like a misplaced cable or an incorrectly flipped switch could knock out a network.

Depending on the site and the size of the outage in question, there’s a potential loss of productivity for hundreds or thousands of people. There are other threats in terms of security, and if we’ve got deadlines to meet from a product standpoint, we could potentially miss those dates. So, these kinds of outages tend to be very critical.

—Anand Meduri, project management lead, AI Operations team

When a site goes down, the resulting cascade of connectivity issues and communication failures trigger a flood of automated incident reports generated by a network monitoring tool called alert sources. Those alerts register as incident tickets for ERE, and it’s the responsibility of the team’s network developers to quickly identify the root cause of the tickets and discover whether the sudden uptick in incidents derives from an overarching site outage. They need to sift through the parent-child relationships of the outage and its resulting incident flood to ensure business continuity and reduce the negative impact.

“Depending on the site and the size of the outage in question, there’s a potential loss of productivity for hundreds or thousands of people,” says Anand Meduri, project management lead on Microsoft’s AI Operations (AIOps) team. “There are other threats in terms of security, and if we’ve got deadlines to meet from a product standpoint, we could potentially miss those dates. So, these kinds of outages tend to be very critical.”

Under the previous manual model, ERE network developers troubleshooting a site outage often didn’t have enough information to discover its root cause since their data came from unstructured logs, performance metrics, and alerts. They spent significant time validating the flood of tickets they received, searching for the single ticket corresponding to the parent outage. Once they determined the source, they would attempt to remedy the issue themselves. If it required a more intensive fix, they would turn the incident over to the Escalation Management team or a high-level Direct Response Individual (DRI) for remediation.

We looked into all of the narrow device signals and system logs, as well as the metrics and health measures we can gather from them, and we found some good signals that we could use. And we also found a pattern: when those signals are acting slightly differently from normal behavior, it should be related to a site outage.

—Eunsil Baik, data scientist, Microsoft Digital Employee Experience

ERE has a 15-minute target time for determining the root cause of any major outage. They have 30 minutes to engage the proper team and effect a repair—a demanding timeline considering the sheer number of alerts deriving from any parent outage. With around 15,000 tickets a month, each needing validation from an ERE member, their attention was spread extremely thin. As a result, network developers were spending more time validating downstream tickets than resolving the underlying issue, slowing a key metric for their team: mean time to resolve (MTTR).

The solution was to automate root-cause analysis with machine learning and AI.

Designing a solution from the data up

ERE partnered with Microsoft Digital Employee Experience’s AIOps team to develop an outage detection system based on machine learning and AI. They began by building out the data set from scratch, setting the standards for collecting and categorizing outage information. The core principle is simple: Any major site outage will reliably lead to a series of system alerts and tickets arising from downstream breakages. With enough training, machine learning should be able to correlate those signals and identify the root cause.

Data scientists and engineers collected and categorized the relevant data points. They identified alerts, device and system logs, and performance logs as signal sources since those would capture the wealth of data emerging from any cascading outage.

“We looked into all of the narrow device signals and system logs, as well as the metrics and health measures we can gather from them, and we found some good signals that we could use,” says Eunsil Baik, a data scientist in Microsoft Digital Employee Experience. “And we also found a pattern: when those signals are acting slightly differently from normal behavior, it should be related to a site outage.”

The device signals and system logs that indicate outages came from three main sources:

• SMARTS: The tool for monitoring networked devices like routers, switches, and CPU processors, which generates alerts when it observes faults.
• Splunk: A collection of real-time logs related to power events, systems, audits, activities, and configuration changes.
• SevOne Performance: A monitor of different performance metrics, including availability, inflow, outflow, jitter, and connectivity from network devices polled at regular intervals.

Assembling and managing the information that these monitoring tools can provide required a variety of tools throughout the Microsoft Azure Data Stack:

• Azure Data Lake: Storing data for ingestion from different sources and storing gold data
• Event Hub: Streaming data ingestion
• Stream Analytics: Filtering data
• Blog Storage: Housing data
• Data Factory: Scheduling data processing and model training
• Azure Key Vault: Storing sensitive authentication data
• Azure Data Bricks: Developing notebooks for data processing and wrangling, conducting experiments, and building models
• Azure Monitor: Overseeing resource health
• Azure DevOps: Automating deployments and delivery
• Azure Web App: Deploying model as an endpoint
• Azure Functions: Processing data, combining signals, and creating single data points while doing real-time detection

Making the model work

Outage classification is fundamentally a binary problem, so the team designed the system to create an incident based on an alert received from its source. The signals related to that alert route to the machine learning service as an input and yield up a binary “yes” or “no” classification.

To establish that binary classification, they focused the machine learning model on determining whether an alert is a site-outage or non-site-outage through the three independent signal sources: SMARTS, Splunk, and SevOne.

By codifying the three signals associated with fault monitoring, real-time logs, and performance metrics into patterns, the team established the variables for training the AI to detect outages. In effect, the process transformed the manual actions that engineers would undertake during their search for an outage into static business rules. Those rules made up the labeled data set they used to train the machine learning model.

The AIOps team selected the XGBoost Classifier machine learning model to work on the data. From there, they had to wait a month for the model to accumulate enough data to begin developing a high-quality machine learning model.

The next step was evaluating and validating the model. When searching for site outages, the system could predict one of four results: a false negative, a false positive, a true negative, or a true positive. By assembling those results into a confusion matrix, it would be able to quantify the degree of accuracy. Finally, the team determined accuracy by generating a ratio of correct predictions—true positive or true negative—against all predictions made by the algorithm.

In its training environment, the team found that the model accurately identified a site outage 92 percent of the time, a level of accuracy on par with lab environments. The system has now been trained rigorously and deployed in a live environment for six months, and it successfully predicts 49 percent of real-world outages. As the algorithm trains over time, that number continues to improve.

The outcomes of automation

By successfully identifying around half of all site outages, the machine learning model eliminates an immense amount of manual labor associated with sifting through cascading alerts and tickets. Instead, ERE can go straight to the parent issue—the primary site outage—and quickly repair it themselves or pass it along to Escalation Management or DRIs.

When the system correctly traces a flood of alerts to a single site outage, identifying and repairing that outage clears 99 percent of the tickets associated with the disruption. As a result, ticket clearance has increased in both quantity and speed. In an organization where revenue and reputation are on the line, speed is central to outage response, and the machine learning model has drastically accelerated ERE’s response to incidents.

If I’m spending my energy on fixing access point issues or downed switch issues, those are less critical, and they can wait. But site outages are super critical, and we want to detect them as soon as possible so we can fix them right away. This AIOps solution accomplishes that.

—Laxman Rao Bhinnale, problem manager, Experience and Reliability Engineering

15,000 tickets per month work out to around 500 tickets per day, and on average, an ERE member would take about one minute to validate each incident. With machine learning eliminating the need to validate almost all those tickets and pointing directly to the parent site outage, the average number of tickets requiring validation is down from 15,000 to typically under 50, saving around 240 hours of work in an average month.

In addition, the machine learning site outage detection system saves an average of 10 minutes during the process of identifying and triaging an outage. That time savings represents a substantial portion of ERE’s 15-minute target for determining an outage and their 30-minute target to engage the proper team for a repair.

The success we’ve achieved has taken us from manual to automated, and we’ve built this solution for just one use case. This collaboration will definitely drive the data-driven discussion and expand its scope to other scenarios.

—Satish Aradhya, senior service engineering manager, Experience and Reliability Engineering

Ultimately, the system’s value comes from eliminating the need for human intervention and the possibility of error, freeing ERE’s network developers to focus on the most critical aspect of their work: repairing outages and resolving complex tickets.

“If I’m spending my energy on fixing access point issues or downed switch issues, those are less critical, and they can wait,” says Laxman Rao Bhinnale, problem manager for ERE. “But site outages are super critical, and we want to detect them as soon as possible so we can fix them right away. This AIOps solution accomplishes that.”

Extending the applications of automation

Now that AI and machine learning are supporting outage detection, potential applications for the system are on the horizon.

“The success we’ve achieved has taken us from manual to automated, and we’ve built this solution for just one use case,” says Satish Aradhya, senior service engineering manager for ERE. “This collaboration will definitely drive the data-driven discussion and expand its scope to other scenarios.”

A system like this could be useful for any number of infrastructure challenges. Teams might use a similar model to correlate other scenarios or even work toward self-healing solutions that eliminate the need for human intervention in outages at all.

“Our vision is very ambitious,” Bhinnale says. “We have a lot of things in the pipeline, and this is just the beginning.”

• Stay laser-focused on customer outcomes to help guide your process.
• Choose the right data sources and patterns to mimic what a human would do.
• Engage with stakeholders to reduce churn through frequent course corrections.
• Be aware that in large systems, you won’t have complete data.
• Leverage subject-matter experts across teams to get input from diverse engineers on the ground.
• Start thinking about the applications from the outset and infuse those techniques early on.
• Recognize that you’re creating a data-driven solution and start with the “why.”

See how Azure Anomaly Detector is helping Microsoft examine SAP transactions.

Learn how Microsoft applied Azure Cognitive Services to automate partner claim validation.

Discover ways that Microsoft Dynamics 365 and AI automate complex business processes and transactions.

The post Responding to site outages at Microsoft with machine learning and AI appeared first on Inside Track Blog.