Like many companies, Microsoft pays royalties for games, videos, content, software, and other creative work it uses in its products and services.

The company pays a lot in royalties—nearly $5 billion a year.

“The quality review and approval of payments has been a time-consuming manual process for our operations teams,” says Rohit Amberker, a finance director on Microsoft Finance’s royalty team. “We review nearly 30,000 statements each year.”

Now that frustration is going away.

How?

Microsoft Digital has built an experience that digitizes the company’s royalty payment process from end to end, says Jagannathan Venkatesan, principal group engineering manager for Microsoft Digital, which provides IT support for Microsoft Finance.

[Go here to learn more about Driving Microsoft’s transformation with AI. Click here to find out how Microsoft increases sales by using AI for lead qualification. Learn how Microsoft uses machine learning to develop smart energy solutions.]

“The Royalties team has been making significant strides in our service-maturity journey, enabling our business partners to close our month-end activities in a stable manner,” Venkatesan says. “Having accomplished a successful migration from an on-premises system to a cloud-based distributed system performing at scale, we focused on helping our business partners with the effectiveness of operational processes.”

Venkatesan adds: “We targeted the statement-approval process, a monthly recurring activity, involving many thousands of contracts, currently performed manually. The ML/AI-based solution for statement approval dovetails nicely into a larger effort across Microsoft Digital to digitize how it supports internal business partners like Microsoft Finance.”

Complexity untangled

Part of the challenge has been the complex nature of royalty payments, says Katie Lencioni, a principal program manager at Microsoft Digital.

“We collect all transactions that are subject to a royalty payment based on views, downloads, or plays depending on the type of content,” she says. “From there, we calculate the payment amount according to the contract of the content owner, create a statement, and ensure the payments are processed.”

With 30,000 statements per year, there is ample opportunity for errors.

“We perform three-way recon, comparing source transactions from Microsoft Marketplace with auto-calculated statements and against a third independent source,” says Rajmohan Venkatesan, a software engineer at Microsoft Digital.

“This was a completely manual process, using reports imported to Excel sheets and macros to identify mismatches,” Venkatesan says. “It was labor-intensive, with multiple people performing the analysis in different regions. Now, we pull the various streams into an HD Insight cluster and use business rules to filter the royalty-bearing transactions to ensure parity.”

From there, the team had to apply machine learning to determine anomalies in payment variances. To do this, they had to understand the thought process and business rules used by analysts to determine the validity of a payment.

“The Royalties team expects payments for a contract to fall between a minimum and maximum range. Outside the range is an outlier,” says Geeth Priya Namasivayam, also a Microsoft Digital software engineer. “To truly automate is not just to identify the anomaly, but to determine which dimension caused the anomaly. We use a quantile regression algorithm, which allows us to clearly define the variance based on cause.”

Machine learning requires a lot of data to learn. To get a big enough data sample, the team loaded 24 months of payment and transaction data into the pipeline. This provided transactions, payment history, seasonality, and launch events, which allowed the team to create a model that included all typical variances.

“The machine-learning job is initiated as each statement is released, using current data for the contract and loading historical data for comparison,” Namasivayam says. “If an anomaly is detected, the data set is reduced and additional dimensions such as seasonality, launch periods, and geography are used to isolate the source of the anomaly.”

The goal was to make the application as easy to use as possible.

“The user interface is very simple. A statement is classified into three categories: green meaning no anomalies, yellow meaning an anomaly is detected but it’s due to seasonality or launch, and red meaning an unexplained anomaly is detected,” Namasivayam says. “We incorporated a thumbs-up or thumbs-down button, which allows the user to validate the classification and provide learning reinforcement for the algorithm.”

Trying it out

With the data pipeline and algorithm in place, the team was ready for real-life testing.

“We had to prove to our stakeholders that our algorithm was accurate. This took several months of automated validation and manual review of the data by the finance team. We had to build trust,” Venkatesan says.

After the accuracy of the system was validated, the proof-of-concept was moved into production.

“With confidence in reconciliation accuracy and no anomalies, the finance team was comfortable to automatically approve transactions for amounts below certain thresholds,” Venkatesan says.

Embracing a new technology that the team would trust to take on an important process regulated by federal Sarbanes-Oxley Act (SOX) rules was not an easy leap for the operations team to make. Microsoft’s internal Controls and Compliance teams, along with the Corporate Finance Controls group, partnered closely to ensure the risks were properly addressed.

“It was uncomfortable and a little scary yielding an important process to automation and machine learning,” says Mohammad Shafaqat, global process owner on the Royalties team in Microsoft Finance. “As things went along, we learned that we could keep the machine in check by providing ongoing feedback.”

This work has not only delivered significant time savings but has also improved morale in many ways. Professionals no longer perform painful mechanical work but instead are focused on building partner relationships and delivering insights.

It’s all making a major difference, Amberker says.

“With Azure ML services, approximately 90 percent of Xbox statements and payments are now automated, saving 20,000 hours per year in the steady state,” he says.

Amberker says this is just the start; the anomaly detection-based engineering approach is now being applied in many other functions at Microsoft. “Stay tuned for more updates,” he says.

• Learn more about Driving Microsoft’s transformation with AI
• Find out how Microsoft increases sales by using AI for lead qualification
• Learn how Microsoft uses machine learning to develop smart energy solutions

The post Key SOX control at Microsoft is transformed through AI appeared first on Inside Track Blog.

Microsoft is using a new suite of Microsoft Azure telemetry tools to gain insight into how to better manage expense reports, time away reporting, purchase order creation, and similar business processes that get routed through one of the largest SAP instances in the world. Before Microsoft moved its SAP workload into Azure and before it started using new Azure telemetry tools, there was no way to connect such transactions inside and outside of SAP.

“This made the process feel like a black box to Microsoft employee users, and to our engineers, who needed to figure out what was happening so they could connect the dots and improve our services,” says Enda Sullivan, a senior program manager for Microsoft Digital’s internal SAP implementation.

Because the SAP processes were not connected end-to-end, it was difficult to help Microsoft employees when something went wrong.

“If there was a problem, it would require the user to engage with multiple teams, both SAP and non-SAP, to understand the status of a request,” Sullivan says. “Beyond that, the support teams often wouldn’t have visibility to failed transactions between the various system steps. As a user, I should never have to call a helpdesk, the failure should be detected by the service telemetry and monitoring, and be resolved before I’m even aware.”

Now through Azure, the team has real-time data that tells them when SAP process issues come up, which, importantly, allows them to get resolved before users realize something went wrong.

Telemetry comes a long way

Historically, the use of telemetry to guide how companies like Microsoft use SAP was spotty at best, says Cory Delamarter, a principal program manager tasked with driving telemetry design and implementation standards across Microsoft Digital, which provides IT services for all of Microsoft.

“It’s great that Azure is giving us these new tools to work with, but this is something we were already starting to tackle,” he says. “The opportunity and value of getting all our data in one place is too high to not solve this problem.”

Delamarter’s team is working to bring a consistent approach to telemetry across all Microsoft Digital, an effort that has been tabbed the Unified Telemetry Platform (UTP).

“When we started to architect a new solution for telemetry, it needed to be scalable, reliable, and cost-effective, but most importantly a single common platform across the org,” he says. “Essentially, we are consolidating tools and data stores.”

Standardization was a must, he says.

“We had to design in flexibility that would support more than a system for monitoring a database or website,” Delamarter says. “The power of unified telemetry is the ability to solve problems across boundaries, or service health, which supports the higher-level business processes.”

The companywide approach to telemetry is being built around Azure Monitor Application Insights, Azure Data Lake (Gen2) and Azure Data Explorer. “Application Insights provides the ability to ingest and organize incoming telemetry data, while Azure Data Explorer gives us the ability to aggregate and support queries across very large data sets stored in our data lake,” he says.

Taking away the mystery

When it comes to getting more insights out of Microsoft’s SAP workload, it really came down to taking the mystery away.

“For SAP, we had to get out of the four walls and shine a light so it’s no longer a black box,” says Aron Stern, a senior software engineer inside Microsoft Digital who is responsible for the Azure architecture of the company’s SAP infrastructure.

First, the team needed to simplify everything.

“We thought about the solution in two parts, telemetry, the raw metadata being emitted by our applications, then monitoring, reporting on service health through dashboards and alerting,” Stern says. “From there we separated the data into three layers—infrastructure, application, and business process.”

Then the team needed to implement a bit of customization.

“We built a small custom application using a few Azure tools,” Stern says. “That allowed us to convert our application and business process telemetry events from SAP into common schema—this allowed us to stitch our transactions together so we could get the end-to-end view we were looking for.”

The team then ingested all the SAP data into its Application Insights instance and fed that into an Azure Data Lake for cold storage and Power BI reporting.

This new shift has enabled a powerful transformation, says Blake Barrow, a principal software engineer inside Microsoft Digital. “Business process telemetry is what enables us to measure SLA’s and provide transparency to our users,” he says.

Everyone who has access to this new telemetry are enjoying a whole wave of new insights.

“Our employee users now have better insight into any transaction they have on SAP,” Barrow says. “Our engineering teams are getting the data they need to detect issues before they can create downstream problems. Business executives have access to dashboards that provide near real-time status of transaction volumes, so they look for trends and do health checks on their programs.”

All are changes that wouldn’t be possible without transforming the way Microsoft approaches telemetry. “These are the kinds of improvements that can help us all have more impact,” he says. “This is what digital transformation is all about.”

Barrow, Stern, and Sullivan will be presenting how Microsoft is using UTP to gain insights on its SAP workload at the SAP SapphireNow Conference on Wednesday, May 8th.

• Go here to learn more about their session.
• Go here to learn more about how Microsoft migrated its SAP workload to Microsoft Azure and much more on how the company is managing its SAP workload.

The post Microsoft internal SAP workload gets a telemetry boost with Azure appeared first on Inside Track Blog.