{"id":6318,"date":"2023-06-22T08:00:55","date_gmt":"2023-06-22T15:00:55","guid":{"rendered":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?p=6318"},"modified":"2023-06-25T13:31:50","modified_gmt":"2023-06-25T20:31:50","slug":"detecting-anomalies-at-microsoft-with-unsupervised-machine-learning-in-microsoft-azure","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/insidetrack\/blog\/detecting-anomalies-at-microsoft-with-unsupervised-machine-learning-in-microsoft-azure\/","title":{"rendered":"Detecting anomalies at Microsoft with unsupervised machine learning in Microsoft Azure"},"content":{"rendered":"

\"Microsoft[Editor\u2019s note: This content was written to highlight a particular event or moment in time. Although that moment has passed, we\u2019re republishing it here so you can see what our thinking and experience was like at the time.]<\/em><\/p>\n

Can unsupervised machine learning in Microsoft Azure be used to find errors and anomalies inside financial data?<\/p>\n

Shilpa Tiwari thinks so.<\/p>\n

Tiwari had a vision of using Microsoft\u2019s complex volumes of financial data to develop new methods of identifying risk. Tiwari is a principal group engineering manager with the Financial Services group in Microsoft Digital, the engineering organization at Microsoft that builds and manages the products, processes, and services that Microsoft runs on.<\/p>\n

But first, Tiwari and their team would have to solve the problem of traditional modeling practices.<\/p>\n

\u201cIn the past, we\u2019ve been looking at data in an \u2018if this, then this\u2019 way,\u201d Tiwari says. \u201cThat doesn\u2019t scale. Microsoft\u2019s transaction volume is going up ten-fold, but the financial data is much larger than that. Plus, it\u2019s all done in silos. If a wrong payment is made, you\u2019re only looking at payment data, not the connected datasets.\u201d<\/p>\n

To innovate on the way information is extracted from financial data, Tiwari and the Finance Data Insights Team, a subset of the Financial Data Services group, looked to artificial neural networks, an unsupervised machine learning technique, to identify anomalies.<\/p>\n

With the help of business partners throughout Microsoft, Tiwari and their team are improving the way errors and anomalies are recognized in historically disparate datasets, breaking down silos and proactively predicting risks.<\/p>\n

[<\/em>Learn how Microsoft finance professionals make data-driven decisions on cash flow<\/em><\/a>. <\/em>Read more about Microsoft\u2019s connected and discoverable data<\/em><\/a>. <\/em>Read about Microsoft\u2019s digital transformation with a modern data foundation<\/em><\/a>.]<\/em><\/p>\n

Changing financial data support<\/h2>\n

It should come as no surprise, but Microsoft deals with a lot of data and more is added every day.<\/p>\n

The Financial Data Services group supports Microsoft\u2019s efforts to understand financial data, including corporate data. Several teams, including the Finance Data Insights Team, help oversee and understand the complex and sizable datasets that Microsoft manages.<\/p>\n

\u201cThese teams are all managing, governing, providing access and insights, and reporting for all of Microsoft\u2019s financial data,\u201d Tiwari says.<\/p>\n

The amounts of data the Finance Data Insights Team sees exposes the very heart of the problem.<\/p>\n

Not only are they facing enormous datasets and sources spread across all of Microsoft, but traditional supervised approaches to machine learning cannot keep pace with the complexity or volume.<\/p>\n

\u201cRules-based systems will not scale with the amount of financial data we receive, nor will it give us richer insights through connected datasets,\u201d Tiwari says.<\/p>\n

This makes it incredibly difficult to extract useful intelligence from sources and easy for anomalies to go undetected.<\/p>\n

There are human limitations in terms of thinking several permutations ahead. It\u2019s not like anomalies are going to be able to communicate how they\u2019ll appear.<\/p>\n

\u2013 Shilpa Tiwari, principal group engineering manager, Microsoft Digital<\/p>\n<\/blockquote>\n

Traditional models for parsing, such as rules-based machine learning, require long sprints and heavy involvement from data scientists, engineers, and subject-matter experts (SMEs).<\/p>\n

\u201cIt takes months to launch a rules-based system,\u201d Tiwari says. \u201cYou have to develop rules, test, and then launch them. The SME expertise to do this is finite, and there are human limitations in terms of thinking several permutations ahead. It\u2019s not like anomalies are going to be able to communicate how they\u2019ll appear.\u201d<\/p>\n

The Finance Data Insights Team saw an opportunity to use new technology and Microsoft\u2019s vast network of SMEs to gain richer insights from the company\u2019s growing financial data.<\/p>\n

Identifying risks with artificial neural networks<\/h2>\n

Instead of running from the large amount of data, the team leaned into it.<\/p>\n

\u201cAs you get more data, you want to be able to turn it on in a semi-automated fashion,\u201d Tiwari says. “You don\u2019t want to spend months going to SMEs. Unsupervised machine learning learns from the data without human labelling and engineering. It\u2019s complex, but it\u2019s well suited for this kind of application.\u201d<\/p>\n

Unsupervised machine learning not only performs better as it scales, it\u2019s also faster. Unlike a rules-based approach, unsupervised machine learning develops its own model, benefitting from large amounts of data.<\/p>\n

In addition to quickly processing the information, unsupervised machine learning is able to learn from a variety of sources, which further improves the model as it identifies patterns a rules-based approach might not account for.<\/p>\n

This makes Microsoft\u2019s financial data a good use case to demonstrate impact.<\/p>\n

\u201cFinancial data is prone to risk,\u201d Tiwari says. \u201cIt\u2019s also prone to regulatory controls. You have to be proactive about the risks. The more insights we can extract, the better we get at predicting these risks.\u201d<\/p>\n

But where to start?<\/p>\n

\u201cWe picked a paper based on artificial neural networks,\u201d Tiwari says. \u201cIt\u2019s not easy to introduce a model through a research paper, but a lot of these ideas are totally new.\u201d<\/p>\n

Tiwari needed the Finance Data Insight Team to apply the knowledge within the research, so Tiwari tasked Joy Chepkwony and Abhishek Mehra, two software engineers from the team, to make the unsupervised machine learning vision a reality.<\/p>\n

\u201cIt was a different process for me, starting from a research paper like this,\u201d Chepkwony says. \u201cInitial research leads you to a lot of other papers. Some help solve certain things, but it becomes a lot of trial and error.\u201d<\/p>\n

For Mehra, the process was a complicated balancing act with lots of moving parts.<\/p>\n

\u201cA good insight project needs engineers, scientists, a good dataset, and a feedback loop,\u201d Mehra says. \u201cThat\u2019s a challenge. If you\u2019re running things with only one, you\u2019ll fail.\u201d<\/p>\n

Having read through several papers, Chepkwony built the unsupervised machine learning model in Azure Databricks<\/a>. Unlike a traditional model, the artificial neural network could be engineered without the input of SMEs. Chepkwony designed the model to extract insights from a series of algorithms and services that sit atop data and compute planes.<\/p>\n

\u201cThe pipelines are all on the Azure stack,\u201d Chepkwony says.<\/p>\n

To find anomalies, data passes through an autoencoder, a type of artificial neural network. According to Chepkwony, the autoencoder compresses data to its latent state, then the decoder decompresses the latent representation. The decompressed output, a reconstruction of the original, is compared to its initial source and if there\u2019s a high enough error rate the data is flagged as \u201cnot normal.\u201d<\/p>\n

Items that fall below the \u201cnot normal\u201d threshold could be a variety of things, including erroneous or suspicious activity. Additionally, the model could just be spotting something that\u2019s simply outside of a normal pattern.<\/p>\n

\u201cIt\u2019s hard for humans to look at datasets and follow patterns or explain what\u2019s wrong,\u201d Mehra says. \u201cWe want a feedback loop that includes auditor feedback.\u201d<\/p>\n

Mehra, who also helped acquire the connected data sets, reached out to several SME business partners across Microsoft to help explain the findings of the model.<\/p>\n

They had a map but needed to make sense of it.<\/p>\n

Getting the most out of an unsupervised machine learning model<\/h2>\n

In trying to understand anomalies, context matters.<\/p>\n

That\u2019s why the team relies heavily on SME partners to help explain the anomalies, including Simaan Huda, principal program manager with Microsoft\u2019s Corporate Functions Engineering team.<\/p>\n

\u201cMy team was looking to use transaction level insights,\u201d Huda says. \u201cWe thought about doing it on our own, but we saw the benefits of partnering with the Finance Data Insight Team.\u201d<\/p>\n

Initially, Huda used the neural networks to read through Microsoft\u2019s general ledger.<\/p>\n

\u201cWe were getting a lot of flagged items that weren\u2019t suspicious,\u201d Huda says. \u201cThey were often entries of less significance or manually entered. Anomalies, but not actionable.\u201d<\/p>\n

But it was useful for the Finance Data Insight Team. As the neural network identified deviations, Huda and their team could help explain the anomalies. This feedback fine-tuned the model for a variety of circumstances.<\/p>\n

The large size of the general ledger was initially a challenge.<\/p>\n

\u201cThere are so many practices at Microsoft that it takes a lot of SMEs to understand the general ledger,\u201d Huda says. \u201cBut there are many subprocesses that make up the general ledger, and by learning on these sets we can use machine learning to uncover knowledge, especially what\u2019s not well documented or understood beyond the SMEs.\u201d<\/p>\n

By looking at portions of the general ledger, such as manual entries, Huda and their team were able to hone the model with Chepkowny and Mehra.<\/p>\n

\u201cMachine learning is about running scenarios as expected, but there\u2019s also surrounding data that\u2019s interesting,\u201d Mehra says. \u201cWe can enrich data by running the model again, taking the smaller subsets plus the general ledger to extract insights.\u201d<\/p>\n

With each iteration, the model improves itself.<\/p>\n

This entire approach still comes with a steep learning curve, especially for SME partners who don\u2019t understand how the unsupervised machine learning model identifies data as not normal.<\/p>\n

\u201cIt\u2019s a journey to help people understand machine learning,\u201d Huda says. \u201cWe\u2019re in finance and compliance. We need to have some rationale as to why we\u2019re taking an action. Explainability is important; when you\u2019ve caught an anomaly, you need to show how your model will catch all of the anomalies.\u201d<\/p>\n

Tiwari agrees, which is why the team is taking extra steps to try and explain findings.<\/p>\n

\u201cUnsupervised is a different experience than rules-based,\u201d Tiwari says. \u201cIt\u2019s not as easy to explain why the machine is telling you something, but we\u2019re relying on responsible AI practices to create transparency.\u201d<\/p>\n

An interconnected Microsoft based on rich insights<\/h2>\n

Insights now have the capability of being predictive.<\/p>\n

\u201cThere are lots of risks associated with finance and operations,\u201d Tiwari says. \u201cWe want to proactively identify them based on data alone, then begin to reduce risks overall.\u201d<\/p>\n

The agnostic nature of the model means that financial, compliance, and operational risks can all be recognized and addressed.<\/p>\n

\u201cWe\u2019re able to generalize the inference and training of our pipeline, allowing us to generate models specific to the different facets of the datasets,\u201d Chepkwony says. \u201cThis allows us to scale and provides insights we would never see in a rules-based model.\u201d<\/p>\n

The ability to harness interconnected datasets allows rich insights to be drawn from across Microsoft.<\/p>\n

\u201cRisk used to be viewed in a silo,\u201d Tiwari says. \u201cNow it can be viewed holistically, at scale.\u201d<\/p>\n

Huda sees even more opportunities by changing the review process.<\/p>\n

\u201cToday, really big and key accounts are reviewed first, insignificant entries are moved down the line in priority, and then there\u2019s the audit,\u201d Huda says. \u201cWith anomaly detection, everything is reviewed upstream and continuous.\u201d<\/p>\n

A bright future<\/h2>\n

What started as a research paper is now transforming the way Microsoft approaches anomaly detection.<\/p>\n

Artificial neural networks and unsupervised insights might appear daunting, but it\u2019s the right time to leverage new technology for innovative solutions.<\/p>\n

– Shilpa Tiwari, principal group engineering manager, Microsoft Digital<\/p>\n<\/blockquote>\n

Now that the model is able to successfully identify anomalies, the Finance Data Insight Team will continue to hone their insights with SME input on other datasets. They\u2019re only in the second leg of the journey, but for Tiwari and the Finance Data Insights Team, it\u2019s all upside.<\/p>\n

\u201cArtificial neural networks and unsupervised insights might appear daunting, but it\u2019s the right time to leverage new technology for innovative solutions,\u201d Tiwari says. \u201cNewer technology not only gives better outcomes it\u2019s also simpler.\u201d<\/p>\n

\"Related<\/p>\n