{"id":5856,"date":"2020-11-08T17:53:28","date_gmt":"2020-11-09T01:53:28","guid":{"rendered":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?p=5856"},"modified":"2023-06-28T10:11:03","modified_gmt":"2023-06-28T17:11:03","slug":"getting-data-reliability-right-at-microsoft-with-scalable-engineering-solutions","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/insidetrack\/blog\/getting-data-reliability-right-at-microsoft-with-scalable-engineering-solutions\/","title":{"rendered":"Getting data reliability right at Microsoft with scalable engineering solutions"},"content":{"rendered":"
\n
\n
<\/div>\n

This content has been archived, and while it was correct at time of publication, it may no longer be accurate or reflect the current situation at Microsoft.<\/p>\n<\/div>\n<\/div>\n

After its employees, data is arguably Microsoft\u2019s most valuable asset. The company relies on data to digitally transform how it operates its businesses, develop products and services, and upgrade the experiences it offers to customers and employees.<\/p>\n

To harness the full potential of its data, Microsoft needed a scalable and supportable way to manage these vast flows of information that are the lifeblood of the company.<\/p>\n

Praveen Krishnan is one Microsoft employee who needed a tool to manage his team\u2019s data. An engineering manager for the Marketing Data and Insights team at Microsoft, he manages the team that owns marketing data and insights used for revenue-based marketing. This data is collected at marketing events and via marketing campaigns, and it\u2019s used to create personalized customer experiences that support engagement and retention.<\/p>\n

\u201cWe use the insights and intelligence from this data to empower our customers to use Microsoft technology to accelerate their transformation,\u201d says Praveen, a senior software engineering manager in Microsoft Digital.<\/p>\n

Clearly, customer data is very sensitive and must be secured and well managed\u2014that\u2019s the top priority of Praveen and his team.<\/p>\n

\u201cWe need strict privacy controls in place, so we only enable access to the minimum data required for each use case,\u201d Praveen says. \u201cI always ask, \u2018Who can access the marketing data that I manage, and why do they need it? What\u2019s the potential business impact?\u2019\u201d<\/p>\n

To answer these questions, Praveen used to have to track down all the teams and users at Microsoft who had used his data and prevent unnecessary data copies. This was a largely manual process.<\/p>\n

\u201cThere has to be a better way,\u201d he remembers thinking.<\/p>\n

Data reliability is foundational to enabling responsible data democratization and developing impactful data applications. We saw an opportunity to deliver proactive and scalable engineering solutions for enterprise data management and operations.<\/p>\n

– Gowri Krishnan, senior software engineer on the Microsoft Digital team<\/p>\n<\/blockquote>\n

Praveen wasn\u2019t alone in this problem. There was no easy way to track all the sources, copies, and use of data across the company. Many teams would often end up generating data and creating data products with the same goals. Such duplication caused data proliferation and increased the risks of data exposure.<\/p>\n

Gowri Krishnan, a senior software engineer, and her MS Digital Data team were ready to take on these challenges.<\/p>\n

\u201cSince teams did not have full visibility in how their data was used, there were multiple copies of the same data across the company,\u201d Gowri says. \u201cPreventing proliferation required manual and time-intensive data governance practices that were hard to scale.\u201d<\/p>\n

Such risks are hard to recover from, especially when they\u2019re found further into the production process.<\/p>\n

\u201cWith the scale of our data and the complexity of data exposure risks, it can be challenging to detect and act on every data management risk across Microsoft\u2019s data estate,\u201d Gowri says.<\/p>\n

It was clear that employees like Praveen needed scalable engineering solutions for holistic data reliability to secure their data assets, prevent data proliferation, and enable compliant use of their data.<\/p>\n

\u201cData reliability is foundational to enabling responsible data democratization and developing impactful data applications,\u201d Gowri says. \u201cWe saw an opportunity to deliver proactive and scalable engineering solutions for enterprise data management and operations.\u201d<\/p>\n

Enterprise Data Reliability Engineering (DRE) is Microsoft\u2019s approach to scale its data management and operations capabilities. The DRE approach is centered on proactive and scalable engineering solutions for data reliability, characterized by a secure, discoverable, high-quality, compliant, and operationally efficient enterprise data estate.<\/p>\n

“<\/strong>Enterprise DRE offers scalable solutions for each facet of big data operations,\u201d Gowri says. \u201cIt benefits everyone along the data management pipeline from data publishers to data consumers.”<\/p>\n

[<\/em>Read about how Microsoft turned to DevOps engineering practices to democratize data access at Microsoft.<\/em><\/a> Learn how Microsoft powers digital transformation with modern data foundations.<\/em><\/a> Find out how Microsoft unleashes the power of data with a modern data platform.<\/em><\/a>]<\/p>\n

Democratizing enterprise data management<\/strong><\/p>\n

The DRE team\u2019s goal is to deliver scalable and intelligent engineering solutions to proactively prevent such risks.<\/p>\n

\u201cOur goal is to deliver scalable solutions as DRE foundations that enable every data publisher and data consumer at Microsoft to contribute to our enterprise data estate and adhere to our data management standard,\u201d Gowri says.<\/p>\n

The DRE foundation gives me actionable insights on my data estate and on how my users are using my data. With such optics in a single place, I can easily monitor the health of my data estate and use the built-in controls to fulfill essential data compliance and governance requirements.<\/p>\n

– Praveen Krishnan, engineering manager on the Marketing Data and Insights team at Microsoft<\/p>\n<\/blockquote>\n

Gowri and her team are onboarding teams in Microsoft Digital and across Microsoft onto the DRE.<\/p>\n

\u201cWith DRE, we want to proactively detect, mitigate, and resolve data management risks that impact data reliability,\u201d Gowri says. \u201cWe wanted to have scalable solutions with built-in intelligence to proactively prevent, detect, and address data security, reliability, and compliance risks. This enables us to scale data applications to support digital transformation at Microsoft.\u201d<\/p>\n

The DRE foundation connects employees like Praveen with a single integrated view of data assets at Microsoft with insights on their management and operations health, to proactively detect and mitigate data reliability risks with automated solutions and human actions. This is achieved by capturing telemetry related to all facets of data management and detecting and mitigating anomaly conditions by triggering automated actions and human engagement workflows.<\/p>\n

\u201cThe DRE foundation gives me actionable insights on my data estate and on how my users are using my data,\u201d Praveen says. \u201cWith such optics in a single place, I can easily monitor the health of my data estate and use the built-in controls to fulfill essential data compliance and governance requirements.\u201d<\/p>\n

Praveen, and soon others who manage data at Microsoft, can use DRE to gain full line-of-sight visibility of their data estate, the health of their data assets, and the use of their data by teams across Microsoft. Such insights enable Microsoft teams to manage their data estates efficiently and with compliance.<\/p>\n

\u201cRather than building and operating services to ensure that our data is compliant, we can focus our engineering investments on differentiated marketing data applications,\u201d Praveen says.<\/p>\n

Building an enterprise DRE foundation<\/strong><\/p>\n

Building reusable foundations to enable the proactive management of such data reliability facets is key to enabling an organization to responsibly scale its data use for greater value outcomes and impact.<\/p>\n

\u201cData management solutions must be scalable across an enterprise,\u201d Gowri says. \u201cThey must be built and operated with consistency and as shared capabilities.\u201d<\/p>\n

The enterprise DRE foundation is made of the following components:<\/p>\n