{"id":7851,"date":"2024-01-10T01:57:51","date_gmt":"2024-01-10T09:57:51","guid":{"rendered":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?p=7851"},"modified":"2024-01-10T10:47:58","modified_gmt":"2024-01-10T18:47:58","slug":"simplifying-microsofts-royalty-ecosystem-with-connected-data-service","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/insidetrack\/blog\/simplifying-microsofts-royalty-ecosystem-with-connected-data-service\/","title":{"rendered":"Simplifying Microsoft\u2019s royalty ecosystem with connected data service"},"content":{"rendered":"

\"MicrosoftIn any given month, Microsoft manages about 1.5 billion royalty transactions related to an estimated nine million products that span more than 3,000 partners, whose relationships are governed by over 5,000 active contracts.<\/p>\n

That\u2019s a lot of transacting.<\/p>\n

Through smart data assembly, natural language processing, and a custom Microsoft Teams bot, the engineers on Microsoft’s Royalties Team have created Royalties Assurance as a Service (RaaS), the company\u2019s new internal royalty transaction platform.<\/p>\n

Our royalties system is dynamic and complicated\u2014it processes multiple millions of transactions per day, dealing with hundreds of thousands of products, processing and calculating earned royalties according to the specific contracts in an accurate, timely, and compliant manner.<\/p>\n

\u2014Jagannathan Venkatesan, principal group engineering manager, Microsoft Global Payments and Cash<\/p>\n<\/blockquote>\n

\u201cIt\u2019s one thing to build an extensible and scalable system processing vast quantities of data spread across multiple dimensions and business rules to meet the accuracy, timeliness, and compliance needs of a multi-billion dollar business,” says Jagannathan Venkatesan, principal group engineering manager in Microsoft\u2019s Global Payments and Cash organization. \u201cIt is an entirely different challenge to be able to reduce that complexity through building a fully connected data system to bring it to a single canvas that is easy to interact with. With RaaS, our Royalties team was able to do just that with the help of RaaS APIs that enable system-to-system integration\u2014including Microsoft Teams integration\u2014and human augmented exploratory analytics.\u201d<\/p>\n

\"Mehrabi
Ehsan Mehrabi, a senior finance manager on the Royalties team, is among those using the transformed royalty transaction insights system. (Photo by Ehsan Mehrabi)<\/figcaption><\/figure>\n

Enterprise royalties, complex connections<\/h2>\n

Like many companies, Microsoft manages complex royalties relationships with other organizations. For example, the Microsoft Store sells Xbox games that leverage intellectual property from third party vendors. Or partners sell services through Microsoft that necessitate royalties based on consumption. In each of these cases, once an incoming transaction occurs and Microsoft has obtained the funds, a complex orchestration of calculations must take place to ensure each party receives the proper payments.<\/p>\n

\u201cRoyalties payouts are a critical aspect of Microsoft business, enabling our global digital content partners to realize the value of the products they have onboarded onto the Microsoft ecosystem,\u201d Venkatesan says. \u201cOur royalties system is dynamic and complicated\u2014it processes multiple millions of transactions per day, dealing with hundreds of thousands of products, processing and calculating earned royalties according to the specific contracts in an accurate, timely, and compliant manner.\u201d<\/p>\n

Gaining clarity throughout that entire system of relationships is essential for accuracy in accounting and payouts, and it is an integral part of generating organization-wide insights.<\/p>\n

\u201cWhen we generate a statement for a partner for a particular period, our system must be able to enable the business to walk back from the statement to products to transactions along with bringing appropriate contact and partner data including historical information,\u201d Venkatesan says.<\/p>\n

To achieve that level of clarity and trust in the system, the royalties team needed to aggregate the datasets underlying all of Microsoft’s royalties relationships and transactions, then make the results available in one easily accessible place.<\/p>\n

“The challenge is the complexity around calculating payouts and retrieving that information,” says Ehsan Mehrabi, a senior finance manager on the Royalties team. “We need to make sure everything is correct before payments go out the door or transactions get their accounting treatment.”<\/p>\n

The engineering team took up the challenge.<\/p>\n

[Learn how Microsoft’s finance team uses anomaly detection and automation to transform royalty statements processing<\/a>. Find out how Microsoft designed a modern data catalog to enable business insights<\/a>. Explore how AI and chatbots simplify finance tools at Microsoft<\/a>.]<\/em><\/p>\n

Transforming Microsoft\u2019s royalties ecosystem with a connected data system<\/h2>\n

Unifying the 300 to 400 million financial data points that flow in and out of the company each month was an enormous undertaking for the royalties engineering team.<\/p>\n

It encapsulated three main challenges.<\/p>\n

The first and most complicated task was defining and canonicalizing the millions of data points associated with the royalties business. The engineering team needed to use automation to identify distinct entities that could be assembled and visualized as a graph of connected data-points.<\/p>\n

Defining words such as “contract” and “product” seems like a simple job, but it presents challenges when it comes to automating data in complex business relationships. The data definitions needed to reflect the royalties team’s business needs and be simultaneously consumable by data processing systems.<\/p>\n

\"Janam
Ram Janam Singh (left to right), Sundeep Ratnagiri, Abhijit Mandal, and Jagannathan Venkatesan worked with other engineers on Microsoft’s Royalties Team to develop a royalty transaction insights system to aggregate data holistically through a Microsoft Teams bot, simplifying complex information and providing organization-wide insights. (Photo by Rajmohan Venkatesan)<\/figcaption><\/figure>\n

Sundeep Ratnagiri, engineering manager for Microsoft Royalties, outlines how the team defines these terms, explaining what the word \u201ccontract\u201d means when it comes to managing Microsoft\u2019s royalties system.<\/p>\n

\u201cFor a businessperson, a contract is a legally binding document that defines business terms,\u201d Ratnagiri says. \u201cFor an engineer, it is a set of parameters codified in the system to function the way the legal document is written. Similarly, a product is an asset that is transacted upon, with rich attributes that can be referenced in a contract.\u201d<\/p>\n

From the start, engineers partnered with their peers across the royalties business and engineering landscape, including the accounting, business, and partner engineering teams. They spoke to a wide array of stakeholders to ensure they could assemble the system’s 300-400 million data point connections per month in ways that would support everyday usage. The result was a single, connected data output with analytical (like aggregation, for example) capabilities powered by the team\u2019s different processing calculation systems (also for example).<\/p>\n

The second major task was to represent the different data sets in a connected graph exposed with a single API set, enabling team members to navigate from any point of the royalties system to anywhere else. The engineers utilized Apache Spark for the data modeling pipeline, then modelled it as a graph of connected entities using Microsoft Azure Cosmos DB. The result was a trustworthy, independently validated source for all canonical data that was ready for access and interpretation.<\/p>\n

“The natural connective tissue across all these platforms exists,” Ratnagiri says. “Some are straightforward connections. Others are inferred connections. When we link them up, it opens a plethora of analytics.”<\/p>\n

The data wouldn’t be helpful to anyone if it wasn’t available for queries, so the team’s third task was enabling access through an API layer. The business users wanted the system to output expressive, incremental information when they submit queries, so they included natural language support in the API.<\/p>\n

Like any search tool, the API’s query terms needed to seem natural enough to be intuitive to users but sufficiently rich to accomplish the full range of possible queries. So, the engineering team interviewed stakeholders to define the most relevant search activities and build a series of canonical queries. Each of these queries sets the API off on a traversal through the entire Microsoft Azure Cosmos DB graph to locate and assemble the relevant data for the user.<\/p>\n

When customers start to look at connecting multiple data sets, it is important to spend an appropriate amount of time early in the project on entity modeling and relationship curation across these entities. On the storage side, it is particularly important to pick the right partition key on the Cosmos DB side. This can have a significant impact on the latencies of queries in terms of defining in-edges and out-edges.<\/p>\n

\u2014Abhijit Mandal, senior software engineer, Microsoft Royalties<\/p>\n<\/blockquote>\n

To maximize accessibility, the team built access to the API layer into a Microsoft Teams bot. Together, the team calls this end-to-end data solution Royalties as a Service (RaaS). Despite the system’s complexity, the outcomes are all about simplicity and empowerment.<\/p>\n

Additionally, the API layer enforces security and confidentiality perimeters depending on who is using the system and what permissions they have.<\/p>\n

Query execution<\/h2>\n

A user simply navigates to RaaS within Microsoft Teams and submits a natural language query like “payee 100010 drilldown” or “contract <abccdd<\/em>> assurance.” This query passes through several different stages of execution in the pipeline before results are assembled and shown to the user on the Teams bot UI canvas.<\/p>\n

These stages in order are:<\/p>\n

    \n
  1. Entity resolution:<\/em>
    \nThe natural language query is parsed to     extract entities<\/a>, sub-entities, and values. This is done using Azure cognitive service\u2014Language Understanding Intelligence service (LUIS)<\/a>. Related entities are extracted as relationships and used in graph traversals. For the query \u201cpayee 100010 drilldown,\u201d the entities and entity values extracted are \u201cPayee\u201d:\u201c100010\u201d<\/li>\n
  2. Intent formation:<\/em>
    \nIntents are formed from the LUIS layer as well. Along with parsed entities, the user-intended action is added to form the     intent object<\/a>.<\/li>\n
  3. Dynamic Gremlin query generation:<\/em>
    \nThe intent object is passed through a query generation layer. The layer converts an intent object to a gremlin query that can be executed against a Cosmos graph DB instance. This is an example of a dynamic gremlin query:<\/p>\n
    Example one:\r\n\r\ng.V().hasLabel('payee').has('payeeid','100010').range(0,1000).as('ct')\r\n\r\n.select('ct')\r\n\r\n.local(properties('column1','column2','column3','column4').group().by(key()).by(value())).dedup()<\/pre>\n
    Example two: The query below applies a contract ID filter on a contract node and traverses from the contract node over to product across connected edges, selecting the products associated with the contract.<\/p>\n
    g.V().hasLabel('contract').has('contractid','1000010').as('contract')\r\n.outE('contract_to_product').inV().as('product')\r\n.select('product')\r\n.local(properties('column1', 'column2', 'column2', 'column3', 'column4',\r\n'column5').group().by(key()).by(value())).dedup()<\/pre>\n<\/li>\n
  4. Gremlin query execution:<\/em>
    \nThe final stage in the query layer is the execution of the dynamic gremlin query and converting the response to JObject of the relevant entities being selected.<\/li>\n<\/ol>\n
    The intent of sharing what a query looks like is to give customers an example of how they could tackle something similar in related efforts.<\/p>\n
    \u201cWhen customers start to look at connecting multiple data sets, it is important to spend an appropriate amount of time early in the project on entity modeling and relationship curation across these entities,\u201d says Abhijit Mandal, a senior software engineer working on the platform. \u201cOn the storage side, it is particularly important to pick the right partition key on the Cosmos DB side. This can have a significant impact on the latencies of queries in terms of defining in-edges and out-edges.\u201d<\/p>\n
    The RaaS system today serves queries within sub-second latencies over a graph of 32 million entities connected through 110 million relationships. It\u2019s been a long, important journey to launch RaaS, one that brought together disconnected tools that Microsoft uses to manage the agreements and relationships that define the company\u2019s underlying royalties.<\/p>\n
    Aggregation and insights powered by connected data<\/h2>\n
    Previously, users pulled data directly from several different sources, assembled it into meaningful formats, and validated the information through several layers of manual cross-checking. This was onerous for the engineering team\u2014they had to understand each request, craft appropriate queries and mechanisms to harvest the data, and collate and aggregate the queries so they would be available to the business for further handling.<\/p>\n
    Sometimes, that process had a multi-day cycle time.<\/p>\n
    I’ve always thought of RaaS as a data-quality tool. This knowledge is now baked into a system. We’re getting a reliable answer through a unified process because it’s been structured properly.<\/p>\n
    \u2014Chris Roozen, senior project manager, Microsoft Royalties<\/p>\n<\/blockquote>\n
    \"Chris
    Chris Roozen, a senior project manager on the Royalties team, says the biggest benefit of RaaS is how it gives the team better data insights. (Photo by Chris Roozen)<\/figcaption><\/figure>\n
    \u201cWith our RaaS system, retrieval and presentation of relevant information is automatic and driven by the end user with no time lost on the engineering and business sides, with the additional advantage of eliminating human error,\u201d Venkatesan says.<\/p>\n
    That means it\u2019s more difficult for errors to be entered into the system and that access is improved, which boosts accuracy and improves user satisfaction.<\/p>\n
    \u201cI\u2019ve always thought of RaaS as a data-quality tool,\u201d says Chris Roozen, senior project manager on the Royalties team. \u201cThis knowledge is now baked into a system. We\u2019re getting a reliable answer through a unified process because it\u2019s been structured properly.\u201d<\/p>\n
    The easier it is to get a clear picture of individual data pools, the simpler it is to look at the big picture and gain business-wide insights.<\/p>\n
    Opening up the connected data landscape<\/h2>\n
    For now, RaaS is a relatively new capability on the Royalties team. As the internal experts on the query tool, engineers are RaaS’ primary frontline users, handling queries for the rest of the royalties business to help validate their data. In the future, they hope to simplify the search process with intelligent and predictive searches so it’s more user-friendly for non-engineers. In that scenario, anyone on the team will be able to submit queries and navigate the aggregated data independently.<\/p>\n
    Because team members will source their information through RaaS queries, fewer people will need access to the original data sourcing utilities. Limiting access to those tools helps decrease compliance risk within a large organization like Microsoft.<\/p>\n
    Similarly, as natural turnover occurs on the team, administrators won’t have to juggle access and training for multiple complicated data tools. Instead, RaaS will help all team members spend their time where it’s most valuable: validating data and building business insights.<\/p>\n
    There’s even the possibility of an outward-facing portal that customers, partners, and vendors can access to benefit from the ease and transparency that RaaS provides for Microsoft. But for now, RaaS is already demonstrating its value by saving time, eliminating error, and providing holistic insights.<\/p>\n
    \"Key<\/p>\n