Content downloads.<\/li>\n<\/ul>\nWhen someone signs up for a Microsoft product trial, sends us an email, or downloads content, the person or business becomes a\u00a0lead<\/i>\u00a0for the purposes of sales and marketing. We market to between 5 million and 10 million leads per year. These names\u2014which can be a company or a person\u2014become records in our marketing and sales system. Our salespeople take the leads they receive from our marketing and sales system and work to convert the unknown prospective customer into a sales-qualified opportunity (SQO): a customer who is ready to buy Microsoft products or services.<\/p>\n
As a sales organization, our primary goal is to be as effective as possible in taking unknown leads and converting them to Microsoft customers\u2014users and purchasers of our products. We begin the process by identifying who the unknown entity is, by obtaining the name, organization, and other basic information. We then take this prospective customer and try to ascertain how likely they are to purchase Microsoft software. We do this by assessing qualifying information in a continuing process that attempts to move them from a prospect, to a lead, to an identified sales opportunity, and finally, to a sales win and customer success.<\/p>\nFigure 1. The commercial customer journey through the Microsoft sales process<\/figcaption><\/figure>\nHistorically, the leads that our salespeople receive come with very little qualification. Eighty percent of our leads do not respond to an initial contact from our salespeople. A lead might have been generated by using false or incomplete information, or a customer might have no intention of purchasing products when they download a product trial. As a result, our salespeople had to review up to 2 million leads annually to identify commercial customers who are ready to purchase a product. Leads that didn\u2019t respond were abandoned, and salespeople pursued other leads on the list.<\/p>\n
Our sales and marketing team knew that they needed a better way to qualify the massive amount of leads that our salespeople were processing. We wanted our salespeople to focus on selling. They were spending too much time trying to find prospective purchasers and not enough time focusing on customers who were ready to buy. We needed a more intelligent solution.<\/p>\n
Initiating a more intelligent lead-qualification process<\/h2>\n To improve the leads\u2019 quality for our salespeople, we needed be able to define and measure that success. We established several goals that we wanted to achieve in improving the lead-qualification process by using AI and machine learning:<\/p>\n
\nIncrease revenue.<\/strong>\u00a0As a sales organization, our objective is to sell. If more customers buy our products and services, revenues increase, making Microsoft stronger and more profitable.<\/li>\nAccelerate sales.<\/strong>\u00a0Simply put, we wanted to move our customers from interest to product purchase more quickly. High quality, ready-to-buy leads remained untapped in our sales and marketing system because we had no way to move them to the top of the leads list.<\/li>\nIncrease sales system throughput.<\/strong>\u00a0We wanted our salespeople to be able to process more leads. By processing up to 10 million leads per year, our salespeople worked through tens of thousands of leads each day. It took time to contact and dismiss leads that wouldn\u2019t result in sales, and we wanted our salespeople to be able to recoup that wasted time and use it more efficiently.<\/li>\nImprove the customer experience.<\/strong>\u00a0We wanted to be able to better understand customers and ensure that their needs are met. This would require analyzing the information our leads contained more thoroughly and better understanding each lead\u2019s underlying data.<\/li>\n<\/ul>\nOur primary measure of success for lead conversion is the rate at which our sales team converts a lead to a sales-qualified opportunity (SQO) in MSX: someone ready to purchase. Before we started using AI to qualify leads, our sales team was converting approximately 4 percent of the between 1 million and 2 million commercial leads annually into SQOs. We wanted to improve that rate significantly.<\/p>\n
Considering AI for lead-qualification improvement<\/h3>\n The journey into AI-driven lead qualification started before AI became part of our lead-assessment process, and it has occurred in three distinct phases:<\/p>\n
\nBusiness rules.<\/strong>\u00a0This where we started, before AI was incorporated into lead qualification. Our sales and marketing system contained business rules that provided a relative value indicating the likelihood of a lead being converted to a sales opportunity. These rules assigned a point value to a customer to create a lead-scoring structure.<\/li>\nAI-assisted lead scoring.<\/strong>\u00a0During this phase, we implemented AI to improve our lead-scoring system. AI lead scoring focuses on customer behavior and assigns values to the different ways that a customer interacts with Microsoft, creating a more accurate scoring result.<\/li>\nBot-driven engagement and conversational intent detection.<\/strong>\u00a0The most recent phase involved including an AI bot and a machine-learning-driven engine that examines and manages email communications between the customer and our sales team. We\u2019ve named the engine Bot (and AI) Enabled Augmented Marketing (BEAM). BEAM focuses on using conversational intent detection to determine the likelihood that a customer is ready to purchase our products or services.<\/li>\n<\/ol>\nFigure 2. Improving our sales and marketing processes with AI-powered lead qualification<\/figcaption><\/figure>\nExamining our pre-existing basic business rules<\/h3>\n Prior to AI and machine learning, we used basic business rules that we configured in our sales and marketing system to identify incoming leads. It provided a basic scoring system to help our salespeople choose leads that might yield sales opportunities. However, the scoring system\u2019s business rules couldn\u2019t account for the wide variety of lead quality and intent, so it still produced many leads that weren\u2019t interested in purchasing Microsoft products. Our conversion rate to SQO under this system was approximately 4 percent.<\/p>\n
Improving basic lead qualification with AI lead scoring<\/h3>\n Our first step in improving lead quality was to implement AI-assisted lead scoring. We designed the scoring system to examine the different aspects of incoming leads and assign relative values to those aspects as they related to lead quality. AI lead scoring improved on the static, business-rules-based system by focusing on behavior and attributing a dynamic value to a lead based on that behavior. Our algorithm accounted for important information, such as:<\/p>\n
\nDemographic information, including region or industry.<\/li>\n The lead\u2019s role in their organization as an influencer or business decision maker.<\/li>\n Prior history with Microsoft, including purchase history and contribution to product reviews and product feedback.<\/li>\n Recent activity that the lead has performed, such as downloading additional content, responding to a Microsoft promotional email, attending a Microsoft event, or filling out an online form.<\/li>\n<\/ul>\nThe system generated a score for each lead based on algorithm output, and then stored that information in our sales and marketing system. Leads given a higher score were pushed to the top of the list while lower scoring leads were placed at the bottom. The results provided our salespeople with a ranked list from which they could take the highest-scoring leads and engage the customer.<\/p>\n
We designed the scoring to go deeper into business processes to create context around a lead, thus enabling our salespeople to better respond to the lead. It helped us begin to move from high-volume, low-quality leads to high-quality, scored leads. Our sellers could now focus on the leads that were most likely to result in an SQO. However, we recognized even more potential in AI to improve the lead qualification further. The scoring process often missed leads that didn\u2019t fit into the typical lead-qualification categories, even though the customer was ready to buy. In the past, such leads remained at the bottom of the list, where a long time might elapse before a salesperson pursued them. It also left leads in the system that exhibited the behaviors of a potential customer but one who was not interested in purchasing. For example, someone studying for certification in one of our products might sign up for trial software and attend workshops but does not intend to make any purchases. While AI lead scoring improved our SQO efficiency from 4 percent to 6 percent, we recognized the potential to add more intelligence into our qualification process.<\/p>\n
Facilitating intelligent intent detection using BEAM<\/h2>\n BEAM is the centerpiece of our current AI-based lead-qualification solution. We built BEAM as an orchestration engine to capture incoming lead contact information through email and automate the contact process, determine lead intent by using machine learning, and provide a clearer lead-ranking process.<\/p>\n
The basic BEAM process consists of the following steps:<\/p>\n
\nTake leads from our marketing and sales system and send an automated email asking how the salesperson might best help the customer.<\/li>\n Ingest email response and process email data for intent detection. This involves preparation to ensure that the data is accurate and useable.<\/li>\n Use machine learning to analyze email content and detect intent and context.<\/li>\n Assign intent to a lead.<\/li>\n Forward intent to a validation user interface to confirm intent classification.<\/li>\n Process leads in preparation for injection into the sales and marketing system.<\/li>\n Place the leads back into the sales and marketing system with appropriate intent and scoring information attached.<\/li>\n<\/ol>\nUnderstanding intent and context detection<\/h3>\n BEAM\u2019s intent-detection engine examines and identifies specific phrases as indicators of a customer\u2019s intent to purchase products and services. For example, BEAM might recognize the phrase \u201cwondering about pricing\u201d as an indicator that a lead is ready to purchase. On the other hand, a phrase such as \u201cNever contact me again\u201d is a pretty clear indicator that a customer is not interesting in purchasing. BEAM uses machine-learning models to perform two assessments of the underlying data:<\/p>\n
\nIntent detection.<\/strong>\u00a0Using natural language processing, intent detection assesses content and matches it with an intended task or direction. In the case of BEAM, we have four intent classifications for leads:\n\nHot.<\/strong>\u00a0A lead is classified as\u00a0hot\u00a0<\/i>if the customer is ready to purchase. Usually language in the email, such as questions about pricing, licensing or availability, indicates clear intent to purchase. Leads classified as hot automatically go to the top of the sales queue.<\/li>\nNot Interested.<\/strong>\u00a0BEAM classifies a lead as\u00a0not interested<\/i>\u00a0if there is clear indication that a customer is not ready to buy at this time but should remain in the system for potential purchase in the future. These leads are removed from the sales pipeline at the current time.<\/li>\nDo not contact.<\/strong>\u00a0If an email contains language indicating a clear desire for no contact or phrases that indicate anger or frustration, BEAM removes the lead from the sales pipeline and our marketing consent list.<\/li>\nNone of the above.<\/strong>\u00a0If the lead doesn\u2019t fall into clear intent categories, the lead is classified as\u00a0none of the above<\/i>\u00a0and is taken out of the sales pipeline and, in most cases, the original message is redirected to an appropriate channel based on context.<\/li>\n<\/ul>\n<\/li>\nContext detection.<\/strong>\u00a0We use context detection to provide another layer to BEAM\u2019s decision-making capabilities. Many of the emails that come in through the BEAM bot are not from customers seeking sales help. They could be support requests, questions about feature education opportunities, solicitations, or other messages unrelated to the sales pipeline. Context detection uses natural language processing to determine what contexts are contained in a message and use those contexts for redirection, more detailed classification, or other decision-making capabilities.<\/li>\n<\/ul>\nExamining BEAM architecture and infrastructure<\/h3>\n The BEAM platform is built on Microsoft Azure technologies and open-source AI and machine learning tools, including Microsoft Cognitive Services, Azure Machine Learning Server, and Azure Machine Learning Studio\u00a0and then integrated into Dynamics 365. We built BEAM to be modular and flexible; we can interchange components and modify component behavior without negatively affecting the solution as a whole or refactoring.<\/p>\n
<\/p>\nFigure 3. The conversational AI sales intent detection architecture model for BEAM<\/figcaption><\/figure>\nIngesting and preparing data for machine learning input<\/h4>\n One critical component of our BEAM architecture model is data preparation and cleanup. The machine-learning models for BEAM accept only plain-text inputs. The plain text is parsed in its entirety for intent and context detection. All BEAM inputs come from an email source, which has its own inherent data-quality issues when converted to plain text. Several parts of the email message must be stripped from the plain text input because they provide no value in intent detection and might cause issues with model-training behavior. These include:<\/p>\n
\nEmail greetings, signatures, and sign-offs.<\/li>\n Text included as part of a reply.<\/li>\n New line characters, formatting characters, and other unnecessary characters.<\/li>\n<\/ul>\nWe use Bing Spell Check in Microsoft Cognitive Services to spell check incoming email text to ensure proper recognition by the machine-learning models. We also created code to remove private personal information from ingested text.<\/p>\n
Machine-learning architecture and models<\/h4>\n At BEAM\u2019s core are its machine-learning models. We use several models that each provide intent detection and work together to provide BEAM functionality.<\/p>\n
Language understanding with LUIS<\/h5>\n Our first model, and the one that does the most intensive work, is the Language Understanding Intelligent Service (LUIS) machine-learning service from Microsoft Cognitive Services. LUIS is a natural-language processing model designed to identify valuable information in conversations, interpret intent, and distill information from sentences for a high-quality, nuanced language model. LUIS integrates seamlessly with several API interfaces, which makes it extremely easy to incorporate into BEAM.<\/p>\n
We created a layer in BEAM that sends API calls to LUIS by using JavaScript Object Notification (JSON). The JSON code is the primary input for LUIS, and it contains the prepared, cleaned data that LUIS ingests. A second layer translates the LUIS output into a standard response, in JSON, that is consumed by the rest of the BEAM architecture.<\/p>\n
We\u2019ve used LUIS to create applications in BEAM for both intent detection and context detection. LUIS\u2019 out-of-the-box usability made it easy to quickly integrate LUIS-based applications into BEAM. LUIS also provides a flexible base for building machine-learning applications. Even though context detection isn\u2019t one of the natively supported uses for LUIS, we were able to leverage the service to create an easy-to-integrate context-detection service for BEAM that could run alongside the intent detection engines (including LUIS) that we were using in BEAM. We\u2019re continually examining LUIS for more opportunities to build other functionality into BEAM.<\/p>\n
Custom-coded models from scikit-learn<\/h5>\n We also implemented three separate models from scikit-learn, an open-source data mining and analysis tool set that performs machine learning in Python. We host each model in Azure Machine Learning Server, which enables us to deploy the model to whatever scale we require and keep our BEAM environment hosted entirely on the Azure platform. We\u2019re using the following scikit-learn models:<\/p>\n
\nMultinomial Na\u00cfve Bayes (MultinomialNB)<\/li>\n C-Support Vector Classification (SVM.SVC)<\/li>\n Linear classifiers with stochastic gradient descent (SGD) training (SGDClassifier)<\/li>\n<\/ul>\nEach of these models provides an evaluation of the input data from a distinct perspective and provides an equally weighted output that contributes to the determined intent.<\/p>\n
Combining machine learning output with ensemble methods<\/h5>\n Each machine-learning model produces a discrete result, indicating the evaluated intent from a lead email. We use ensemble methods meta-algorithms to assess all four models\u2019 results and provide output to the BEAM engine from the model that generates the most accurate results. Ensemble methods allow us to freely insert and remove machine-learning models into the BEAM pipeline. We can test model effectiveness more quickly and adapt our machine learning practices according to our business needs.<\/p>\n
We use the Ensemble Majority_MaxOrder method, which considers all model inputs and chooses the intent that received the majority vote from all models. If there is a contention (two intents have an equal number of votes), the one with the highest probability score is chosen. If several have the same probability score, the order of the models determines which one is chosen. We are continually evaluating the models\u2019 output and considering other potential ensemble methods as BEAM intent detection matures.<\/p>\n
Training machine-learning models and providing classification feedback<\/h4>\n Our models are continually trained through an automated process that uses the previous month\u2019s data and tests model output against the current month\u2019s data. We also provide model training through our manual intent review portal, where our validation team can review BEAM output for validity and accelerate the training process.<\/p>\n
This manual training process was extremely important in the early stages of BEAM implementation, and continued use helped our machine-learning models make more accurate predictions very quickly. We also have an automated review process that takes obvious intent phrases such as \u201cDo not contact me\u201d and assigns the appropriate classification.<\/p>\n
Surfacing AI within Dynamics 365<\/h3>\n Once we have an intelligent lead qualification score for every lead, it is passed on to our MSX system, running on Dynamics 365. Using Dynamics 365 as our sellers\u2019 main productivity tool and surfacing intelligent insights into that tool creates the best of both worlds for our sales force. Dynamics 365 and AI operate together to give sellers a view of their prioritized workload based on the highest potential customers in the opportunity pipeline.<\/p>\n
Sellers now have the information they need to optimize their activities in their day, week and month. Sellers have a personalized view of prioritize opportunities. These are opportunities that are unique to them based on the product they focus on and their country and location. And they get to see the entire BEAM conversation with the customer so that can understand the full context of the customers\u2019 needs before they place a call to that customer. This is just a start. We will continue to infuse AI into MSX, augmenting our sellers\u2019 capabilities, so they can focus on the most important aspect of their job: taking care of their customers.<\/p>\n
Implementing BEAM at Microsoft<\/h3>\n We completed BEAM implementation over a six-month period at Microsoft. We provided proof-of-concept (POC) data and results, and then ran a two-month pilot project with 180 customers in two regions. The pilot allowed us to production test our architecture and end-to-end processes. We then rolled out BEAM to the rest of our sales organization over the ensuing three months. In the time that we\u2019ve had to review its effectiveness, the combination of AI lead scoring and BEAM has more than quadrupled our SQO conversion rate, from 4 percent to 18 percent\u00a0in MSX.<\/p>\n
Benefits<\/h2>\n BEAM and AI-based lead scoring have radically changed our salespeople\u2019s ability to identify and pursue productive sales leads and deliver Microsoft products and services to more customers far more efficiently. We\u2019ve experienced several specific benefits from BEAM and AI-based lead qualification, including:<\/p>\n
\nA better sales environment.<\/strong>\u00a0We\u2019ve increased our sales conversion rate from 4 percent to 18 percent, and that means a stronger sales organization across the board.<\/li>\nIncreased revenue.<\/strong>\u00a0Our salespeople are providing product and service sales to more customers because of the improved lead-qualification efficiency. They\u2019re pursuing high-value customers who are ready to purchase products and services.<\/li>\nAccelerated sales.<\/strong>\u00a0We\u2019re moving our customers from product interest to product purchase more quickly. High-quality leads are identified quickly, and our salespeople can deliver products and services more rapidly.<\/li>\nIncreased sales-system throughput.<\/strong>\u00a0Our salespeople process more leads. They don\u2019t waste time on low-quality leads or chase customers who aren\u2019t interested in purchasing our products.<\/li>\nImproved customer experience.<\/strong>\u00a0Our customers get their needs met better and more quickly. We also have a better understanding of who are customers are and what they want.<\/li>\n<\/ul>\nBest Practices<\/h2>\n\nClean your inputs.<\/strong>\u00a0Machine learning thrives on clean, accurate data. We worked hard to ensure that our inputs to BEAM were as clean as possible before relying on BEAM output for lead qualification. We factor in legal compliance considerations such as General Data Protection Regulation (GDPR) and ensure that our customers\u2019 information remains private and protected.<\/li>\nUse engineering best practices.<\/strong>\u00a0Modern machine-learning interfaces and design patterns are built on modern development practices such as continuous integration and continuous deployment (CI\/CD), agile development, and effective testing and logging practices. Adopting these in BEAM made it easier to change and grow the solution with our business needs.<\/li>\nAdopt a modular approach to machine-learning components.<\/strong>\u00a0Machine learning is an ever-changing field, and new models and algorithms appear often. We designed BEAM as an open platform to allow the use of any machine-learning models in the lead-qualification pipeline. This allows for simple insertion, testing, and removal models and the potential expansion of BEAM functionality in the future.<\/li>\n<\/ul>\nLooking forward<\/h2>\n We\u2019re excited about future possibilities for BEAM and other AI and machine-learning-based development at Microsoft. We plan to improve and expand in several areas:<\/p>\n
\nImproved context detection. We want to dig deeper into our data and move from simple determinative results to deeper composite results. For example, we want to be able to detect not only that a customer is out-of-office, but also when the customer will return.<\/li>\n Increased automation. We\u2019re investigating ways to create additional automation levels within BEAM, such as enabling automated meeting scheduling for our customers with our salespeople to discuss and finalize product sales.<\/li>\n Expanded languages. We want to expand BEAM to process the native language of any of our customers.<\/li>\n<\/ul>\nConclusion<\/h2>\n Infusing AI and machine learning into Dynamic 365 for\u00a0the lead-qualification process has greatly improved the effectiveness of our sales team, creating a more intelligent and efficient method for identifying potential customers. AI lead scoring and BEAM\u00a0integrated into MSX have significantly improved the effectiveness of our sales pipeline by providing our sales team with higher quality leads that are ready to buy our products and grow their relationship with Microsoft. AI and machine learning have enabled our digital transformation, allowing us to empower our employees to achieve more and optimize our lead qualification process.<\/p>\n","protected":false},"excerpt":{"rendered":"
This content has been archived, and while it was correct at time of publication, it may no longer be accurate or reflect the current situation at Microsoft. At Microsoft, our sales and marketing team receives millions of potential sales leads each year that they must follow-up with to determine how they can best meet each […]<\/p>\n","protected":false},"author":146,"featured_media":10447,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"_hide_featured_on_single":false,"_show_featured_caption_on_single":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[1],"tags":[],"coauthors":[674],"class_list":["post-10445","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","m-blog-post"],"jetpack_publicize_connections":[],"yoast_head":"\n
Microsoft increases sales by using AI for lead qualification - Inside Track Blog<\/title>\n \n \n \n \n \n \n \n \n \n \n \n \n\t \n\t \n\t \n \n \n \n\t \n\t \n\t \n