{"id":10445,"date":"2019-04-09T13:49:20","date_gmt":"2019-04-09T20:49:20","guid":{"rendered":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?p=10445"},"modified":"2023-06-12T16:03:25","modified_gmt":"2023-06-12T23:03:25","slug":"microsoft-increases-sales-by-using-ai-for-lead-qualification","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/insidetrack\/blog\/microsoft-increases-sales-by-using-ai-for-lead-qualification\/","title":{"rendered":"Microsoft increases sales by using AI for lead qualification"},"content":{"rendered":"
This content has been archived, and while it was correct at time of publication, it may no longer be accurate or reflect the current situation at Microsoft.<\/p>\n<\/div>\n<\/div>\n
At Microsoft, our sales and marketing team receives millions of potential sales leads each year that they must follow-up with to determine how they can best meet each potential customer\u2019s needs. Microsoft Digital has helped sales and marketing embed AI and machine learning into the lead-qualification process in our Dynamics 365 sales solution, quadrupling our sales effectiveness and creating an environment of optimized operations and empowered employees that helps drive our digital transformation.<\/p>\n
At Microsoft, our sales and marketing team receives millions of potential sales leads each year that they must follow-up with to determine how they can best meet each potential customer\u2019s needs. Microsoft Digital has helped Microsoft sales and marketing infuse AI and machine learning into the lead-qualification process, creating a more intelligent and efficient method for identifying customers who might purchase our products. The new solution, using AI and machine learning\u00a0integrated with our Microsoft Seller Experience (MSX) which runs on Dynamics 365, allows our sales and marketing team to more efficiently address customers\u2019 needs. It has also quadrupled our sales effectiveness and created an environment of optimized operations and empowered employees that helps drive our digital transformation.<\/p>\n
AI and machine learning are quickly becoming important foundations for digital transformation in many organizations, including Microsoft. With recent data science advances and more easy-to-implement AI services, combined with the inundation of smart technology into almost every aspect of business, the terms AI and machine learning have entered the vocabulary of the business world and the general public.<\/p>\n
Machine learning and AI are being used increasingly and are becoming an integral part of companies\u2019 digital-transformation strategy. At Microsoft, we\u2019re using AI and machine learning across our entire organization to digitally transform the way we do business.<\/p>\n
We used these technologies for lead qualification to support two of the four outcomes of our digital transformation strategy at Microsoft:<\/p>\n
Our primary motivation was to create a more productive experience for our salespeople and a more focused sales and marketing follow-up process for our customers.<\/p>\n
To generate global demand for our products and services, our marketing and sales organization collects leads when people request information or access tracked content with an online form. Our system collects leads by using marketing vehicles such as:<\/p>\n
When someone signs up for a Microsoft product trial, sends us an email, or downloads content, the person or business becomes a\u00a0lead<\/i>\u00a0for the purposes of sales and marketing. We market to between 5 million and 10 million leads per year. These names\u2014which can be a company or a person\u2014become records in our marketing and sales system. Our salespeople take the leads they receive from our marketing and sales system and work to convert the unknown prospective customer into a sales-qualified opportunity (SQO): a customer who is ready to buy Microsoft products or services.<\/p>\n
As a sales organization, our primary goal is to be as effective as possible in taking unknown leads and converting them to Microsoft customers\u2014users and purchasers of our products. We begin the process by identifying who the unknown entity is, by obtaining the name, organization, and other basic information. We then take this prospective customer and try to ascertain how likely they are to purchase Microsoft software. We do this by assessing qualifying information in a continuing process that attempts to move them from a prospect, to a lead, to an identified sales opportunity, and finally, to a sales win and customer success.<\/p>\n Historically, the leads that our salespeople receive come with very little qualification. Eighty percent of our leads do not respond to an initial contact from our salespeople. A lead might have been generated by using false or incomplete information, or a customer might have no intention of purchasing products when they download a product trial. As a result, our salespeople had to review up to 2 million leads annually to identify commercial customers who are ready to purchase a product. Leads that didn\u2019t respond were abandoned, and salespeople pursued other leads on the list.<\/p>\n Our sales and marketing team knew that they needed a better way to qualify the massive amount of leads that our salespeople were processing. We wanted our salespeople to focus on selling. They were spending too much time trying to find prospective purchasers and not enough time focusing on customers who were ready to buy. We needed a more intelligent solution.<\/p>\n To improve the leads\u2019 quality for our salespeople, we needed be able to define and measure that success. We established several goals that we wanted to achieve in improving the lead-qualification process by using AI and machine learning:<\/p>\n Our primary measure of success for lead conversion is the rate at which our sales team converts a lead to a sales-qualified opportunity (SQO) in MSX: someone ready to purchase. Before we started using AI to qualify leads, our sales team was converting approximately 4 percent of the between 1 million and 2 million commercial leads annually into SQOs. We wanted to improve that rate significantly.<\/p>\n The journey into AI-driven lead qualification started before AI became part of our lead-assessment process, and it has occurred in three distinct phases:<\/p>\n Prior to AI and machine learning, we used basic business rules that we configured in our sales and marketing system to identify incoming leads. It provided a basic scoring system to help our salespeople choose leads that might yield sales opportunities. However, the scoring system\u2019s business rules couldn\u2019t account for the wide variety of lead quality and intent, so it still produced many leads that weren\u2019t interested in purchasing Microsoft products. Our conversion rate to SQO under this system was approximately 4 percent.<\/p>\n Our first step in improving lead quality was to implement AI-assisted lead scoring. We designed the scoring system to examine the different aspects of incoming leads and assign relative values to those aspects as they related to lead quality. AI lead scoring improved on the static, business-rules-based system by focusing on behavior and attributing a dynamic value to a lead based on that behavior. Our algorithm accounted for important information, such as:<\/p>\n The system generated a score for each lead based on algorithm output, and then stored that information in our sales and marketing system. Leads given a higher score were pushed to the top of the list while lower scoring leads were placed at the bottom. The results provided our salespeople with a ranked list from which they could take the highest-scoring leads and engage the customer.<\/p>\n We designed the scoring to go deeper into business processes to create context around a lead, thus enabling our salespeople to better respond to the lead. It helped us begin to move from high-volume, low-quality leads to high-quality, scored leads. Our sellers could now focus on the leads that were most likely to result in an SQO. However, we recognized even more potential in AI to improve the lead qualification further. The scoring process often missed leads that didn\u2019t fit into the typical lead-qualification categories, even though the customer was ready to buy. In the past, such leads remained at the bottom of the list, where a long time might elapse before a salesperson pursued them. It also left leads in the system that exhibited the behaviors of a potential customer but one who was not interested in purchasing. For example, someone studying for certification in one of our products might sign up for trial software and attend workshops but does not intend to make any purchases. While AI lead scoring improved our SQO efficiency from 4 percent to 6 percent, we recognized the potential to add more intelligence into our qualification process.<\/p>\n BEAM is the centerpiece of our current AI-based lead-qualification solution. We built BEAM as an orchestration engine to capture incoming lead contact information through email and automate the contact process, determine lead intent by using machine learning, and provide a clearer lead-ranking process.<\/p>\n The basic BEAM process consists of the following steps:<\/p>\n BEAM\u2019s intent-detection engine examines and identifies specific phrases as indicators of a customer\u2019s intent to purchase products and services. For example, BEAM might recognize the phrase \u201cwondering about pricing\u201d as an indicator that a lead is ready to purchase. On the other hand, a phrase such as \u201cNever contact me again\u201d is a pretty clear indicator that a customer is not interesting in purchasing. BEAM uses machine-learning models to perform two assessments of the underlying data:<\/p>\n The BEAM platform is built on Microsoft Azure technologies and open-source AI and machine learning tools, including Microsoft Cognitive Services, Azure Machine Learning Server, and Azure Machine Learning Studio\u00a0and then integrated into Dynamics 365. We built BEAM to be modular and flexible; we can interchange components and modify component behavior without negatively affecting the solution as a whole or refactoring.<\/p>\n <\/p>\n One critical component of our BEAM architecture model is data preparation and cleanup. The machine-learning models for BEAM accept only plain-text inputs. The plain text is parsed in its entirety for intent and context detection. All BEAM inputs come from an email source, which has its own inherent data-quality issues when converted to plain text. Several parts of the email message must be stripped from the plain text input because they provide no value in intent detection and might cause issues with model-training behavior. These include:<\/p>\n We use Bing Spell Check in Microsoft Cognitive Services to spell check incoming email text to ensure proper recognition by the machine-learning models. We also created code to remove private personal information from ingested text.<\/p>\n At BEAM\u2019s core are its machine-learning models. We use several models that each provide intent detection and work together to provide BEAM functionality.<\/p>\n Our first model, and the one that does the most intensive work, is the Language Understanding Intelligent Service (LUIS) machine-learning service from Microsoft Cognitive Services. LUIS is a natural-language processing model designed to identify valuable information in conversations, interpret intent, and distill information from sentences for a high-quality, nuanced language model. LUIS integrates seamlessly with several API interfaces, which makes it extremely easy to incorporate into BEAM.<\/p>\n We created a layer in BEAM that sends API calls to LUIS by using JavaScript Object Notification (JSON). The JSON code is the primary input for LUIS, and it contains the prepared, cleaned data that LUIS ingests. A second layer translates the LUIS output into a standard response, in JSON, that is consumed by the rest of the BEAM architecture.<\/p>\n We\u2019ve used LUIS to create applications in BEAM for both intent detection and context detection. LUIS\u2019 out-of-the-box usability made it easy to quickly integrate LUIS-based applications into BEAM. LUIS also provides a flexible base for building machine-learning applications. Even though context detection isn\u2019t one of the natively supported uses for LUIS, we were able to leverage the service to create an easy-to-integrate context-detection service for BEAM that could run alongside the intent detection engines (including LUIS) that we were using in BEAM. We\u2019re continually examining LUIS for more opportunities to build other functionality into BEAM.<\/p>\n We also implemented three separate models from scikit-learn, an open-source data mining and analysis tool set that performs machine learning in Python. We host each model in Azure Machine Learning Server, which enables us to deploy the model to whatever scale we require and keep our BEAM environment hosted entirely on the Azure platform. We\u2019re using the following scikit-learn models:<\/p>\n Each of these models provides an evaluation of the input data from a distinct perspective and provides an equally weighted output that contributes to the determined intent.<\/p>\n Each machine-learning model produces a discrete result, indicating the evaluated intent from a lead email. We use ensemble methods meta-algorithms to assess all four models\u2019 results and provide output to the BEAM engine from the model that generates the most accurate results. Ensemble methods allow us to freely insert and remove machine-learning models into the BEAM pipeline. We can test model effectiveness more quickly and adapt our machine learning practices according to our business needs.<\/p>\n We use the Ensemble Majority_MaxOrder method, which considers all model inputs and chooses the intent that received the majority vote from all models. If there is a contention (two intents have an equal number of votes), the one with the highest probability score is chosen. If several have the same probability score, the order of the models determines which one is chosen. We are continually evaluating the models\u2019 output and considering other potential ensemble methods as BEAM intent detection matures.<\/p>\n Our models are continually trained through an automated process that uses the previous month\u2019s data and tests model output against the current month\u2019s data. We also provide model training through our manual intent review portal, where our validation team can review BEAM output for validity and accelerate the training process.<\/p>\n This manual training process was extremely important in the early stages of BEAM implementation, and continued use helped our machine-learning models make more accurate predictions very quickly. We also have an automated review process that takes obvious intent phrases such as \u201cDo not contact me\u201d and assigns the appropriate classification.<\/p>\n Once we have an intelligent lead qualification score for every lead, it is passed on to our MSX system, running on Dynamics 365. Using Dynamics 365 as our sellers\u2019 main productivity tool and surfacing intelligent insights into that tool creates the best of both worlds for our sales force. Dynamics 365 and AI operate together to give sellers a view of their prioritized workload based on the highest potential customers in the opportunity pipeline.<\/p>\n Sellers now have the information they need to optimize their activities in their day, week and month. Sellers have a personalized view of prioritize opportunities. These are opportunities that are unique to them based on the product they focus on and their country and location. And they get to see the entire BEAM conversation with the customer so that can understand the full context of the customers\u2019 needs before they place a call to that customer. This is just a start. We will continue to infuse AI into MSX, augmenting our sellers\u2019 capabilities, so they can focus on the most important aspect of their job: taking care of their customers.<\/p>\n We completed BEAM implementation over a six-month period at Microsoft. We provided proof-of-concept (POC) data and results, and then ran a two-month pilot project with 180 customers in two regions. The pilot allowed us to production test our architecture and end-to-end processes. We then rolled out BEAM to the rest of our sales organization over the ensuing three months. In the time that we\u2019ve had to review its effectiveness, the combination of AI lead scoring and BEAM has more than quadrupled our SQO conversion rate, from 4 percent to 18 percent\u00a0in MSX.<\/p>\n BEAM and AI-based lead scoring have radically changed our salespeople\u2019s ability to identify and pursue productive sales leads and deliver Microsoft products and services to more customers far more efficiently. We\u2019ve experienced several specific benefits from BEAM and AI-based lead qualification, including:<\/p>\n We\u2019re excited about future possibilities for BEAM and other AI and machine-learning-based development at Microsoft. We plan to improve and expand in several areas:<\/p>\n Infusing AI and machine learning into Dynamic 365 for\u00a0the lead-qualification process has greatly improved the effectiveness of our sales team, creating a more intelligent and efficient method for identifying potential customers. AI lead scoring and BEAM\u00a0integrated into MSX have significantly improved the effectiveness of our sales pipeline by providing our sales team with higher quality leads that are ready to buy our products and grow their relationship with Microsoft. AI and machine learning have enabled our digital transformation, allowing us to empower our employees to achieve more and optimize our lead qualification process.<\/p>\n","protected":false},"excerpt":{"rendered":" This content has been archived, and while it was correct at time of publication, it may no longer be accurate or reflect the current situation at Microsoft. At Microsoft, our sales and marketing team receives millions of potential sales leads each year that they must follow-up with to determine how they can best meet each […]<\/p>\n","protected":false},"author":146,"featured_media":10447,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"_hide_featured_on_single":false,"_show_featured_caption_on_single":false,"footnotes":""},"categories":[1],"tags":[],"coauthors":[674],"class_list":["post-10445","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","m-blog-post"],"yoast_head":"\nInitiating a more intelligent lead-qualification process<\/h2>\n
\n
Considering AI for lead-qualification improvement<\/h3>\n
\n
Examining our pre-existing basic business rules<\/h3>\n
Improving basic lead qualification with AI lead scoring<\/h3>\n
\n
Facilitating intelligent intent detection using BEAM<\/h2>\n
\n
Understanding intent and context detection<\/h3>\n
\n
\n
Examining BEAM architecture and infrastructure<\/h3>\n
Ingesting and preparing data for machine learning input<\/h4>\n
\n
Machine-learning architecture and models<\/h4>\n
Language understanding with LUIS<\/h5>\n
Custom-coded models from scikit-learn<\/h5>\n
\n
Combining machine learning output with ensemble methods<\/h5>\n
Training machine-learning models and providing classification feedback<\/h4>\n
Surfacing AI within Dynamics 365<\/h3>\n
Implementing BEAM at Microsoft<\/h3>\n
Benefits<\/h2>\n
\n
Best Practices<\/h2>\n
\n
Looking forward<\/h2>\n
\n
Conclusion<\/h2>\n