Getting to ‘search completeness’ internally at Microsoft

Microsoft is a big company with thousands of teams working in different ways based on the work they do. Despite that complexity, when our employees go looking for something, they expect an internal search portal that will find exactly what they’re looking for instantly—just like when they search on the internet. Yet when talking to these employees, each of them defines the scope of what they’re looking for quite differently.

• A developer may want HR info, stack overflow, other technical info specific to their organization, or technical info from places like Microsoft Azure and Microsoft.com.
• A salesperson may want HR info, customer information from our account management software and support services, or the latest public information about their customers.

This blog explores the challenge of delivering the full scope of content each employee expects to find in search from their subjective view. This is what we call search completeness.

To start on the journey of getting to search completeness, you first must understand your user community:

• How do they search? Do they use smart phones? Do they use Bing’s Work search tool? Do they use a corporate SharePoint portal? For Microsoft employees, it’s a mix of all of these.
• Why are they searching? Are they trying to find another person? Are they researching content? Are they trying to find reference material?
• What are they searching for? What content is most important for your employees to find?

[Read the first blog in our series, making content more accessible and searches more efficient at Microsoft.]

Understanding your user community

Reviewing search term frequency was one of our early steps in understanding our users. Looking at the number of times each search term was used, then looking at a sampling of those search terms made it very clear that the most common searches are for common employee actions, and that less common searches are typically persona specific. The chart below shows this well: high volume search terms that are common across most employees, and low-volume ones that tend to be org- or persona-specific.

Sometimes we could easily identify desired content from these popular search terms such as search terms related to documents. Microsoft.com and Stack overflow were also fairly popular.

Next, we realized there was a lot of content that was impossible to identify from search terms. We needed some other way of identifying desired content and found a way via Microsoft Azure Active Directory (AAD).

By using its authentication volume, we are able to see the most popular registered apps within the company. Many of these are included in Microsoft Search by default. SharePoint and OneDrive are good examples. Others have their own search capability that meets user expectations and doesn’t need its content included in enterprise search. Outlook is such an example. This left us with a significant volume of highly used apps whose content would be beneficial to add to enterprise search. The chart below gives you a taste of these results.

Gathering the list of popular apps left us with a challenge of identifying popular content that isn’t defined as an app in AAD. We explored various ways of capturing this information but, so far, have not found any better method than user feedback and surveys.

The result of this work has yielded a “Top 100” list of content we want to add to enterprise search. So how do we go about getting this content added into our search results?

Methods of achieving search completeness

Microsoft Search provides a number of different methods with which to bring in all the content. Each method has its own strengths and weaknesses, which we’ve summarized in the table below.

What we are doing

So now the stage is set, we know the content we want to include, we know the methods available for doing it, we just need to implement the right method in each case.

We also use Microsoft Viva Topics and other product capabilities, which will be discussed in a future blog post.

At this point, search indexing encompasses 70 percent of the AAD Top Apps list as weighted by usage volume. We expect to reach 80 percent within the next year.

• The content added through connectors and federated search is receiving 75,000 clicks per month––about 8 percent of our total click volume.
• These connections have added 10 percent to the admin effort. For more detail, see the previous blog post in this series: Generating great results: Administering search at Microsoft.

We’ve also realized there are occasions where content should not be included in enterprise search but should be included in targeted custom search portals. The same methods described above can typically be used to support such custom portals. Our learning thus far will also be described in a future post.

We see some continuing challenges for which we do not yet have answers:

1. At some point the administrative and resource overhead associated with adding additional sources of content will outweigh the benefit because we will be getting down to very seldom used content. We don’t know where that boundary is yet.
2. We need to figure out how to stay in touch with continuing changes across the company, deprecating content when appropriate while adding new content sources when they come up.
3. We haven’t figured out how to tune search relevance in a manner that works well for each persona.

Please return to this space for future stories in our ongoing series on transforming search completeness here at Microsoft.

• Read the first blog in our series, making content more accessible and searches more efficient at Microsoft.
• Unpack how internal search bookmarks boost productivity at Microsoft.
• Explore generating great results: Administering search at Microsoft.