Editor’s note: We’ve republished this blog with a new companion video.
Search is part of our everyday life. It’s useful—we all know that—but how can you quantify that impact?
That was the challenge faced by Dodd Willingham, principal program manager and internal search administrator in Microsoft Digital. “There’s an obvious value, we can see that by the existence of Bing,” Willingham says. “But how do you put it in numbers?”
Lots of searches happen in a company, but when asked to demonstrate the business impact as part of justifying more investment, Willingham had an epiphany. He could use telemetry to make the argument for him.
Click the image to learn how Microsoft is using Microsoft Search internally to dramatically improve the finding experience for company employees.
Microsoft Search is unifying search for Microsoft 365 customers across Microsoft Outlook, Microsoft 365 apps on Windows, Microsoft OneDrive for Business, Microsoft SharePoint, and Microsoft Bing. More specifically, the Microsoft Search team strives to bring complete, company-wide results to each individual, no matter where they’re searching from. No longer should they need to search in separate products to ensure that they search all possible content.
Internally at Microsoft, this shift is proving to be very powerful.
“Employees no longer need to change platforms to get the results they’re looking for,” Willingham says. “They do a single search and get all the results they need.”
Within the company, Microsoft Digital manages the internal deployment of search across the company. “The purpose of active search administration is to deliver the most complete search results, with good relevancy and good quality,” Willingham says. “These improvements to search are helping us do that.”
One crucial way that Willingham and his team help deliver better search results is through corporate bookmarks that allow internal teams like Corporate Communications and Human Resources to select the top results employees get when they search specific sets of keywords.
These bookmarks aren’t the kind used to save your favorite sites—they’re curated results that search administrators can use to point people to content located someplace that can’t be indexed. They highlight authoritative sources of content, and ensure popular content is accessible.
Bookmarks boost employee productivity because they get employees the right results very quickly.
Dodd Willingham, principal program manager and internal search administrator in Microsoft Digital
And they’re fast.
“Bookmarks boost employee productivity because they get employees the right results very quickly,” Willingham says.
The business value of search
Including telemetry in the overall improvements to internal corporate searching—a feature built into Microsoft Enterprise SharePoint—allowed Willingham and his team to measure how much time employees spend on a search.
And what story is the data telling?
“We found that bookmarks net a direct benefit of 6,250 hours a month and 17,160 hours in indirect benefits,” Willingham says. “Combined, 23,410 hours of benefits are being realized each month.”
How did Willingham come to these numbers?
“Forty-five percent of all searches click on a bookmark,” Willingham says. That percentage is across the 1.6 million monthly searches that take place internally at Microsoft within Microsoft Bing and Microsoft SharePoint Enterprise Search.
Scaled to an enterprise level, the business value of bookmarks quickly became apparent.
“Conservatively, our basic measurement of search success was yielding results of 60 seconds per search using a bookmark versus an average of 115 seconds across all searches,” Willingham says. “That’s one whole minute of productivity re-captured for every bookmark-backed search.”
Multiplied across Microsoft’s population and search usage, that one minute of search time netted 6,250 hours a month in productivity. But it’s not just time gained from quick search results, it’s also about getting the right answers.
There’s a measurement based on telemetry of whether a search succeeded or failed to find useful content. Using that metric, Willingham found that a person who uses a bookmark appears to be successful 98 percent of the time. By contrast, searches without a bookmark average 72 percent for the same calculation.
“The absolute calculation [of search success] is kind of meaningless; what’s important is that it moved by a significant margin,” Willingham says. “It suggests that with bookmarks, more people find the content they need faster.”
In direct benefits, you’re gaining 6,000 hours at the cost of 300. When you include indirect, you can triple that. The return on investment is 2,000 percent, and that’s using conservative estimates.
Dodd Willingham, principal program manager and internal search administrator in Microsoft Digital
Faster is a direct productivity gain. Getting the right content to the right person at the right time is an indirect benefit. But the biggest insight is that delivering these benefits only requires investing less than 300 hours per month, spread across several staff.
“In direct benefits, you’re gaining 6,000 hours at the cost of 300. When you include indirect, you can triple that,” Willingham says. “The return on investment is 2,000 percent, and that’s using conservative estimates.”
How Microsoft uses bookmarks
With new practices in hand and telemetry to chart impact, Willingham and his team set out to optimize using bookmarks in search.
“Over the course of three years, we took the volume of bookmarks from around 1,100 to a peak of 1,800,” he says. “We’re currently sitting at around 1,200.”
Bookmarks were already being used before Microsoft Search was rolled out.
“We didn’t do anything revolutionary, we just opened up the guidelines so that more bookmarks could be added when appropriate,” Willingham says. “We then tuned them based on actual usage so that only those being used were kept.”
The technology for bookmarks had previously been part of Microsoft SharePoint and Microsoft OneDrive, made visible in the employee portal for Microsoft SharePoint Enterprise, MSW. Bookmarks had a set of configuration rules and standards for what could and couldn’t be a bookmark, but that’s it.
Librarians from the Microsoft Library Services team create and manage the company’s search bookmarks.
“It’s a multifaceted role,” says Beck Keller, also a member of the Microsoft Digital Enterprise Search team. “My responsibilities as a librarian at the Microsoft Library are far broader—bookmarks are just a small part of my job. This doesn’t take up my entire work week.”
What does she do for search administration?
Every month, Keller pulls search query metrics and analyzes them for areas of interest that currently lack a bookmark or good naturalized results. From this analysis, Keller can update the enterprise bookmarks across Microsoft.
“Sometimes this means removing or changing bookmarks that don’t currently meet our standards,” Keller says. “I also review proposed bookmarks and offer guidance to Microsoft teams looking to create bookmarks for their own sites, outside of Enterprise Search.”
This is the administrative work Willingham is talking about—bookmarks can be added, removed, or updated with ease. But the impact can be bigger than recapturing lost productivity.
“A year ago, there were no searches for COVID-19,” Willingham says. “We now get hundreds and thousands of searches a month. We went from zero to around 200 [between October and February]. There was no way to surface relevant results about COVID-19 because there were so few of them.”
But this was the trait the administrative search team was looking for—how to get better and proactive insights on Microsoft Search. Informed by current events, the team sought to anticipate which results users would be looking for.
“We asked if there should be a bookmark for the right COVID-19 link,” Keller says.
Willingham and Keller reached out to Corporate Communications about where to direct Microsoft users searching for information on COVID-19. That team was putting together a landing page for employees dedicated to content on the topic, including a FAQ. The bookmark was quickly built and deployed.
This was February 2020.
“The next month, the volume of searches for COVID-19 went up 40-fold,” Willingham says. “Maybe users would have found the info on their own, but as search volume was growing, 8,000 times a month they would nearly always find what they were looking for quickly, thanks to the bookmark.”
That’s the main goal of a search administrator.
Bright future for bookmarks
So, what’s next for Microsoft Search and bookmarks?
“More telemetry,” Willingham says. “The custom telemetry that we created is something any customer can do. It’s a capability within SharePoint.”
Having even more metrics will also help to further quantify Willingham’s findings.
“We erred on the low side for our productivity numbers, but it shows what’s possible for a medium or large company.”
Both Willingham and Keller are excited to see others adopt bookmarks as a way of improving Microsoft Search.
“Bookmarks are easy to put in,” Keller says. “The owner of the content tells us what the URL is, and some basic info such as a preliminary title and description. We figure out the appropriate keywords, update the basic info where needed, and then say ‘Go.’”
It all adds up to a better experience for employees when they need to go looking for something.
“The same tools we use to optimize bookmarks are available to everyone,” Willingham says. “That’s why they’re so useful for productivity. When combined with telemetry, you can really gain some unexpected insights into the productivity of your organization.”
