{"id":169655,"date":"2008-10-13T12:18:08","date_gmt":"2008-10-13T12:18:08","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/project\/mavis\/"},"modified":"2016-07-28T09:36:32","modified_gmt":"2016-07-28T16:36:32","slug":"mavis","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/mavis\/","title":{"rendered":"MAVIS"},"content":{"rendered":"<p>The Microsoft Audio Video Indexing\u00a0Service (MAVIS)\u00a0uses state of the art speech recognition technology developed at Microsoft Research to enable searching of audio and video files with speech.\u00a0Additionally, MAVIS automatically generates closed captions and keywords\u00a0which can increase\u00a0accessibility and discoverability of audio\u00a0and video files with speech content. MAVIS is available as a cloud service running on the Windows Azure platform.<\/p>\n<p>MAVIS is now available programmatically through\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/azure.microsoft.com\/en-us\/services\/media-services\/\" target=\"_self\">Azure Media Services<\/a>\u00a0and referred to as the &#8220;Azure Media Services Indexer&#8221; (Indexer). The <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/azure.microsoft.com\/blog\/2014\/09\/10\/introducing-azure-media-indexer\/\">Introducing: Azure Media Indexer<\/a> blog post describes how to\u00a0submit media files to be processed and get results. The <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/mavisportal.azurewebsites.net\/\">MAVIS Portal<\/a> can be used to try out the service.<\/p>\n<p>At this time\u00a0the Indexer supports English speech content.<\/p>\n<p>For more information on MAVIS please <a href=\"mailto:mmms%40microsoft.com?subject=Info on MAVIS\" target=\"_self\">contact us<\/a>.<\/p>\n\t\t\t<div class=\"moray-accordion\" data-show=\"false\">\n\t\t\t\t\t\t\t<div class=\"clearfix\">\n\t\t\t\t\t<div\n\t\t\t\t\t\tclass=\"btn-group align-items-center mb-g float-sm-right\"\n\t\t\t\t\t\trole=\"group\"\n\t\t\t\t\t\tdata-mount=\"collapse-controls\"\n\t\t\t\t\t\tdata-bi-aN=\"accordion-collapse-controls\"\n\t\t\t\t\t\tdata-target=\"#accordion-1\">\n\t\t\t\t\t\t<button class=\"btn btn-link m-0\" aria-disabled=\"false\" aria-pressed=\"false\" data-action=\"expand\" type=\"button\" data-bi-cN=\"Expand all\">Expand all<\/button>\n\t\t\t\t\t\t<span aria-hidden=\"true\"> | <\/span>\n\t\t\t\t\t\t<button class=\"btn btn-link m-0\" aria-disabled=\"false\" aria-pressed=\"false\" data-action=\"collapse\" type=\"button\" data-bi-cN=\"Collapse all\">Collapse all<\/button>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<ul class=\"accordion\" id=\"accordion-1\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-2\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-2\">\n\t\tMAVIS Features\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-2\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<p><strong>Search audio for spoken words<\/strong> &#8211; MAVIS generates a binary file which can be searched in Microsoft SQL server using full text search.\u00a0The user experience is much like searching for text in documents and on the web as demonstrated on the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/www.msravs.com\/audiosearch_demo\/default.aspx\">MAVIS trial<\/a> site. Users type in search terms, the result\u00a0is a set of links, which when clicked on, will start playing the audio\u00a0or video\u00a0from where those terms were spoken.<\/p>\n<p><strong>Highly accurate audio search Results<\/strong> &#8211; MAVIS uses state of the art <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/blogs.technet.com\/b\/inside_microsoft_research\/archive\/2012\/06\/14\/deep-neural-network-speech-recognition-debuts.aspx\">Deep Neural Net (DNN) based speech recognition<\/a> technology developed at Microsoft Research to convert digital audio signals into words. Furthermore,\u00a0MAVIS\u00a0reduces errors in speech recognition by automatically expanding its vocabulary, and storing word alternatives using a technique referred to as Probabilistic Word-Lattice Indexing, explained in the technical background.\u00a0These techniques help provide highly accurate search results.<\/p>\n<p><b>Closed Captions\u00a0<\/b>&#8211; <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/en.wikipedia.org\/wiki\/Closed_caption\">Closed captions<\/a> can make audio and video content accessible to the hearing impaired, or translated so that the content can be used by a broader audience in different languages. MAVIS generates closed captions in the\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/en.wikipedia.org\/wiki\/SAMI\">SAMI<\/a>\u00a0and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/en.wikipedia.org\/wiki\/Timed_Text\" target=\"_self\">TTML<\/a> formats. The accuracy of closed captions generated by MAVIS will depend mainly on the clarity of speech in the media content. There are a number of subtitle editing tools available on the web which can be used to edit the closed captions generated by\u00a0MAVIS for improved accuracy. MAVIS provides an estimated level of accuracy to help determine if post editing is required.<\/p>\n<p><strong>Keyword generation<\/strong> &#8211;\u00a0MAVIS\u00a0generates keywords\u00a0from the speech content. The keywords are stored in an XML file with frequency and offset information. The keywords generated by MAVIS can be used to perform speech analytics, or exposed to\u00a0search engines such as Bing, Google\u00a0or Microsoft SharePoint to make the media files more discoverable,\u00a0or used to deliver\u00a0more relevant ads.<\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-3\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-3\">\n\t\tAbout MAVIS\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-3\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<p>As the role of multimedia continues to grow in the enterprise, Government, and the Internet, the need for technologies that better enable discovery and search of such content becomes all the more important.<\/p>\n<p>Microsoft Research has been working in the area of speech recognition for over two decades, and speech-recognition technology is integrated in a number of Microsoft products, such as the\u00a0Microsoft Windows OS, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/www.tellme.com\/\">TellMe.com<\/a>, <a href=\"http:\/\/www.microsoft.com\/exchange\/2010\/en\/us\/default.aspx\">Exchange 2010<\/a>, and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/office.microsoft.com\/en-us\/onenote\/FX100487701033.aspx\">Office OneNote<\/a>. Using integrated speech-recognition technology in the Windows OS, users can dictate into applications like Microsoft Word, or use speech to interact with their Windows system. The TellMe.com service allows mobile users to get directory services using speech while on the go. Microsoft Exchange 2010 and above, provides an automatic <a href=\"http:\/\/www.microsoft.com\/exchange\/2010\/en\/us\/unified-messaging.aspx\">transcript of incoming voicemails<\/a> , and in Office OneNote, users can search their speech recordings using keywords.<\/p>\n<p>MAVIS Adds to the list of Microsoft applications and services that use speech recognition. MAVIS is designed to enable searching of 100s or even 10,000s of hours of conversational speech with different speakers, on different topics. As illustrated below, the user can type in a search term or phrase and get back links to where those words were spoken. <span id=\"788b6dda-dba0-46d0-a8da-5b86af513e37\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image788b6dda-dba0-46d0-a8da-5b86af513e37\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mavis-mavis_ui_2.jpg\" alt=\"\" \/><span id=\"ImageCaption788b6dda-dba0-46d0-a8da-5b86af513e37\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n<p>Searching media files requires the installation of the\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/aka.ms\/indexersql\">MAVIS SQL add-on<\/a> on\u00a0a machine running Microsoft SQL server 2008 or later. The MAVIS SQL Add-on includes\u00a0the software components that\u00a0perform full text index on binary files generated by MAVIS, an API for searching media files and a sample web application to help develop a media search\u00a0site as illustrated by the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/www.msravs.com\/audiosearch_demo\/default.aspx\">MAVIS trial<\/a> site.<\/p>\n<p>MAVIS also generates closed captions in SMI and TTML format which can be used to make audio and video files accessible to people with hearing disability. Additionally, MAVIS generates keywords from the speech content which is stored in an XML formatted file with offset and frequency information.<\/p>\n<h2>MAVIS Architecture<\/h2>\n<p><span id=\"75ad10c6-3742-4b0d-8d26-a6da6512e72e\" class=\"ImageBlock fn\"><img decoding=\"async\" id=\"Image75ad10c6-3742-4b0d-8d26-a6da6512e72e\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/mavis-mavis_architecture.jpg\" alt=\"\" \/><span id=\"ImageCaption75ad10c6-3742-4b0d-8d26-a6da6512e72e\" class=\"ImageCaptionCoreCss ImageCaption\"><\/span><\/span><\/p>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-4\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-4\">\n\t\tNews about MAVIS\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-4\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<ul>\n<li>Sept 10, 2014 <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/blogs.technet.com\/b\/inside_microsoft_research\/archive\/2014\/09\/10\/indexing-audio-video-content-with-a-bit-of-research-assistance.aspx\" target=\"_self\">MAVIS announced publically as Microsoft Azure Media Services Indexer<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/blogs.technet.com\/b\/inside_microsoft_research\/archive\/2012\/06\/14\/deep-neural-network-speech-recognition-debuts.aspx\" target=\"_self\">Deep-Neural-Network Speech Recognition Debuts<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/science.energy.gov\/news\/in-focus\/2011\/05-25-11-3\/\">CERN\u00a0multimedia searchable on DOE ScienceCinema using MAVIS<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/www.osti.gov\/news\/pressreleases\/2011\/february\/science_cinema.shtml\">ScienceCinema uses MAVIS to enable searching of Videos from US Department of Energy<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/blogs.technet.com\/b\/next\/archive\/2011\/02\/08\/microsoft-research_1920_s-mavis-video-search-hits-the-doe.aspx\">Microsoft Research&#8217;s video search hits the DOE<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/www.techflash.com\/seattle\/2008\/10\/State_program_uses_Microsoft_technology_to_search_audio_archives33609614.html\" target=\"_self\">TechFlash<\/a>, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/blog.seattlepi.com\/microsoft\/archives\/152951.asp\" target=\"_self\">SeattlePI<\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/blog.seattletimes.nwsource.com\/techtracks\/2008\/10\/30\/more_microsoft_research_worldwidetelescope_updated.html\" target=\"_self\">Seattle Times <\/a>blogs\u00a0on MAVIS and the State of Washington Digital Archives<\/li>\n<\/ul>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-5\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-5\">\n\t\tMore Information\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-5\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<ul>\n<li>Technical background<\/li>\n<li><a href=\"http:\/\/www.microsoft.com\/casestudies\/Windows-Azure\/The-Georgia-Archives\/State-of-Georgia-Makes-Audio-and-Video-Recordings-Accessible-and-Searchable-Online\/710000000165\">State of Georgia case study<\/a><\/li>\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/live.visitmix.com\/MIX10\/Sessions\/FTL03\">Unlocking Audio\/Video Content with Speech Recognition<\/a>\u00a0(Mix 2010 talk)<\/li>\n<\/ul>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t<li class=\"m-0\">\n\t\t\t<h4 class=\"accordion-header\">\n\t\t\t\t<button\n\t\t\t\t\tclass=\"btn btn-collapse m-0\"\n\t\t\t\t\ttype=\"button\"\n\t\t\t\t\tdata-mount=\"collapse\"\n\t\t\t\t\tdata-target=\"#collapse-6\"\n\t\t\t\t\taria-expanded=\"false\"\n\t\t\t\t\taria-controls=\"collapse-6\">\n\t\tContact Us\t\t\t\t<\/button>\n\t\t\t<\/h4>\n\t\t\t<div class=\"collapse\" id=\"collapse-6\">\n\t\t\t\t<div class=\"accordion-body\">\n\t\t\t\t\t\t\t\t\t\t<ul>\n<li>Questions or comments? Send us\u00a0email at <a href=\"mailto:mmms@microsoft.com\">mmms@microsoft.com<\/a><\/li>\n<\/ul>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<\/ul>\n\t\t<\/div>\n\t\t\n","protected":false},"excerpt":{"rendered":"<p>The Microsoft Audio Video Indexing\u00a0Service (MAVIS)\u00a0uses state of the art speech recognition technology developed at Microsoft Research to enable searching of audio and video files with speech.\u00a0Additionally, MAVIS automatically generates closed captions and keywords\u00a0which can increase\u00a0accessibility and discoverability of audio\u00a0and video files with speech content. MAVIS is available as a cloud service running on the [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"research-area":[13551,13554],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-169655","msr-project","type-msr-project","status-publish","hentry","msr-research-area-graphics-and-multimedia","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"2008-10-13","related-publications":[],"related-downloads":[],"related-videos":[],"related-groups":[144778],"related-events":[],"related-opportunities":[],"related-posts":[],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[],"msr_research_lab":[199560,199565],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/169655"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":1,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/169655\/revisions"}],"predecessor-version":[{"id":214546,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/169655\/revisions\/214546"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=169655"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=169655"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=169655"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=169655"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=169655"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}