{"id":1134901,"date":"2012-03-21T13:56:00","date_gmt":"2012-03-21T20:56:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-video&#038;p=1134901"},"modified":"2025-03-31T10:20:53","modified_gmt":"2025-03-31T17:20:53","slug":"the-assistant-situated-interaction-project-2012","status":"publish","type":"msr-video","link":"https:\/\/www.microsoft.com\/en-us\/research\/video\/the-assistant-situated-interaction-project-2012\/","title":{"rendered":"The Assistant: Situated Interaction Project (2012)"},"content":{"rendered":"\n<p>The Assistant was a long-running AI system developed as part of the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/elevating-human-computer-interaction-to-a-new-level-of-sophistication\/\">Situated Interaction project<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> at Microsoft Research. Designed to function as a working administrative assistant, it was stationed outside the office of Eric Horvitz\u2014then Lab Director at Microsoft Research Redmond. This video showcases the Assistant in action, highlighting its capabilities across a variety of scenarios. You can also see the system operate <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.youtube.com\/watch?v=dpoVh9xwdD4&t=1017s\" target=\"_blank\" rel=\"noreferrer noopener\">\u201cin the wild\u201d in this TED talk.<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/p>\n\n\n\n<p>The Assistant served as an exploratory AI research testbed, blending multiple strands of AI into a unified, real-world application. Built to operate in the dynamic environment of a research lab, the Assistant helped coordinate meetings with Eric and briefed him on missed events upon his return. It was capable of engaging in multiparty dialogue, drawing on natural language processing, machine vision, speech recognition, and acoustical sensing. The Assistant project was co-led by Dan Bohus and Eric Horvitz, with significant contributions from Anne Loomis Thompson, Paul Koch, Tomislav Pejsa, Michael Cohen, James Mahoney.<\/p>\n\n\n\n<p>The Assistant was a descendant of the earlier <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/video\/open-world-dialog-challenges-directions-prototype-sidemo\/\">Receptionist project<\/a>, a research effort on multiparty dialog capabilities. The project took an \u201cintegrative AI\u201d approach\u2014bringing together a constellation of technologies to create a cohesive, intelligent agent with the intuitions of a long-term administrative assistant. The Assistant leveraged several specialized systems that had previously been developed as standalone research efforts, including:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/coordinate-probabilistic-forecasting-presence-availability\/\">Coordinate<\/a> \u2013 Uses machine learning to predict someone\u2019s presence and availability, including forecasts of return times and when they would next check email.&nbsp;System considered predictions of meetings someone was likely to skip, allowing others to \u201cpencil in\u201d meetings accordingly.<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/busybody-creating-fielding-personalized-models-cost-interruption\/\">BusyBody<\/a> \u2013 Assesses the cost of interrupting someone based on contextual information such as desktop activity, conversation, and location. Busybody was part of longer-term studies on the use of machine learning about the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/learning-and-reasoning-about-interruption\/\">cost of interruption<\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/disruption-recovery-computing-tasks-field-study-analysis-directions\/\">recovery from disruptions<\/a>.&nbsp;<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/jogger-models-for-context-sensitive-reminding\/\">Jogger<\/a> \u2013 Use of machine learning to predict likelihood of forgetting information that would be valuable in a setting.<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/attention-sensitive-alerting\/\">Priorities<\/a> \u2013 Ranks unread emails by estimating the cost of delayed review.<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/models-multiparty-engagement-open-world-dialog\/\">Models for multiparty engagement <\/a>\u2013 Enables systems to recognize and support dialog with multiple people in a joint conversation.<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/natural-communication-uncertainties-situated-interaction\/\">Multichannel grounding<\/a> \u2013 Considers uncertainty at multiple levels, including vision, speech recognition, natural language understanding, and core assistant domain and provided linguistic and gestural cues about uncertainty aimed at resolution.<\/li>\n<\/ul>\n\n\n\n<p>The Assistant operated for several years, acting as an auxiliary aide until Eric transitioned to a new role as Director of Microsoft Research and moved to a different office.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Assistant was a long-running AI system developed as part of the Situated Interaction project (opens in new tab) at Microsoft Research. Designed to function as a working administrative assistant, it was stationed outside the office of Eric Horvitz\u2014then Lab Director at Microsoft Research Redmond. This video showcases the Assistant in action, highlighting its capabilities [&hellip;]<\/p>\n","protected":false},"featured_media":1134956,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13556,13545,13554],"msr-video-type":[],"msr-locale":[268875],"msr-post-option":[269148,269142],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-1134901","msr-video","type-msr-video","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-research-area-human-language-technologies","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-include-in-river"],"msr_download_urls":"","msr_external_url":"https:\/\/youtu.be\/O2uNo4nS1hc","msr_secondary_video_url":"","msr_video_file":"http:\/\/0","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video\/1134901","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-video"}],"version-history":[{"count":8,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video\/1134901\/revisions"}],"predecessor-version":[{"id":1134957,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video\/1134901\/revisions\/1134957"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1134956"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1134901"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1134901"},{"taxonomy":"msr-video-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video-type?post=1134901"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1134901"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1134901"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1134901"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=1134901"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}