Intelligent digital assistants help people perform simple tasks such\u00a0as question answering, event scheduling, etc. Complex task assistance requires monitoring goals, intentions, and task state. Additionally, complex tasks may exceed the capabilities of one single\u00a0device or application, giving rise to the need to support people as\u00a0they engage with two or more devices (e.g., smartphones, smart\u00a0wearable devices, tablets, and desktop or laptop computers) simultaneously. Using multiple connected devices to support tasks has\u00a0now become a viable design option for digital assistance since many\u00a0users already own multiple devices. In this paper, we introduce a\u00a0framework for developing multiple device experiences (MDX), then\u00a0demonstrate some of the early capabilities of the framework by\u00a0means of the AskChef cooking assistant. AskChef allows people\u00a0to use their existing mobile devices (e.g., tablets), together with\u00a0existing headless smart speakers (e.g., the Amazon Echo) for rich\u00a0step-by-step guidance and contextual question answering over the\u00a0culinary recipe loaded in their mobile device\u2019s web browser. We\u00a0conducted a pilot study in which people engaged in cooking recipes\u00a0with the AskChef MDX system. All participants reported finding\u00a0the support to be helpful, and indicated a willingness to use the\u00a0system in the future. Nevertheless, AskChef erred on occasion, affording an opportunity to study how people respond to errors in multi-device scenarios. We examine participants\u2019 perceptions and\u00a0actions, and outline some directions for future research.<\/p>\n","protected":false},"excerpt":{"rendered":"
Intelligent digital assistants help people perform simple tasks such\u00a0as question answering, event scheduling, etc. Complex task assistance requires monitoring goals, intentions, and task state. Additionally, complex tasks may exceed the capabilities of one single\u00a0device or application, giving rise to the need to support people as\u00a0they engage with two or more devices (e.g., smartphones, smart\u00a0wearable devices, […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"msr-content-type":[3],"msr-research-highlight":[],"research-area":[13556,13545,13554,13555],"msr-publication-type":[193716],"msr-product-type":[],"msr-focus-area":[],"msr-platform":[],"msr-download-source":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[246691,254263,248485,254260,253570,254266,248587,248338,254269,254272],"msr-conference":[],"msr-journal":[],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-566706","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-human-language-technologies","msr-research-area-human-computer-interaction","msr-research-area-search-information-retrieval","msr-locale-en_us","msr-field-of-study-computer-science","msr-field-of-study-event-scheduling","msr-field-of-study-human-computer-interaction","msr-field-of-study-laptop","msr-field-of-study-mobile-device","msr-field-of-study-multiple-device","msr-field-of-study-perception","msr-field-of-study-question-answering","msr-field-of-study-web-browser","msr-field-of-study-willingness-to-use"],"msr_publishername":"ACM","msr_edition":"","msr_affiliation":"","msr_published_date":"2019-1-31","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"","msr_volume":"","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":1,"msr_main_download":"","msr_publicationurl":"","msr_doi":"","msr_publication_uploader":[{"type":"url","viewUrl":"false","id":"false","title":"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/02\/NouriTI2019.pdf","label_id":"243132","label":0}],"msr_related_uploader":"","msr_attachments":[{"id":566709,"url":"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2019\/02\/NouriTI2019.pdf"}],"msr-author-ordering":[{"type":"user_nicename","value":"Elnaz Nouri","user_id":39336,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Elnaz Nouri"},{"type":"user_nicename","value":"Adam Fourney","user_id":30820,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Adam Fourney"},{"type":"user_nicename","value":"Robert Sim","user_id":36650,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Robert Sim"},{"type":"user_nicename","value":"Ryen W. White","user_id":33481,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Ryen W. White"}],"msr_impact_theme":[],"msr_research_lab":[],"msr_event":[558867],"msr_group":[],"msr_project":[670338],"publication":[],"video":[],"download":[],"msr_publication_type":"inproceedings","related_content":{"projects":[{"ID":670338,"post_title":"Enabling Multidevice Experiences for the Modern Workplace","post_name":"enabling-multidevice-experiences-for-the-modern-workplace","post_type":"msr-project","post_date":"2020-06-29 11:51:49","post_modified":"2020-06-29 11:51:49","post_status":"publish","permalink":"https:\/\/www.microsoft.com\/en-us\/research\/project\/enabling-multidevice-experiences-for-the-modern-workplace\/","post_excerpt":"As the future of work is rapidly evolving there is a growing opportunity towards incorporating new practices and new workforces to enable flexible productivity practices. Expanding what is possible on the traditional work setting, this research initiative looks at how we can conduct work using multiple devices, and in multiple environments, and how we can offer best practices for getting things done for people with varying mobility demands. We study how multiple devices can work…","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/670338"}]}}]},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/566706"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":1,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/566706\/revisions"}],"predecessor-version":[{"id":566712,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/566706\/revisions\/566712"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=566706"}],"wp:term":[{"taxonomy":"msr-content-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-content-type?post=566706"},{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=566706"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=566706"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=566706"},{"taxonomy":"msr-product-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-product-type?post=566706"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=566706"},{"taxonomy":"msr-platform","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-platform?post=566706"},{"taxonomy":"msr-download-source","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-download-source?post=566706"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=566706"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=566706"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=566706"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=566706"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=566706"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=566706"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=566706"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}