{"id":937770,"date":"2023-05-01T11:36:24","date_gmt":"2023-05-01T18:36:24","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/"},"modified":"2023-05-01T15:35:16","modified_gmt":"2023-05-01T22:35:16","slug":"escapement-a-tool-for-interactive-prototyping-with-video-via-sensor-mediated-abstraction-of-time","status":"publish","type":"msr-research-item","link":"https:\/\/www.microsoft.com\/en-us\/research\/publication\/escapement-a-tool-for-interactive-prototyping-with-video-via-sensor-mediated-abstraction-of-time\/","title":{"rendered":"Escapement: A Tool for Interactive Prototyping with Video via Sensor-Mediated Abstraction of Time"},"content":{"rendered":"<p>We present Escapement, a video prototyping tool that introduces a powerful new concept for prototyping screen-based interfaces by flexibly mapping sensor values to dynamic playback control of videos. This recasts the time dimension of video mock-ups as sensor-mediated interaction.<\/p>\n<p>This abstraction of time as interaction, which we dub <em>video-escapement prototyping<\/em>, empowers designers to rapidly explore and viscerally experience direct touch or sensor-mediated interactions across one or more device displays. Our system affords cross-device and bidirectional remote (tele-present) experiences via cloud-based state sharing across multiple devices. This makes Escapement especially potent for exploring multi-device, dual-screen, or remote-work interactions for screen-based applications.<\/p>\n<p>We introduce the core concept of sensor-mediated abstraction of time for quickly generating video-based interactive prototypes of screen-based applications, share the results of observations of long-term usage of video-escapement techniques with experienced interaction designers, and articulate design choices for supporting a reflective, iterative, and open-ended creative design process.<\/p>\n<div id=\"attachment_937800\" style=\"width: 778px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-937800\" class=\"wp-image-937800\" style=\"margin-bottom: -1ex\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Escapement-overview-Fig.jpg\" alt=\"Figure 1: On the left, a person tilts a hand-held tablet forward - this gesture is labeled \u201cInteractive prototyping with video.\u201d On the right, images show sequential steps through a video sequence. Each frame lies along a timeline. A single frame in the timeline is labeled as the \u201ctimeframe index\u201d. The timeline is normalized from 0 to 1. Boxes on the right show options for the sensor input (tilt, rotation, touch, location, angle, proximity), which lead to the mapping of this sensor input to timeframe index. Icons show other manipulations for sensor data: filtering, or transfer functions.\" width=\"768\" height=\"188\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Escapement-overview-Fig.jpg 1507w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Escapement-overview-Fig-300x73.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Escapement-overview-Fig-1024x250.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Escapement-overview-Fig-768x188.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Escapement-overview-Fig-240x59.jpg 240w\" sizes=\"(max-width: 768px) 100vw, 768px\" \/><p id=\"caption-attachment-937800\" class=\"wp-caption-text\"><em>Escapement is a prototyping tool that reifies video snippets as sensor-mediated interactive prototypes, for screen-based applications, and across one or more devices. The tool flexibly maps a variety of real-time sensor inputs (such as tilt, motion, or touch) to the time index of a pre-recorded video (or series of still images). This empowers designers to work directly with \u201ctime as a design material\u201d in the prototyping process, exploring the feel of an interaction in response to sensor data and corresponding visual feedback.<\/em><\/p><\/div>\n<div id=\"attachment_937890\" style=\"width: 522px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-937890\" class=\"wp-image-937890\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Fig-Escapement-Prototyping.png\" alt=\"A table showing parameters (ways to map video time to sensor data) and strategies (prototyping strategies). The parameters are: manual, touch, tilt, angle motion, gyroscope, and generic sensor. The strategies are: Using time as a design material, using video to shape the design, experiencing embodied explorations, exploring many sensors and degrees of freedom, tweening animations, and decoupling input and output.\" width=\"512\" height=\"361\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Fig-Escapement-Prototyping.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Fig-Escapement-Prototyping-300x211.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Fig-Escapement-Prototyping-1024x722.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Fig-Escapement-Prototyping-768x541.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Fig-Escapement-Prototyping-1536x1082.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Fig-Escapement-Prototyping-240x169.png 240w\" sizes=\"(max-width: 512px) 100vw, 512px\" \/><p id=\"caption-attachment-937890\" class=\"wp-caption-text\"><em>Parameters & Strategies with Escapement: The tool provides designers access to various parameters of control, which allows for the manipulation of video with time abstracted out. Throughout the years the tool has been in use, we identified prototyping strategies in use by designers as they create video-escapement prototypes.<\/em><\/p><\/div>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We present Escapement, a video prototyping tool that introduces a powerful new concept for prototyping screen-based interfaces by flexibly mapping sensor values to dynamic playback control of videos. This recasts the time dimension of video mock-ups as sensor-mediated interaction. This abstraction of time as interaction, which we dub video-escapement prototyping, empowers designers to rapidly explore [&hellip;]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"msr-content-type":[3],"msr-research-highlight":[],"research-area":[13554],"msr-publication-type":[193716],"msr-product-type":[],"msr-focus-area":[],"msr-platform":[],"msr-download-source":[],"msr-locale":[268875],"msr-field-of-study":[248485],"msr-conference":[],"msr-journal":[],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-937770","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-field-of-study-human-computer-interaction"],"msr_publishername":"ACM","msr_edition":"","msr_affiliation":"","msr_published_date":"2023-4-23","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"","msr_volume":"","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":1,"msr_main_download":"","msr_publicationurl":"","msr_doi":"","msr_publication_uploader":[{"type":"file","viewUrl":"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Escapement-CHI-2023.pdf","id":"937803","title":"escapement-chi-2023","label_id":"243109","label":0},{"type":"doi","viewUrl":"false","id":"false","title":"https:\/\/doi.org\/10.1145\/3544548.3581115","label_id":"243106","label":0}],"msr_related_uploader":"","msr_attachments":[{"id":937803,"url":"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/05\/Escapement-CHI-2023.pdf"}],"msr-author-ordering":[{"type":"text","value":"Molly Jane Nicholas","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Nicolai Marquardt","user_id":42630,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Nicolai Marquardt"},{"type":"user_nicename","value":"Michel Pahud","user_id":33007,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Michel Pahud"},{"type":"user_nicename","value":"Nathalie Henry Riche","user_id":33058,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Nathalie Henry Riche"},{"type":"guest","value":"hugo-romat","user_id":696222,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=hugo-romat"},{"type":"text","value":"Christopher Collins","user_id":0,"rest_url":false},{"type":"text","value":"David Ledo","user_id":0,"rest_url":false},{"type":"text","value":"Rohan Kadekodi","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Badrish Chandramouli","user_id":31166,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Badrish Chandramouli"},{"type":"user_nicename","value":"Ken Hinckley","user_id":32521,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Ken Hinckley"}],"msr_impact_theme":[],"msr_research_lab":[199565],"msr_event":[],"msr_group":[379814,957177],"msr_project":[937905,698833],"publication":[],"video":[],"download":[],"msr_publication_type":"inproceedings","_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/937770"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":7,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/937770\/revisions"}],"predecessor-version":[{"id":937902,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/937770\/revisions\/937902"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=937770"}],"wp:term":[{"taxonomy":"msr-content-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-content-type?post=937770"},{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=937770"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=937770"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=937770"},{"taxonomy":"msr-product-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-product-type?post=937770"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=937770"},{"taxonomy":"msr-platform","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-platform?post=937770"},{"taxonomy":"msr-download-source","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-download-source?post=937770"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=937770"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=937770"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=937770"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=937770"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=937770"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=937770"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}