{"id":284699,"date":"2014-10-05T18:00:46","date_gmt":"2014-10-06T01:00:46","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=284699"},"modified":"2016-08-30T08:20:53","modified_gmt":"2016-08-30T15:20:53","slug":"strong-sense-natural-interactions","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/strong-sense-natural-interactions\/","title":{"rendered":"A Strong Sense for Natural Interactions"},"content":{"rendered":"
\"Hrvoje

Hrvoje Benko<\/p><\/div>\n

This week, Microsoft researcher Hrvoje Benko (opens in new tab)<\/span><\/a> (@hrvojebenko (opens in new tab)<\/span><\/a>) is in Hawaii, but not on one of the islands\u2019 beautiful beaches. As conference chair for UIST 2014 (opens in new tab)<\/span><\/a>\u2014the 27th Association for Computing Machinery (ACM) Symposium on User Interface Software and Technology\u2014Benko will be busy ensuring that the event, the premier forum for innovations in the software and technology of human-computer interaction (HCI), proceeds smoothly.<\/p>\n

In addition to serving as conference chair, Benko\u2014subject of a new Microsoft Research Luminaries video (opens in new tab)<\/span><\/a>\u2014has contributed to three of the eight Microsoft papers being presented during the conference, including Sensing Techniques for Tablet+Stylus Interaction<\/em> (opens in new tab)<\/span><\/a>, which won an ACM UIST 2014 Best Paper Award. This is the fourth paper he\u2019s co-written that\u2019s received a best-paper award by a major conference, evidence of a remarkable career for a young scientist who joined Microsoft Research in 2007, just after earning his Ph.D.<\/p>\n

Bringing Augmented Reality to Ordinary Spaces<\/h2>\n

The award-winning paper, though, might not be the most attention-grabbing contribution to UIST 2014 from Benko and his colleagues. Many attendees are certain to be amazed by the research featured in the paper RoomAlive: Magical Experiences Enabled by Scalable, Adaptive Projector-Camera Units<\/em> (opens in new tab)<\/span><\/a>, written by Benko and a host of Microsoft and academic collaborators.<\/p>\n

RoomAlive (opens in new tab)<\/span><\/a> uses a unified, scalable, multiprojector system that adapts gaming content to its room and explores additional methods for physical interaction in the room, resulting in an immersive gaming experience.<\/p>\n

\u201cRoomAlive,\u201d Benko says, \u201cenables any space to be transformed into an augmented, interactive display.\u201d<\/p>\n

It will be a while, though, before that proof-of-concept could become a practical reality, but it certainly is tantalizing. The RoomAlive prototype deploys projectors and depth cameras to cover the entire room\u2014including the furniture and people inside\u2014with pixels capable of being used for both input and output. The presentation results in an experience that coexists seamlessly with the existing physical environment.<\/p>\n

The system that drives RoomAlive uses multiple projector-camera units\u2014referred to as \u201cprocams\u201d\u2014consisting of a depth camera, a wide-field-of-view projector, and a computer. These devices are combined via a scalable, distributed framework to cover an entire room. The procams are auto-calibrating and can self-localize as long as their views have some overlap.<\/p>\n

\u201cOur system enables new interactive projection mapping experiences that dynamically adapts content to any room,\u201d Benko says. \u201cUsers can touch, shoot, stomp, dodge, and steer projected content that seamlessly coexists with their existing physical environment.\u201d<\/p>\n

Another augmented-reality paper being presented during UIST 2014, written by Benko, Microsoft researcher Andy Wilson (opens in new tab)<\/span><\/a> and designer Federico Zannier, uses different technologies, this time to support face-to-face\u2014\u201cdyadic\u201d\u2014interaction with 3-D virtual objects (opens in new tab)<\/span><\/a>. Dyadic Projected Spatial Augmented Reality<\/em> (opens in new tab)<\/span><\/a> combines dynamic projection mapping, multiple perspective views, and device-less interaction.<\/p>\n

The main advantage of spatial augmented reality (SAR) over more traditional augmented-reality approaches, such as handheld devices with composited graphics or see-through, head-worn displays, is that users are able to interact with 3-D virtual objects and each other without bulky equipment with a limited field of view that hinders face-to-face interaction. Instead, SAR projects the augmenting graphics over the physical object itself and avoids diverting the users\u2019 attention from the real world.<\/p>\n

\u201cThat used to be the domain of theme parks or highly immersive theaters,\u201d Benko says. \u201cYou needed a space that was designed for the experience. But now, with enough computing power, depth cameras, and projectors, it\u2019s possible to create these immersive environments within an ordinary living space.<\/p>\n

\u201cAugmented reality fundamentally changes the nature of communication, with rich interactions not just for entertainment, but also for work and collaboration.\u201d<\/p>\n

Enabling More Natural, Nuanced Interactions<\/h2>\n

The paper that won a UIST 2014 best-paper award represents a huge joint effort. Written by team lead Ken Hinckley (opens in new tab)<\/span><\/a> along with colleagues, Benko, Michel Pahud (opens in new tab)<\/span><\/a>, Pourang Irani, Fran\u00e7ois Guimbreti\u00e8re, Marcel Gavriliu, Xiang ‘Anthony’ Chen, Fabrice Matulic, Bill Buxton (opens in new tab)<\/span><\/a>, and Wilson, it explores grip and motion sensing with a tablet and a stylus (opens in new tab)<\/span><\/a>.<\/p>\n

\u201cWe\u2019re at the point now where small mobile devices, such as pens, tablets, or phones, can be equipped with sensors to help us understand their use,\u201d Benko says. \u201cThe way you grasp the tablet, whether you hold the stylus in a writing grip or tucked between your fingers\u2014these all affect the position and movement of your gestures.\u201d<\/p>\n

The biomechanics behind each task are anything but ordinary, involving the interplay between two hands, each containing 27 bones, more than 30 muscles, nearly 50 nerves, and 30 or so arteries.<\/p>\n

The team\u2019s goal was to capitalize on the hand\u2019s dexterity to explore new frontiers in human-computer interaction using new sensing techniques.<\/p>\n

\u201cHow can we interpret the signals correctly, more accurately?\u201d Benko muses. \u201cThat\u2019s the larger goal of the work: Instead of assuming explicit interaction, how can we enable a more natural and nuanced interface based on context of use? How can we impart new subtleties to interactions on mobile devices?\u201d<\/p>\n

Current device-interaction models tend to rely on having the user explicitly select the state of interaction. Models in which devices present context-based behaviors are few and still fairly simple: accelerometers, which flip screens between portrait and landscape modes, or Bluetooth, which makes automatic, appropriate connections to a vehicle, a home phone, or a computer. The researchers wanted to extend the interaction vocabulary and build on the existing range of gestures so that more nuanced context is possible.<\/p>\n

\u201cIt\u2019s an attempt to create a set of gestures that are meaningful to a particular task and span the space of possibilities,\u201d he says. \u201cWe know that not all the gestures are going to be successful. But it\u2019s only by introducing them and obtaining feedback that this vocabulary will eventually reduce down to a few highly functional interactions that get widely adopted. This is still a long ways off from moving into the mainstream, but this is the exciting part for researchers\u2014opening up new possibilities and working to make them useful.\u201d<\/p>\n

Fascinated by the Human Aspects of Computing<\/h2>\n

HCI first caught Benko\u2019s interest during his undergraduate years.<\/p>\n

\u201cI was doing a lot of programming but found myself asking, \u2018Where do people<\/em> come into the equation?\u2019\u201d says Benko, a native of Croatia. \u201cSo for my graduate studies at Columbia University, I continued on with computing but focused on the human aspects. I was really lucky to work there with some of the pioneers in augmented reality.\u201d<\/p>\n

Currently, augmented reality is what captures his imagination the most. The notion that technology can augment our senses and alter how we comprehend reality was what initially inspired him to go into HCI research.<\/p>\n

\u201cIt\u2019s almost like you\u2019re giving people superpowers,\u201d he says, laughing. \u201cI was fascinated with the idea that computing could be a tool that lets you do things you couldn\u2019t do before, or do them faster, or change how you perceive reality. I was just really interested in the notion that computing should be about interacting with people.\u201d<\/p>\n

Flying High at Microsoft Research<\/h2>\n

Since joining Microsoft, Benko\u2019s work has spanned many different areas, from augmented reality, computational illumination, surface computing, and new input form factors and devices, to touch and freehand gestural input. When he arrived in the United States at age 16, though, Benko never imagined he would contribute to award-winning papers or collaborate with eminent scientists on scientific papers and journals.<\/p>\n

\u201cI came to the U.S. through an exchange-student scholarship,\u201d he says. \u201cI attended a prep school, at one of those places with old buildings straight out of the Dead Poets Society<\/em>. I had a really good time, my scholarship got extended for a second year, and I went on to university in the U.S., and then graduate school.\u201d<\/p>\n

Benko calls his 2005 internship at Microsoft an \u201ceye-opener.\u201d For one thing, he found himself in an office a few doors down the hall from researchers whose papers he had been reading and referencing.<\/p>\n

\u201cThere were so many luminaries whose work I revered,\u201d he recalls. \u201cNot in my wildest dreams could I have imagined having lunch and chatting about my work with Andy Wilson or Ken Hinckley. They\u2019re world-renowned experts in their fields.\u201d<\/p>\n

While sometimes working at Microsoft Research can feel like a day at the beach for Benko, this week won\u2019t. With three papers and a demo to present, not to mention his responsibilities as conference chair, the beach, ironically, might have to be experienced virtually.<\/p>\n","protected":false},"excerpt":{"rendered":"

This week, Microsoft researcher Hrvoje Benko (@hrvojebenko) is in Hawaii, but not on one of the islands\u2019 beautiful beaches. As conference chair for UIST 2014\u2014the 27th Association for Computing Machinery (ACM) Symposium on User Interface Software and Technology\u2014Benko will be busy ensuring that the event, the premier forum for innovations in the software and technology […]<\/p>\n","protected":false},"author":39507,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[194476,194481],"tags":[211175,211178,210983,201873,205455,211217,187412,211220,211172],"research-area":[13552,13554],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-284699","post","type-post","status-publish","format-standard","hentry","category-devices-and-hardware","category-human-centered-computing","tag-acm-symposium-on-user-interface-software-and-technology","tag-acm-uist-2014-best-paper-award","tag-association-for-computing-machinery-acm","tag-hrvoje-benko","tag-human-computer-interaction-hci","tag-immersive-gaming-experience","tag-roomalive","tag-spatial-augmented-reality-sar","tag-uist-2014","msr-research-area-hardware-devices","msr-research-area-human-computer-interaction","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[171413,171406],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"October 5, 2014","formattedExcerpt":"This week, Microsoft researcher Hrvoje Benko (@hrvojebenko) is in Hawaii, but not on one of the islands\u2019 beautiful beaches. As conference chair for UIST 2014\u2014the 27th Association for Computing Machinery (ACM) Symposium on User Interface Software and Technology\u2014Benko will be busy ensuring that the event,…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/284699"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/39507"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=284699"}],"version-history":[{"count":2,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/284699\/revisions"}],"predecessor-version":[{"id":284717,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/284699\/revisions\/284717"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=284699"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=284699"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=284699"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=284699"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=284699"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=284699"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=284699"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=284699"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=284699"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=284699"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=284699"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}