{"id":284729,"date":"2014-04-08T09:00:06","date_gmt":"2014-04-08T16:00:06","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=284729"},"modified":"2016-09-06T08:54:14","modified_gmt":"2016-09-06T15:54:14","slug":"can-robots-social-intelligence","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/can-robots-social-intelligence\/","title":{"rendered":"Can Robots Have Social Intelligence?"},"content":{"rendered":"

Visiting Eric Horvitz (opens in new tab)<\/span><\/a> at Microsoft Research headquarters in Redmond, Wash., is a full-service experience.<\/p>\n

At the building\u2019s first-floor bank of elevators, situated on a busy corridor, a \u201csmart elevator\u201d opens its doors, sensing that you need a ride.<\/p>\n

When you arrive on the third floor, a cute, humanoid robot makes eye contact, senses your interest in getting assistance, and cheerfully asks, \u201cDo you need directions?\u201d You tell him you\u2019re looking for Eric, and the robot gives you the office number and directions, gesturing with his arms and hands to explain the turns and twists of the path to Eric\u2019s office.<\/p>\n

\u201cBy the way,\u201d the robot adds, \u201cI\u2019ll let Monica, his virtual assistant, know that you\u2019re on your way.\u201d<\/p>\n

Down a couple of hallways, Monica is waiting for you, right outside Horvitz\u2019s office. She\u2019s an onscreen avatar with a British accent.<\/p>\n

\u201cI was expecting you,\u201d she says. \u201cThe robot told me you were coming.<\/p>\n

\"Monica,

Monica, a virtual assistant, greets Eric Horvitz outside his office<\/p><\/div>\n

Horvitz is in his office working on his computer, but Monica reflects about the costs and benefits of an unplanned interruption\u2014considering nuances of Horvitz\u2019s current desktop activity, calendar, and past behavior. Using a model learned from this trove of data, the system determines that a short interruption would be OK and suggests that you go on in.<\/p>\n

These experiences are all part of the Situated Interaction project, a research effort co-led by Horvitz, a Microsoft distinguished scientist and managing director of Microsoft Research Redmond (opens in new tab)<\/span><\/a>, and his colleague Dan Bohus (opens in new tab)<\/span><\/a>. They are creating a code base\u2014an architecture\u2014that enables many forms of complex, layered interaction between machines and humans.<\/p>\n

The interaction is \u201csituated\u201d in the sense that the machine has situational awareness\u2014it can take into account the physical optics of the space, the geometry of people\u2019s comings and goings, their gestures and facial expressions, and the give and take of conversation between multiple individuals\u2014among other factors that most humans consider second nature.<\/p>\n

\u201cWhen it comes to language-based interaction, people immediately think \u2018speech recognition,\u2019\u201d Bohus says, \u201cbut it is much more than that.\u201d<\/p>\n

The project relies on intensive integration of multiple computational competencies and methods, including machine vision, natural language processing, machine learning, automated planning, speech recognition, acoustical analysis, and sociolinguistics. It also pushes into a new area of research: how to automate processes and systems that understand multiparty interaction.<\/p>\n

\u201cFor years,\u201d Horvitz says, \u201cwe\u2019ve both been passionate about the question of how people, when they talk to each other in groups or in one-on-one settings, do so well to coordinate both meaning and turn taking\u2014often building a mutual understanding of one another.\u201d<\/p>\n

This includes discerning where attention is focused and understanding the differing roles of the people engaged in the conversation.<\/p>\n

\u201cWe\u2019re addressing core challenges in artificial intelligence,\u201d Horvitz says. \u201cThe goal is to build systems that can coordinate and collaborate with people in a fluid, natural manner.\u201d<\/p>\n

Enhancing Realism via Understanding Shared Context<\/h2>\n
\"Situated

Analysis of multiple perceptual streams on the Situated Interactions project.<\/p><\/div>\n

One direction in making interactions with computers more natural is incorporating an awareness of time and space. This work includes efforts to endow systems with notions of short-term and long-term memory, and to have access to histories of experience similar to the abilities of people to recall.<\/p>\n

Horvitz recalls a moment a couple of years back when he returned to his office from an all-day meeting and his virtual assistant said, \u201cHi, Eric, long time, no see!\u201d He knew immediately that research intern, Stephanie Rosenthal (opens in new tab)<\/span><\/a>, then a graduate student at Carnegie Mellon University, had successfully integrated humanlike short- and long-term memory into the system that day.<\/p>\n

\u201cThe interactions with the system, especially when we integrate new competencies, seem magical,\u201d Horvitz says, \u201cwith the magic emerging from a rich symphony of perception, reasoning, and decision-making that\u2019s running all the time.\u201d<\/p>\n

The evolving platform has continued to fuel new research directions. As the system\u2019s overall level of competency and naturalness increases, previously unrecognized gaps in ability become salient\u2014leading to new directions of research, such as the work on short- and long-term memory.<\/p>\n

\u201cAs you increase the degree of realism,\u201d Bohus says, \u201cand you get to something that acts and looks more human, it becomes uncanny when things that would be deficits in a human being show up.\u201d<\/p>\n

Horvitz\u2019s virtual assistant has a rich set of skills that includes the ability to access his online calendar, detect his presence and absence in the office, infer how busy he is, predict when he\u2019ll finish a certain task based on his past habits, predict when he\u2019ll return to his office and when he\u2019ll next read email, and even predict when he\u2019ll conclude a conversation, based on the length of his past conversations.<\/p>\n

The smart elevator, which uses a wide-angle camera attached to the ceiling to perceive the positions and movements of passers-by, takes the data and calculates the probability that someone needs a ride and isn\u2019t just walking past to get to the cafeteria or the building\u2019s atrium.<\/p>\n

\"Directions

The Directions Robot, part of the Situated Interaction project, is a welcoming, informative presence at Microsoft Research headquarters.<\/p><\/div>\n

The Directions Robot makes real-time inferences about the engagement, actions, and intentions of the people who approach it, with the aid of multiple sensors, a wide-angle camera, the Kinect for Windows (opens in new tab)<\/span><\/a> speech-recognition engine, and knowledge of the building and the people who work there. The robot can direct people to anyone and any place in the building, using the building directory and the building map.<\/p>\n

\u201cEach of these projects,\u201d Bohus notes, \u201cpushes the boundaries in different directions.\u201d<\/p>\n

Other Situated Interaction projects have included a virtual receptionist that schedules shuttle rides between buildings on the Microsoft campus. Like Monica and the Directions Robot, it can detect multiple people, infer a range of things about them, and conduct a helpful interaction. All of the systems can be refined continually based on different streams of collected data.<\/p>\n

\"Situated

Eric Horvitz (left) and Dan Bohus have had lots of help in constructing the Situated Interaction project.<\/p><\/div>\n

All of the Situated Interaction work, Bohus adds, has benefited from contributions by multiple researchers and engineers at Microsoft Research. Horvitz agrees.<\/p>\n

\u201cBeyond passionate researchers who have pursued these core challenges for decades,\u201d he says, \u201cthe projects rely on the contributions and creativity of a team of engineers.\u201d<\/p>\n

Among those on the Situated Interaction team are Anne Loomis Thompson (opens in new tab)<\/span><\/a> and Nick Saw (opens in new tab)<\/span><\/a>.<\/p>\n

The Next Frontiers of Situated Interaction<\/h2>\n

The array of possible applications for Situated Interaction technology is vast, from aerospace and medicine to education and disaster relief.<\/p>\n

The magical capabilities of the systems developed at Microsoft Research provide a technical vision and long-term directions for what Cortana aspires to deliver. A personal assistant for the recently released Windows Phone 8.1, Cortana uses natural language, machine learning, and contextual signals in providing assistance to users.<\/p>\n

Horvitz anticipates vigorous competition among technology companies to create personal assistants that operate across a person\u2019s various devices\u2014and provide a unified, supportive experience across all of them over time and space. He sees this competition as fueling progress in artificial intelligence.<\/p>\n

\u201cIt is going to be a cauldron of creativity and competition,\u201d he says. \u201cWho\u2019s going to produce the best digital assistant?\u201d<\/p>\n

Such an assistant could coordinate with the assistants of other people, helping to schedule social engagements, work commitments, and travel. It could anticipate your needs based on past activities\u2014such as where you have enjoyed dining\u2014and coordinate with businesses that offer special deals. It could help you select a movie based on which ones your friends liked.<\/p>\n

\u201cIntelligent, supportive assistants that assist and complement people are a key aspiration in computer science,\u201d Horvitz says, \u201cand basic research in this space includes gathering data and observing people conversing, collaborating, and assisting one another so we can learn how to best develop systems that can serve in this role.\u201d<\/p>\n

Horvitz also envisions deeper forms of collaboration between computers and people in the workplace. As you create presentations or documents, for example, computers could make suggestions about relevant graphical and informational content in a non-invasive manner. The interaction would be a fluid exchange, with both parties sharing and responding to ideas and making inferences about mutual goals.<\/p>\n

Horvitz believes that we are in \u201cthe early dawn of a computational revolution.\u201d As he said in a recent TEDx talk (opens in new tab)<\/span><\/a>, \u201c[M]y view is that these methods will come to surround people in a very supportive and nurturing way. The intellects we\u2019re developing will complement our own human intellect and empower us to do new kinds of things and extend us in new kinds of ways according to our preferences and our interests.\u201d<\/p>\n

Publications<\/strong><\/p>\n