{"id":282431,"date":"2014-04-17T12:01:57","date_gmt":"2014-04-17T19:01:57","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=282431"},"modified":"2016-08-23T20:20:59","modified_gmt":"2016-08-24T03:20:59","slug":"anticipating-more-from-cortana","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/anticipating-more-from-cortana\/","title":{"rendered":"Anticipating More from Cortana"},"content":{"rendered":"

Most of us can only dream of having the perfect personal assistant, one who is always there when needed, anticipating our every request and unobtrusively organizing our lives. Cortana<\/a>, the new digital personal assistant powered by Bing<\/a> that comes with Windows Phone 8.1<\/a>, brings users closer to that dream.<\/p>\n

For Larry Heck<\/a>, a distinguished engineer in Microsoft Research, this first release offers a taste of what he has in mind. Over time, Heck wants Cortana to interact in an increasingly anticipatory, natural manner.<\/p>\n

Cortana already offers some of this behavior. Rather than just performing voice-activated commands, Cortana continually learns about its user and becomes increasingly personalized, with the goal of proactively carrying out the right tasks at the right time. If its user asks about outside temperatures every afternoon before leaving the office, Cortana will learn to offer that information without being asked.<\/p>\n

Furthermore, if given permission to access phone data, Cortana can read calendars, contacts, and email to improve its knowledge of context and connections. Heck, who plays classical trumpet in a local orchestra, might receive a calendar update about a change in rehearsal time. Cortana would let him know about the change and alert him if the new time conflicts with another appointment.<\/p>\n

Research Depth and Breadth an Advantage<\/h2>\n

While many people would categorize such logical associations and humanlike behaviors under the term “artificial intelligence” (AI), Heck points to the diversity of research areas that have contributed to Cortana\u2019s underlying technologies. He views Cortana as a specific expression of Microsoft Research\u2019s work on different areas of personal-assistant technology.<\/p>\n

\u201cThe base technologies for a virtual personal assistant include speech recognition, semantic\/natural language processing, dialogue modeling between human and machines, and spoken-language generation,\u201d he says. \u201cEach area has in it a number of research problems that Microsoft Research has addressed over the years. In fact, we\u2019ve pioneered efforts in each of those areas.\u201d<\/p>\n

\"Cortana

The Cortana user interface.<\/p><\/div>\n

Cortana\u2019s design philosophy is therefore entrenched in state-of-the-art machine-learning and data-mining algorithms. Furthermore both developers and researchers are able to use Microsoft\u2019s broad assets across commercial and enterprise products, including strong ties to Bing web search and Microsoft speech algorithms and data.<\/p>\n

If Heck has set the bar high for Cortana\u2019s future, it\u2019s because of the deep, varied expertise within Microsoft Research.<\/p>\n

\u201cMicrosoft Research has a long and broad history in AI,\u201d he says. \u201cThere are leading scientists and pioneers in the AI field who work here. The underlying vision for this work and where it can go was derived from Eric Horvitz<\/a>\u2019s work on conversational interactions and understanding, which go as far back as the early \u201990s. Speech and natural language processing are research areas of long standing, and so is machine learning. Plus, Microsoft Research is a leader in deep-learning and deep-neural-network research.\u201d<\/p>\n

From Foundational Technology to Overall Experience<\/h2>\n

In 2009, Heck started what was then called the conversational-understanding (CU) personal-assistant effort at Microsoft.<\/p>\n

\u201cI was in the Bing research-and-development team reporting to Satya Nadella<\/a>,\u201d Heck says, \u201cworking on a technology vision for virtual personal assistants. Steve Ballmer<\/a> had recently tapped Zig Serafin to unify Microsoft\u2019s various speech efforts across the company, and Zig reached out to me to join the team as chief scientist. In this role and working with Zig, we began to detail out a plan to build what is now called Cortana.\u201d<\/p>\n

\"Cortana

Researchers who worked on the Cortana product (from left): top row, Malcolm Slaney, Lisa Stifelman, and Larry Heck; bottom row, Gokhan Tur, Dilek Hakkani-T\u00fcr, and Andreas Stolcke.<\/p><\/div>\n

Heck and Serafin established the vision, mission, and long-range plan for Microsoft\u2019s digital-personal-assistant technology, based on scaling conversations to the breadth of the web, and they built a team with the expertise to create the initial prototypes for Cortana. As the effort got off the ground, Heck\u2019s team hired and trained several Ph.D.-level engineers for the product team to develop the work.<\/p>\n

\u201cBecause the combination of search and speech skills is unique,\u201d Heck says, \u201cwe needed to make sure that Microsoft had the right people with the right combination of skills to deliver, and we hired the best to do it.\u201d<\/p>\n

After the team was in place, Heck and his colleagues joined Microsoft Research to continue to think long-term, working on next-generation personal-assistant technology.<\/p>\n

Some of the key researchers in these early efforts included Microsoft Research senior researchers Dilek Hakkani-T\u00fcr and Gokhan Tur, and principal researcher Andreas Stolcke<\/a>. Other early members of Heck\u2019s team included principal research software developer Madhu Chinthakunta, and principal user-experience designer Lisa Stifelman<\/a>.<\/p>\n

\u201cWe started out working on the low-level, foundational technology,\u201d Heck recalls. \u201cThen, near the end of the project, our team was doing high-level, all-encompassing usability studies that provided guidance to the product group. It was kind of like climbing up to the crow\u2019s nest of a ship to look over the entire experience.<\/p>\n

\u201cResearch manager Geoff Zweig<\/a> led usability studies in Microsoft Research. He brought people in, had them try out the prototype, and just let them go at it. Then we would learn from that. Microsoft Research was in a good position to study usability, because we understood the base technology as well as the long-term vision and how things should work.\u201d<\/p>\n

The Long-Term View<\/h2>\n

Heck has been integral to Cortana since its inception, but even before coming to Microsoft in 2009, he already had contributed to early research on CU personal assistants. While at SRI International in the 1990s, his tenure included some of the earliest work on deep-learning and deep-neural-network technology.<\/p>\n

Heck was also part of an SRI team whose efforts laid the groundwork for the CALO AI project funded by the U.S. government\u2019s Defense Advanced Research Projects Agency. The project aimed to build a new generation of cognitive assistants that could learn from experience and reason intelligently under ambiguous circumstances. Later roles at Nuance Communications and Yahoo! added expertise in research areas vital to contributing to making Cortana robust.<\/p>\n

\"notebook

The notebook menu for Cortana.<\/p><\/div>\n

Not surprisingly, Heck\u2019s perspectives extend to a distant horizon.<\/p>\n

\u201cI believe the personal-assistant technology that\u2019s out there right now is comparable to the early days of search,\u201d he says, \u201cin the sense that we still need to grow the breadth of domains that digital personal assistants can cover. In the mid-\u201990s, before search, there was the Yahoo! directory. It organized information, it was popular, but as the web grew, the directory model became unwieldy. That\u2019s where search came in, and now you can search for anything that\u2019s on the web.\u201d<\/p>\n

He sees personal-assistant technology traveling along a similar trajectory. Current implementations target the most common functions, such as reminders and calendars, but as technology matures, the personal assistant has to extend to other domains so that users can get any information and conduct any transaction anytime and anywhere.<\/p>\n

\u201cMicrosoft has intentionally built Cortana to scale out to all the different domains,\u201d Heck says. \u201cHaving a long-term vision means we have a long-term architecture. The goal is to support all types of human interaction\u2014whether it\u2019s speech, text, or gestures\u2014across domains of information and function and make it as easy as a natural conversation.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

Most of us can only dream of having the perfect personal assistant, one who is always there when needed, anticipating our every request and unobtrusively organizing our lives. Cortana, the new digital personal assistant powered by Bing that comes with Windows Phone 8.1, brings users closer to that dream. For Larry Heck, a distinguished engineer […]<\/p>\n","protected":false},"author":39507,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[194466,194467,194455,194456,194462],"tags":[186604,195185,210413,186925,201271,210410,210398,186936,210395,210401,197281,210404,210407,204609],"research-area":[13561,13556,13545],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-282431","post","type-post","status-publish","format-standard","hentry","category-algorithms","category-artifical-intelligence","category-machine-learning","category-natural-language-processing","category-speech-and-dialog","tag-bing","tag-cortana","tag-data-mining-algorithms","tag-deep-learning","tag-deep-neural-network","tag-dialogue-modeling","tag-larry-heck","tag-natural-language-processing","tag-perfect-personal-assistant","tag-personal-assistant-technology","tag-speech-recognition","tag-spoken-language-generation","tag-virtual-personal-assistant","tag-windows-phone-8-1","msr-research-area-algorithms","msr-research-area-artificial-intelligence","msr-research-area-human-language-technologies","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","byline":"","formattedDate":"April 17, 2014","formattedExcerpt":"Most of us can only dream of having the perfect personal assistant, one who is always there when needed, anticipating our every request and unobtrusively organizing our lives. Cortana, the new digital personal assistant powered by Bing that comes with Windows Phone 8.1, brings users…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/282431"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/39507"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=282431"}],"version-history":[{"count":5,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/282431\/revisions"}],"predecessor-version":[{"id":282482,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/282431\/revisions\/282482"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=282431"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=282431"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=282431"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=282431"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=282431"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=282431"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=282431"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=282431"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=282431"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=282431"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=282431"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}