Jogger: Models for Context-Sensitive Reminding

The Tenth International Conference on Autonomous Agents and Multiagent Systems |

Short Paper

Download BibTex

We describe research on principles of context-sensitive reminding that show promise for serving in systems that work to jog peoples’ memories about information that they may forget. The methods center on the construction and use of a set of distinct probabilistic models that predict (1) items that may be forgotten, (2) the expected relevance of the items in a situation, and (3) the cost of interruption associated with alerting about a reminder. We describe the use of this set of models in the Jogger prototype that employs predictions and decision-theoretic optimization to compute the value of reminders about meetings.

The Assistant: Situated Interaction Project (2012)

The Assistant was a long-running AI system developed as part of the Situated Interaction project (opens in new tab) at Microsoft Research. Designed to function as a working administrative assistant, it was stationed outside the office of Eric Horvitz—then Lab Director at Microsoft Research Redmond. This video showcases the Assistant in action, highlighting its capabilities across a variety of scenarios. You can also see the system operate “in the wild” in this TED talk. (opens in new tab)

The Assistant served as an exploratory AI research testbed, blending multiple strands of AI into a unified, real-world application. Built to operate in the dynamic environment of a research lab, the Assistant helped coordinate meetings with Eric and briefed him on missed events upon his return. It was capable of engaging in multiparty dialogue, drawing on natural language processing, machine vision, speech recognition, and acoustical sensing. The Assistant project was co-led by Dan Bohus and Eric Horvitz, with significant contributions from Anne Loomis Thompson, Paul Koch, Tomislav Pejsa, Michael Cohen, James Mahoney.

The Assistant was a descendant of the earlier Receptionist project, a research effort on multiparty dialog capabilities. The project took an “integrative AI” approach—bringing together a constellation of technologies to create a cohesive, intelligent agent with the intuitions of a long-term administrative assistant. The Assistant leveraged several specialized systems that had previously been developed as standalone research efforts, including:

  • Coordinate – Uses machine learning to predict someone’s presence and availability, including forecasts of return times and when they would next check email. System considered predictions of meetings someone was likely to skip, allowing others to “pencil in” meetings accordingly.
  • BusyBody – Assesses the cost of interrupting someone based on contextual information such as desktop activity, conversation, and location. Busybody was part of longer-term studies on the use of machine learning about the cost of interruption and recovery from disruptions. 
  • Jogger – Use of machine learning to predict likelihood of forgetting information that would be valuable in a setting.
  • Priorities – Ranks unread emails by estimating the cost of delayed review.
  • Models for multiparty engagement – Enables systems to recognize and support dialog with multiple people in a joint conversation.
  • Multichannel grounding – Considers uncertainty at multiple levels, including vision, speech recognition, natural language understanding, and core assistant domain and provided linguistic and gestural cues about uncertainty aimed at resolution.

The Assistant operated for several years, acting as an auxiliary aide until Eric transitioned to a new role as Director of Microsoft Research and moved to a different office.