Attention-Sensitive Alerting

Proceedings of UAI '99, Conference on Uncertainty and Artificial Intelligence, Stockholm, Sweden, Morgan Kaufmann: San Francisco. |

Download BibTex

We introduce utility-directed procedures for mediating the flow of potentially distracting alerts and communications to computer users. We present models and inference procedures that balance the context-sensitive costs of deferring alerts with the cost of interruption. We describe the challenge of reasoning about such costs under uncertainty via an analysis of user activity and the content of notifications. After introducing principles of attention-sensitive alerting, we focus on the problem of guiding alerts about email messages. We dwell on the problem of inferring the expected criticality of email and discuss work on the Priorities system, centering on prioritizing email by criticality and modulating the communication of notifications to users about the presence and nature of incoming email.

Information Agents: Directions and Futures (2001)

In this internal Microsoft video, produced in 2001 and released publicly in 2020, research scientist Eric Horvitz provides glimpses of a set of research systems developed within Microsoft’s research division between 1998 and 2001. Projects featured in the video include Priorities (opens in new tab), Lookout (opens in new tab), Notification Platform (opens in new tab), DeepListener (opens in new tab), and Bestcom (opens in new tab). The projects show early uses of machine learning, perception, and reasoning aimed at supporting people in daily tasks and at making progress on longer-term missions of augmenting human intellect. The efforts are thematically related in their pursuit of broader understandings of people and context, including a person’s attention, goals, activities, and location, via multimodal signals, involving the analysis of multiple streams of information. Several of the prototype systems were built within the Attentional User Interface (AUI) project (opens in new tab), which had focused on using machine learning and reasoning to understand a computer users’ cognitive load, changing focus of attention, and information needs across multiple devices (NYTimes article, 2000) (opens in new tab).

Demonstration of Priorities & Notification Platform (2001)

Eric Horvitz with Bill Gates at Envision 2001

In this 2001 video, Bill Gates hosts Eric Horvitz at the Envision 2001 meeting. Eric demonstrates the Priorities and Notification Platform systems.

Priorities, fielded internally at Microsoft in 1998, demonstrated the use of machine learning to control email prioritization, alerting, and routing. Priorities is the first system to prioritize email by urgency. The system was an ancestor of the Outlook Mobile Manager and Outlook’s Focus Inbox. Priorities sorts incoming email by assigning a measure of the “expected cost of delayed review” to each incoming email message. The system learns by observing users interact with email or via direct input from users. In a mobile messaging function, Priorities selectively routes the most urgent messages to users’ cellphones via SMS messages. To perform this function, the system considers predictions about the amount of time that users will be away from their desktop machines.

Notification Platform was an experimental system constructed to explore and demonstrate general principles and architectures for balancing the value of information awareness and cost of interruption. The system demonstrates how sensing and inferences about context, attention, and activities of a user can be harnessed to guide the flow of information to users from multiple information sources across multiple devices and alerting modalities. Notification Platform employs Bayesian models that jointly predict likelihoods of activities, location, and attention from a multimodal stream of information, including desktop activity, facial pose recognition and conversation detection. Probabilistic and decision-theoretic procedures are used to perform an economic analysis, weighing the benefits of information awareness with the costs of interruption, assigning dollar values to each item coming into a “universal inbox.” Work on the Notification Platform was featured in a New York Times article in 2000.

Microsoft Notification Platform

Notification Platform

The Assistant: Situated Interaction Project (2012)

The Assistant was a long-running AI system developed as part of the Situated Interaction project (opens in new tab) at Microsoft Research. Designed to function as a working administrative assistant, it was stationed outside the office of Eric Horvitz—then Lab Director at Microsoft Research Redmond. This video showcases the Assistant in action, highlighting its capabilities across a variety of scenarios. You can also see the system operate “in the wild” in this TED talk. (opens in new tab)

The Assistant served as an exploratory AI research testbed, blending multiple strands of AI into a unified, real-world application. Built to operate in the dynamic environment of a research lab, the Assistant helped coordinate meetings with Eric and briefed him on missed events upon his return. It was capable of engaging in multiparty dialogue, drawing on natural language processing, machine vision, speech recognition, and acoustical sensing. The Assistant project was co-led by Dan Bohus and Eric Horvitz, with significant contributions from Anne Loomis Thompson, Paul Koch, Tomislav Pejsa, Michael Cohen, James Mahoney.

The Assistant was a descendant of the earlier Receptionist project, a research effort on multiparty dialog capabilities. The project took an “integrative AI” approach—bringing together a constellation of technologies to create a cohesive, intelligent agent with the intuitions of a long-term administrative assistant. The Assistant leveraged several specialized systems that had previously been developed as standalone research efforts, including:

  • Coordinate – Uses machine learning to predict someone’s presence and availability, including forecasts of return times and when they would next check email. System considered predictions of meetings someone was likely to skip, allowing others to “pencil in” meetings accordingly.
  • BusyBody – Assesses the cost of interrupting someone based on contextual information such as desktop activity, conversation, and location. Busybody was part of longer-term studies on the use of machine learning about the cost of interruption and recovery from disruptions. 
  • Jogger – Use of machine learning to predict likelihood of forgetting information that would be valuable in a setting.
  • Priorities – Ranks unread emails by estimating the cost of delayed review.
  • Models for multiparty engagement – Enables systems to recognize and support dialog with multiple people in a joint conversation.
  • Multichannel grounding – Considers uncertainty at multiple levels, including vision, speech recognition, natural language understanding, and core assistant domain and provided linguistic and gestural cues about uncertainty aimed at resolution.

The Assistant operated for several years, acting as an auxiliary aide until Eric transitioned to a new role as Director of Microsoft Research and moved to a different office.