Models of Attention in Computing and Communication: From Principles to Applications
Over the last five years, our team at Microsoft Research has explored, within the Attentional User Interface (AUI) project, opportunities for enhancing computing and communications systems by treating human attention as a central construct and organizing principle. We introduced to the research community a broad array of AUI challenges and opportunities, including (1) the treatment of attention as a rare commodity and critical currency in sensing and reasoning about the information awareness versus disruption of users, (2) the use of attentional cues as an important source of rich signals about goals, intentions, and topics of interest, and (3) the triaging of computation, bandwidth, and rendering resources in guiding precomputation and prefetching with forecasts of future attention. We shall first describe several principles and methodologies centering on integrating models of attention into human-computer interaction. Then, we shall review representative efforts that illustrate how we can harness these principles in attention-sensitive messaging and mixed-initiative interaction applications.
Information Agents: Directions and Futures (2001)
In this internal Microsoft video, produced in 2001 and released publicly in 2020, research scientist Eric Horvitz provides glimpses of a set of research systems developed within Microsoft’s research division between 1998 and 2001. Projects featured in the video include Priorities (opens in new tab), Lookout (opens in new tab), Notification Platform (opens in new tab), DeepListener (opens in new tab), and Bestcom (opens in new tab). The projects show early uses of machine learning, perception, and reasoning aimed at supporting people in daily tasks and at making progress on longer-term missions of augmenting human intellect. The efforts are thematically related in their pursuit of broader understandings of people and context, including a person’s attention, goals, activities, and location, via multimodal signals, involving the analysis of multiple streams of information. Several of the prototype systems were built within the Attentional User Interface (AUI) project (opens in new tab), which had focused on using machine learning and reasoning to understand a computer users’ cognitive load, changing focus of attention, and information needs across multiple devices (NYTimes article, 2000) (opens in new tab).
Demonstration of Priorities & Notification Platform (2001)
Eric Horvitz with Bill Gates at Envision 2001
In this 2001 video, Bill Gates hosts Eric Horvitz at the Envision 2001 meeting. Eric demonstrates the Priorities and Notification Platform systems.
Priorities, fielded internally at Microsoft in 1998, demonstrated the use of machine learning to control email prioritization, alerting, and routing. Priorities is the first system to prioritize email by urgency. The system was an ancestor of the Outlook Mobile Manager and Outlook’s Focus Inbox. Priorities sorts incoming email by assigning a measure of the “expected cost of delayed review” to each incoming email message. The system learns by observing users interact with email or via direct input from users. In a mobile messaging function, Priorities selectively routes the most urgent messages to users’ cellphones via SMS messages. To perform this function, the system considers predictions about the amount of time that users will be away from their desktop machines.
Notification Platform was an experimental system constructed to explore and demonstrate general principles and architectures for balancing the value of information awareness and cost of interruption. The system demonstrates how sensing and inferences about context, attention, and activities of a user can be harnessed to guide the flow of information to users from multiple information sources across multiple devices and alerting modalities. Notification Platform employs Bayesian models that jointly predict likelihoods of activities, location, and attention from a multimodal stream of information, including desktop activity, facial pose recognition and conversation detection. Probabilistic and decision-theoretic procedures are used to perform an economic analysis, weighing the benefits of information awareness with the costs of interruption, assigning dollar values to each item coming into a “universal inbox.” Work on the Notification Platform was featured in a New York Times article in 2000.
Notification Platform