A Background Perspective on Touch as a Multimodal (and Multisensor) Construct

This chapter illustrates, through a series of examples, seven different perspectives of how touch input can be re-framed and re-conceived as a multimodal, multisensor construct.

These perspectives often can particularly benefit from considering the background of interaction (per Buxton’s 1995 model)—that is, interaction that takes place “behind” the foreground of the user’s conscious attention, in response to sensed contextual information. For example, with touch-screen input, the user’s intentional contact with the screen would comprise the foreground act, but the resulting vibrational forces imparted to the device can be sensed and leveraged “in the background” to infer additional contextual details of the touch.

Over the years, I’ve found this background perspective extremely useful as a tool-for-thought to devise novel interactions, especially when multiple modalities and multiple sensors can be used simultaneously, in complementary and mutually reenforcing ways. This approach can be especially helpful to break out of whatever preconceptions one might have regarding an input modality, even something as seemingly well studied and well understood as touch, which just for that reason will provide us with the bulk of the examples that I draw from in this chapter.

These perspectives of touch range from its traditional view as a modality that affords direct and intentional touchscreen input, to the inadvertent (yet still potentially valuable) phenomenon of unintentional (or “Midas”) touch. We will consider various combinations of touch with other sensor signals such as tilt, inertial motion, grip sensing, and above-screen pre-touch sensing. We’ll also discuss interesting ways to use pen and touch as complementary modalities for bimanual interaction. The focus of the chapter is design-centric, with contributions that focus on invention and innovation, rather than contributions, say, of formal experimental analysis or recognition methodologies. Likewise, the focus here is on the input side of touch, rather than the output side—as afforded by haptic and tactile feedback technologies.

The key point of this chapter is that low-level “sensing” channels such as grip and proximity and motion can be conceived of as new modalities that afford natural interaction with devices. And, likewise, it is possible to subtly shift our perspective of “modalities” to consider novel ways that they may yield insights of sensing, particularly if considered from the oft-neglected perspective that the background of interaction can support the user’s task focus in the foreground of attention most effectively.