{"id":308480,"date":"2016-10-18T15:43:27","date_gmt":"2016-10-18T22:43:27","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-research-item&p=308480"},"modified":"2019-09-25T11:45:19","modified_gmt":"2019-09-25T18:45:19","slug":"sensor-recognition-based-input-interaction","status":"publish","type":"msr-research-item","link":"https:\/\/www.microsoft.com\/en-us\/research\/publication\/sensor-recognition-based-input-interaction\/","title":{"rendered":"Sensor- and Recognition-Based Input for Interaction"},"content":{"rendered":"

Sensors convert a physical signal into an electrical signal that may be manipulated symbolically on a computer. A wide variety of sensors have been developed for aerospace, automotive, and robotics applications. Continual innovations in manufacturing and reductions in cost have allowed many sensing technologies to find application in consumer products. An interesting example is the development of the ubiquitous computer mouse. Douglas Engelbart’s original mouse, so nammed because its wire “tail” came out its end, used two metal wheels and a pair of potentiometers to sense the wheels rolling over a desk surface. Soon, mice used a ball and a pair of optical encoders to convert the movement of the hand into digital signals indicating precise relative motion. Now, even the most inexpensive mice use a specialized camera and image-processing algorithms to sense motions at the scale of one one-thousandth of an inch several thousand times per second. Accelerometers, devices that sense acceleration due to motion and he constant acceleration due to gravity, are another interesting example. Today’s tiny accelerometers were originally developed for application in automotive air-bag systems. Digital cameras now incorporate accelerometers to sense whether a picture is taken in landscape or portrait mode, and save the digital photo appropriately. Many laptops with built-in hard disks also include accelerometers to detect when the laptop\u00a0has been dropped, and park the hard drive before impact. Meanwhile, mobile phone manufacturers are experimenting with phones that use accelerometers to sense motion for use in interaction, such as in-the-air dialing, scrolling, and detecting the user’s walking pattern.<\/p>\n

While a wide array of sensors is available to researchers, rarely does a sensor address exactly the needs of a given application. Consider building into a computer the capability to sense when its user is frustrated. Detection of user frustration would allow a computer to respond by adopting a new strategy of interaction, playing soothing music, or even calling technical support; however, today, no “frustration meter” may be purchased at the local electronics store. What are the alternatives? A microphone could be used to sense when the user mutters or yells at the machine. A pressure sensor in the mouse and keyboard could detect whether the user is typing harder or squeezing the mouse in frustration (Klein, Moon, & Picard, 2002; Reynolds, 2001). A webcam might detect scowling or furrowing of the eyebrows. Sensors in the chair could detect user agitation (Tan, Slivovsky, & Pentland, 2001). Ultimately, the system chosen should probably exploit a consistent, predictable relationship between the output of one of these sensors and the users frustration level; for example, if the mouse is squeezed at a level exceeding some set threshold, the computer may conclude that the user is frustrated.<\/p>\n

In our effort o build a frustration detector, we may find a number of issues confounding the relationship between the sensors and the state to be detected:<\/p>\n