SixthSense

The figure below shows the SixthSense architecture. The key components of the system, including the databases, inference engine, and applications, are run centrally by the enterprise. This provides the (trusted) inference engine access to the complete set of sensed data across all users, objects, and zones, allowing it to make effective inferences. Likewise, the (trusted) application is allowed the flexibility of working with a complete set of inferences (i.e., inferences pertaining to all users and their objects), yet control what processed information is presented to the users to ensure privacy. In contrast, if the inference engine or the application were run by individual users on their own desktop machines, privacy consideration would restrict the set of information made available to these, and hence limit their functionality.

The main components of SixthSense are:

RFID Monitor: The RFIDMonitor issues a read command every 500 ms to the RFID reader. The reader reports the EPC and the signal strength (RSSI) of the tags read via each of its antennas. This data then gets pushed into the raw database.

Other Enterprise Monitors: These monitor the information listed below and push their updates into the the raw database:

• Calendar Monitor: This resides on each user’s desktop machine, and monitors the time and location of the user’s appointments.
• Presence Monitor: This monitors the status of each user’s interaction with their desktop. A machine is said to be idle when it receives no user input for 2 minutes. Transitions from idle to active state are detected and reported.
• Login Monitor: This is similar to the presence monitor except that in general login is a much stronger indication of a user being present than simply a change in their machine’s idle.
• Cameras: Office buildings are often equipped with cameras for security reasons. The camera feed is stored in a video database for future analysis, if the need arises. We show in applications how we combine the camera system in an enterprise with other sensors to build useful applications.

Raw database: The RFID monitor and the other enterprise monitors push data into the raw database.

Inference engine: The inference engine operates on the raw database to draw inferences about people, objects, and workspaces. It makes the following inferences:

• Person-Object Differentiation: The goal of person-object differentiation is to take an undifferentiated mass of tags and classify each tag as either belonging to a person or to an object. A “person tag” refers to a tag affixed, say, to a person’s employee badge, which is (almost) always carried by him/her.
• Object Ownership: Having classified tags as people or objects, the goal of this module is to infer the owner (obviously a person) of each object.
• Zone Identification: We automatically classify RFID zones into individual workspace, shared workspace, reserved shared workspace and common areas.
• Person Identification: After the nature of tags (person vs. object) has been identified, we then infer the identity of the person corresponding to a person tag.
• Person-Object Interaction: Finally, we examine events within an RFID zone.Specifically, we infer that an object has been interacted with, e.g., picked up by a person.

Processed database: The processed database is populated by the inference engine with its inferences, making these available to applications built on top of the SixthSense platform.

API: SixthSense provides a set of APIs for applications to lookup the inferences stored in the processed database or to receive callbacks when new inferences are made.

We have prototyped a few applications using the APIs exposed by the SixthSense platform.

Misplaced Object Alert

The misplaced object alert service tracks the movement of users and objects, and notifies a user when it thinks that he/she may have misplaced an object. An object is said to be misplaced when it is in a shared workspace or a public area, and the object’s owner, who was also in the same zone, moves away to a different zone. When such a situation is detected, the system suspects that the object may have been misplaced and alerts the user via email or phone.

Annotating Video with Physical Events

We have built a simple application that the annotates the video feed from a camera with RFID events corresponding to inter-zone movement and object interaction. The video could, for instance, be from the security cameras in a building. With the video thus annotated with physical events, a user who is searching for a misplaced objected but suspects that it went missing in the lobby could issue a query like: «find me all segments of the security video recording corresponding to times when I passed through the lobby». This would then return the desired segments of video, relieving the user of the burden of sifting through hours of irrelevant video recording. A screenshot of the application we have built is shown below.

Automatic Conference Room Booking

SixthSense automatically identifies a shared space as a reserved space (e.g., a conference room) using the calendar information of users. While reserving such a room may be the norm, people may sometimes occupy it without reservation, if it is not already occupied and they are looking for space to hold an unplanned, last-minute meeting. Despite the barging in, if the room has in fact not been reserved for the time in question, it would be desirable to reserve it for the group that has barged in, to avoid the possibility of someone else trying to reserve the apparently free but in fact occupied room. We have built an automatic conference room booking application to accomplish precisely this.