Theo, right, a 12-year-old boy who is blind, interacts with Microsoft senior research software development engineers Tim Regan, middle, and Martin Grayson, left, during user testing of the Project Tokyo system.

Project Tokyo

Project Tokyo aims to understand how to create a visual agent technology that is both useful and usable in the real world by focusing on how AI technology can help to augment people’s own capabilities. Specifically, how can agent technologies amplify existing skills and abilities to help people do more? Project Tokyo explores these questions through human-centric technical innovation, partnering closely with users. The project has worked closely with people who are blind and low vision to concretely anchor our exploration into their needs, interests, and abilities. As early adopters of agent technologies, we have found that blind and low-vision users are particularly well suited for this research.

Project Tokyo has inspired a wide variety of research: Ethnographic work with Paralympic athletes and spectators guided system concept and design. Work with a team of blind and low-vision users helped to refine interaction approaches. Recent work is exploring how such a system might support blind children building up their social skills. Not least, we are researching how to customize such AI systems through machine teaching techniques so that they are right for the user, regardless of disability label.

Theo, left, a 12-year-old boy who is blind, interacts with Microsoft senior research software development engineers Tim Regan, middle, and Martin Grayson, right, during user testing of the Project Tokyo system. Photo by Jonathan Banks.
Peter Bosher, left background, an audio engineer who is blind, checks out the latest iteration of the Project Tokyo system at Microsoft’s research lab in Cambridge, UK. Bosher wears a modified Microsoft HoloLens that streams images of its visual field to computer hardware for processing. A dashboard on the laptop screen shows the visual field. Microsoft researchers Cecily Morrison, right background, and left on the screen, and Martin Grayson, right on the screen, are visible to the HoloLens. Photo by Jonathan Banks
Peter Bosher, middle, an audio engineer who is blind who worked with the Project Tokyo team early in the design process, checks out the latest iteration of the system at Microsoft’s research lab in Cambridge, UK, with researchers Martin Grayson, left, and Cecily Morrison, right. Photo by Jonathan Banks.