{"id":1084509,"date":"2024-10-24T10:26:26","date_gmt":"2024-10-24T17:26:26","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-story&p=1084509"},"modified":"2024-10-24T10:35:14","modified_gmt":"2024-10-24T17:35:14","slug":"find-my-things","status":"publish","type":"msr-story","link":"https:\/\/www.microsoft.com\/en-us\/research\/story\/find-my-things\/","title":{"rendered":"Find My Things: New teachable AI tool helps blind and low-vision people locate lost personal items"},"content":{"rendered":"\n
<\/span>
<\/div>
\n
\n
<\/div>\n\n\n\n

Find My Things: New teachable AI tool helps blind and low-vision people locate lost personal items<\/h1>\n\n\n\n
<\/div>\n<\/div>\n<\/div><\/div>\n\n\n\n
\n
\n\t\n\t
\n\t\t
\n\t\t\t
<\/div>\t\t<\/div>\n\t<\/div>\n\n\t<\/div>\n\n\n\n
\n
<\/div>\n\n\n\n
\n
<\/div>\n\n\n\n

Assistive technologies have helped millions of people who are blind or have low vision with a variety of daily tasks, from converting text to speech and summarizing complex documents to using public transportation and navigating unfamiliar environments. <\/h3>\n\n\n\n

On the other hand, developing technology that can help people with vision disabilities find frequently used and easily misplaced personal items in almost any physical space has proved much more difficult. Until now.<\/p>\n\n\n\n

<\/div>\n\n\n\n
\n
\n

Find My Things, developed by members of the Microsoft Research Teachable AI Experiences (Tai X) (opens in new tab)<\/span><\/a> team with the help of a citizen design team they assembled for the project, is a teachable AI tool designed to solve that problem. Now available as a feature of the World channel in Seeing AI (opens in new tab)<\/span><\/a>\u2014Microsoft\u2019s phone app for the blind and low-vision community that uses an intelligent camera and the power of AI to describe people, text, currency, color, and objects\u2014Find My Things makes it easy for people to use their phones to recognize and locate the personal items they use every day. Find My Things was recently honored in both the accessible design (opens in new tab)<\/span><\/a> and artificial intelligence (opens in new tab)<\/span><\/a> categories of Fast Company\u2019s 2024 Innovation by Design Awards. <\/p>\n<\/div>\n\n\n\n

\n
\"Rico (opens in new tab)<\/span><\/a><\/figure>\n\n\n\n
<\/div>\n\n\n
\n

Timeline: Assistive technology at Microsoft Research<\/a><\/h3>\n<\/div><\/div>\n<\/div>\n\n\n\n

Unlike other object recognizers that are pre-programmed to recognize a collection of generic objects, Find My Things gives people the power to personalize their experience by teaching the tool to recognize the items they actually use. This may include small items like house keys or earbuds, medium-sized items like backpacks or travel mugs, and even items that sometimes change shape, such as charger cords or a folding guide cane.<\/p>\n\n\n\n

\n

\u201cI have a few small personal items that I always carry with me, and most of them are very important to me. Losing my bus pass means I can\u2019t go anywhere. Losing my house key means I can\u2019t get home. Because these items are small and I use them daily, it\u2019s not unusual for me to lose track of them. With Find My Things, I can locate my personal items if they happen to be lost.\u201d<\/p>\n\u2013<\/em> Karolina Pak\u0117nait\u0117, Citizen Designer<\/cite><\/blockquote>\n\n\n\n

\"Karolina
Karolina Pak\u0117nait\u0117. Photo by Jonathan Banks for Microsoft.<\/figcaption><\/figure>\n\n\n\n

The Find My Things experience in the Seeing AI mobile app consists of two parts: teaching <\/em>and finding<\/em>. To teach Find My Things to recognize a specific object and add it to their list of findable things, a person uses their phone to record four short videos of the item they have selected\u2014from various angles and against different backgrounds. While shooting the videos, the person receives feedback from the tool to help them stay focused on the target object. The videos then serve as training data for the tool\u2019s few-shot object-recognition model and can be personalized on that individual\u2019s device in just a couple of seconds.<\/p>\n\n\n\n

Using the tool to find an item is even easier. A person simply selects an object from the list of findable things they\u2019ve created and then uses their phone to scan the environment\u2014whether it\u2019s their living room, a coffee shop, or a neighborhood park\u2014until the tool locates the object. The tool then uses audio, visual, and vibration cues to guide the person to within arm\u2019s reach of the object.<\/p>\n\n\n\n

\u201cFind My Things is one of the first fully working systems available to members of the public that allows people to teach an AI system to meet their own needs,\u201d says Cecily Morrison, a senior principal research manager at Microsoft Research, who led the Find My Things project and was honored in 2020 as a Member of the Order of the British Empire for her services to inclusive design.<\/p>\n\n\n\n

\n\n\n\n
\n\t\n\t
\n\t\t
\n\t\t\t
\n

In this YouTube video, Theo Holroyd demonstrates how blind and low-vision people can teach the Find My Things AI-powered tool to recognize and locate their lost or misplaced personal items.<\/h3>\n<\/div>