{"id":1087971,"date":"2024-09-30T06:02:05","date_gmt":"2024-09-30T13:02:05","guid":{"rendered":""},"modified":"2024-11-05T06:42:18","modified_gmt":"2024-11-05T14:42:18","slug":"abstracts-september-30-2024","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/abstracts-september-30-2024\/","title":{"rendered":"Abstracts: September 30, 2024"},"content":{"rendered":"\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1400\" height=\"788\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788.jpg\" alt=\"Outline illustration of Daniela Massiceti next to Martin Grayson\" class=\"wp-image-1091709\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788.jpg 1400w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-1280x720.jpg 1280w\" sizes=\"(max-width: 1400px) 100vw, 1400px\" \/><\/figure>\n\n\n<div class=\"wp-block-msr-podcast-container my-4\">\n\t<iframe loading=\"lazy\" src=\"https:\/\/player.blubrry.com\/?podcast_id=136647899&modern=1\" class=\"podcast-player\" frameborder=\"0\" height=\"164px\" width=\"100%\" scrolling=\"no\" title=\"Podcast Player\"><\/iframe>\n<\/div>\n\n\n\n<p>Members of the research community at Microsoft work continuously to advance their respective fields.\u202f <em>Abstracts<\/em>\u202fbrings its audience to the cutting edge with them through short, compelling conversations about new and noteworthy achievements.&nbsp;<\/p>\n\n\n\n<p>In this episode, Senior Researcher <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/dmassiceti\/\">Daniela Massiceti<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and Principal Research Software Development Engineer <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/mgrayson\/\">Martin Grayson<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> join host Amber Tingle to discuss the research project and AI-powered tool Find My Things. Find My Things is a personalizable object recognizer that people who are blind or have low vision can train to find personal items from just a few videos of those objects. It was recently recognized as a <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.fastcompany.com\/innovation-by-design\/list\" target=\"_blank\" rel=\"noreferrer noopener\">2024 Innovation by Design Awards<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> finalist in the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.fastcompany.com\/91128700\/accessible-design-innovation-by-design-2024\" target=\"_blank\" rel=\"noreferrer noopener\">accessibility design<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.fastcompany.com\/91129207\/artificial-intelligence-innovation-by-design-2024\" target=\"_blank\" rel=\"noreferrer noopener\">AI categories<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> by the US-based business media brand Fast Company and, earlier this year, became <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.seeingai.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">available as a feature in the Seeing AI app<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.&nbsp;<\/p>\n\n\n\n<p><em>The Find My Things story is an example of research at Microsoft enhancing Microsoft products and services. To try the Find My Things tool, <\/em><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.seeingai.com\/\" target=\"_blank\" rel=\"noreferrer noopener\"><em>download the free, publicly available Seeing AI app<\/em><span class=\"sr-only\"> (opens in new tab)<\/span><\/a><em>.<\/em><\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a data-bi-type=\"button\" class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/www.seeingai.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Try Find My Things<\/a><\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-group msr-pattern-link-list is-layout-flow wp-block-group-is-layout-flow\">\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading h5\" id=\"learn-more-1\">Learn more:<\/h2>\n\n\n\n<ul class=\"wp-block-list list-unstyled\">\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/find-my-things-personalized-accessibility-through-teachable-ai-for-people-who-are-blind-or-low-vision\/\">Find My Things: Personalized Accessibility through Teachable AI for People who are Blind or Low Vision<\/a><br>Publication, May 2024<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/understanding-personalized-accessibility-through-teachable-ai-designing-and-evaluating-find-my-things-for-people-who-are-blind-or-low-vision\/\">Understanding Personalized Accessibility through Teachable AI: Designing and Evaluating Find My Things for People who are Blind or Low Vision<\/a><br>Publication, October 2023<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/taix\/\">Teachable AI Experiences (Tai X)<\/a><br>Project page<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/podcast\/collaborators-teachable-ai-with-cecily-morrison-and-karolina-pakenaite\/\" target=\"_blank\" rel=\"noreferrer noopener\">Collaborators: Teachable AI with Cecily Morrison and Karolina Pak\u0117nait\u0117<\/a><br>Microsoft Research Podcast, December 2023<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/orbit-dataset\/\" target=\"_blank\" rel=\"noreferrer noopener\">ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition<\/a><br>Publication, October 2021<\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/peoplelens\/\" target=\"_blank\" rel=\"noreferrer noopener\">PeopleLens<\/a><br>Publication, June 2021<\/li>\n<\/ul>\n<\/div>\n\n\n\n<div style=\"height:25px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<section class=\"wp-block-msr-subscribe-to-podcast subscribe-to-podcast\">\n\t<div class=\"subscribe-to-podcast__inner border-top border-bottom border-width-2\">\n\t\t<h2 class=\"h5 subscribe-to-podcast__heading\">\n\t\t\tSubscribe to the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/podcast\">Microsoft Research Podcast<\/a>:\t\t<\/h2>\n\t\t<ul class=\"subscribe-to-podcast__list list-unstyled\">\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/itunes.apple.com\/us\/podcast\/microsoft-research-a-podcast\/id1318021537?mt=2\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"black\" viewBox=\"0 0 32 32\">  <path d=\"M7.12 0c-3.937-0.011-7.131 3.183-7.12 7.12v17.76c-0.011 3.937 3.183 7.131 7.12 7.12h17.76c3.937 0.011 7.131-3.183 7.12-7.12v-17.76c0.011-3.937-3.183-7.131-7.12-7.12zM15.817 3.421c3.115 0 5.932 1.204 8.079 3.453 1.631 1.693 2.547 3.489 3.016 5.855 0.161 0.787 0.161 2.932 0.009 3.817-0.5 2.817-2.041 5.339-4.317 7.063-0.812 0.615-2.797 1.683-3.115 1.683-0.12 0-0.129-0.12-0.077-0.615 0.099-0.792 0.192-0.953 0.64-1.141 0.713-0.296 1.932-1.167 2.677-1.911 1.301-1.303 2.229-2.932 2.677-4.719 0.281-1.1 0.244-3.543-0.063-4.672-0.969-3.595-3.907-6.385-7.5-7.136-1.041-0.213-2.943-0.213-4 0-3.636 0.751-6.647 3.683-7.563 7.371-0.245 1.004-0.245 3.448 0 4.448 0.609 2.443 2.188 4.681 4.255 6.015 0.407 0.271 0.896 0.547 1.1 0.631 0.447 0.192 0.547 0.355 0.629 1.14 0.052 0.485 0.041 0.62-0.072 0.62-0.073 0-0.62-0.235-1.199-0.511l-0.052-0.041c-3.297-1.62-5.407-4.364-6.177-8.016-0.187-0.943-0.224-3.187-0.036-4.052 0.479-2.323 1.396-4.135 2.921-5.739 2.199-2.319 5.027-3.543 8.172-3.543zM16 7.172c0.541 0.005 1.068 0.052 1.473 0.14 3.715 0.828 6.344 4.543 5.833 8.229-0.203 1.489-0.713 2.709-1.619 3.844-0.448 0.573-1.537 1.532-1.729 1.532-0.032 0-0.063-0.365-0.063-0.803v-0.808l0.552-0.661c2.093-2.505 1.943-6.005-0.339-8.296-0.885-0.896-1.912-1.423-3.235-1.661-0.853-0.161-1.031-0.161-1.927-0.011-1.364 0.219-2.417 0.744-3.355 1.672-2.291 2.271-2.443 5.791-0.348 8.296l0.552 0.661v0.813c0 0.448-0.037 0.807-0.084 0.807-0.036 0-0.349-0.213-0.683-0.479l-0.047-0.016c-1.109-0.885-2.088-2.453-2.495-3.995-0.244-0.932-0.244-2.697 0.011-3.625 0.672-2.505 2.521-4.448 5.079-5.359 0.547-0.193 1.509-0.297 2.416-0.281zM15.823 11.156c0.417 0 0.828 0.084 1.131 0.24 0.645 0.339 1.183 0.989 1.385 1.677 0.62 2.104-1.609 3.948-3.631 3.005h-0.015c-0.953-0.443-1.464-1.276-1.475-2.36 0-0.979 0.541-1.828 1.484-2.328 0.297-0.156 0.709-0.235 1.125-0.235zM15.812 17.464c1.319-0.005 2.271 0.463 2.625 1.291 0.265 0.62 0.167 2.573-0.292 5.735-0.307 2.208-0.479 2.765-0.905 3.141-0.589 0.52-1.417 0.667-2.209 0.385h-0.004c-0.953-0.344-1.157-0.808-1.553-3.527-0.452-3.161-0.552-5.115-0.285-5.735 0.348-0.823 1.296-1.285 2.624-1.291z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Apple Podcasts<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/subscribebyemail.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M6.4 6a2.392 2.392 0 00-2.372 2.119L16 15.6l11.972-7.481A2.392 2.392 0 0025.6 6H6.4zM4 10.502V22.8a2.4 2.4 0 002.4 2.4h19.2a2.4 2.4 0 002.4-2.4V10.502l-11.365 7.102a1.2 1.2 0 01-1.27 0L4 10.502z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Email<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/subscribeonandroid.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M12.414 4.02c-.062.012-.126.023-.18.06a.489.489 0 00-.12.675L13.149 6.3c-1.6.847-2.792 2.255-3.18 3.944h13.257c-.388-1.69-1.58-3.097-3.179-3.944l1.035-1.545a.489.489 0 00-.12-.675.492.492 0 00-.675.135l-1.14 1.68a7.423 7.423 0 00-2.55-.45c-.899 0-1.758.161-2.549.45l-1.14-1.68a.482.482 0 00-.494-.195zm1.545 3.824a.72.72 0 110 1.44.72.72 0 010-1.44zm5.278 0a.719.719 0 110 1.44.719.719 0 110-1.44zM8.44 11.204A1.44 1.44 0 007 12.644v6.718c0 .795.645 1.44 1.44 1.44.168 0 .33-.036.48-.09v-9.418a1.406 1.406 0 00-.48-.09zm1.44 0V21.76c0 .793.646 1.44 1.44 1.44h10.557c.793 0 1.44-.647 1.44-1.44V11.204H9.878zm14.876 0c-.169 0-.33.035-.48.09v9.418c.15.052.311.09.48.09a1.44 1.44 0 001.44-1.44v-6.719a1.44 1.44 0 00-1.44-1.44zM11.8 24.16v1.92a1.92 1.92 0 003.84 0v-1.92h-3.84zm5.759 0v1.92a1.92 1.92 0 003.84 0v-1.92h-3.84z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Android<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/open.spotify.com\/show\/4ndjUXyL0hH1FXHgwIiTWU\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M16 4C9.383 4 4 9.383 4 16s5.383 12 12 12 12-5.383 12-12S22.617 4 16 4zm5.08 17.394a.781.781 0 01-1.086.217c-1.29-.86-3.477-1.434-5.303-1.434-1.937.002-3.389.477-3.403.482a.782.782 0 11-.494-1.484c.068-.023 1.71-.56 3.897-.562 1.826 0 4.365.492 6.171 1.696.36.24.457.725.217 1.085zm1.56-3.202a.895.895 0 01-1.234.286c-2.338-1.457-4.742-1.766-6.812-1.747-2.338.02-4.207.466-4.239.476a.895.895 0 11-.488-1.723c.145-.041 2.01-.5 4.564-.521 2.329-.02 5.23.318 7.923 1.995.419.26.547.814.286 1.234zm1.556-3.745a1.043 1.043 0 01-1.428.371c-2.725-1.6-6.039-1.94-8.339-1.942h-.033c-2.781 0-4.923.489-4.944.494a1.044 1.044 0 01-.474-2.031c.096-.023 2.385-.55 5.418-.55h.036c2.558.004 6.264.393 9.393 2.23.497.292.663.931.371 1.428z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">Spotify<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\n\t\t\t\t\t\t\t<li class=\"subscribe-to-podcast__list-item\">\n\t\t\t\t\t<a class=\"subscribe-to-podcast__link\" href=\"https:\/\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noreferrer noopener\">\n\t\t\t\t\t\t<svg class=\"subscribe-to-podcast__svg\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" fill=\"none\" viewBox=\"0 0 32 32\"><path fill=\"currentColor\" d=\"M6.667 4a2.676 2.676 0 00-2.612 2.13v.003c-.036.172-.055.35-.055.534v18.666c0 .183.019.362.055.534v.003a2.676 2.676 0 002.076 2.075h.002c.172.036.35.055.534.055h18.666A2.676 2.676 0 0028 25.333V6.667a2.676 2.676 0 00-2.13-2.612h-.003A2.623 2.623 0 0025.333 4H6.667zM8 8h1.333C17.42 8 24 14.58 24 22.667V24h-2.667v-1.333c0-6.618-5.382-12-12-12H8V8zm0 5.333h1.333c5.146 0 9.334 4.188 9.334 9.334V24H16v-1.333A6.674 6.674 0 009.333 16H8v-2.667zM10 20a2 2 0 11-.001 4.001A2 2 0 0110 20z\"\/><\/svg>\n\t\t\t\t\t\t<span class=\"subscribe-to-podcast__link-text\">RSS Feed<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t<\/ul>\n\t<\/div>\n<\/section>\n\n\n<div class=\"wp-block-msr-show-more\">\n\t<div class=\"bg-neutral-100 p-5\">\n\t\t<div class=\"show-more-show-less\" data-mount=\"show-more-show-less\">\n\t\t\t<div>\n\t\t\t\t<span>\n\t\t\t\t\t\n\n<h2 class=\"wp-block-heading\" id=\"transcript\">Transcript<\/h2>\n\n\n\n<p>[MUSIC]&nbsp;<\/p>\n\n\n\n<p><strong>AMBER TINGLE: <\/strong>Welcome to <em>Abstracts<\/em>, a Microsoft Research Podcast that puts the spotlight on world-class research <em>in brief<\/em>. I&#8217;m Amber Tingle. In this series, members of the research community at Microsoft offer a quick snapshot\u2014or a <em>podcast abstract<\/em>\u2014of their new and noteworthy papers and achievements.&nbsp;<\/p>\n\n\n\n<p>[MUSIC FADES]&nbsp;<\/p>\n\n\n\n<p>Our guests today are Daniela Massiceti and Martin Grayson. Daniela is a senior researcher at Microsoft, and Martin is a software development engineer with the company. They are members of a team creating technology that can be personalized to meet individual needs. Their research project called Find My Things enables people who are blind or have low vision to train an AI system to find their personal items based on a few examples of the objects. Find My Things has now shipped as a new feature within Seeing AI, which is a free app that narrates a person&#8217;s surroundings, including nearby people, text, and objects. The team was also recently recognized by the US-based business media brand Fast Company as an Innovation by Design Awards finalist in both the accessible design and artificial intelligence categories. Daniela and Martin, congratulations and thank you so much for joining us today for <em>Abstracts<\/em>.&nbsp;<\/p>\n\n\n\n\t\t\t\t<\/span>\n\t\t\t\t<span id=\"show-more-show-less-toggle-1\" class=\"show-more-show-less-toggleable-content\">\n\t\t\t\t\t\n\n\n\n<p><strong>MARTIN GRAYSON: <\/strong>Pleasure, thank you.&nbsp;<\/p>\n\n\n\n<p><strong>DANIELA MASSICETI:<\/strong> Thanks very much, Amber. Nice to be here.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE:<\/strong> So, Daniela, let&#8217;s start with a Find My Things overview. What is it, how does it work, and who&#8217;s it for?&nbsp;<\/p>\n\n\n\n<p><strong>MASSICETI:<\/strong> I think the best way I can describe Find My Things is a personalizable object recognizer. So when we think about object recognizers in the past, they&#8217;ve, kind of, been what I would call <em>generic<\/em> object recognizers. So they can only really recognize generic things like maybe chairs, desks, tables. But for the blind and low-vision community, who are really key users of object recognition apps and technologies, for them, that&#8217;s not quite enough. They need to be able to recognize all of their personal objects and items. So things like their sunglasses, their partner&#8217;s sunglasses, um, perhaps their house keys. So a range of these really specific personal objects that generic object recognizers cannot recognize and help them find. And so Find My Things aims to tackle that by being a <em>personalizable<\/em> object recognizer. A user can essentially teach this object recognizer what their personal items look like, and then the personalized feature can then help them locate those objects at any point in the future. The experience is divided into two phases: a teaching phase and a finding phase. So in a teaching phase, a user would capture four short videos of each of their personal objects, and those videos are then ingested into the app<a id=\"_ftnref1\" href=\"#_ftn1\">[1]<\/a>, and the machine learning model that sits underneath that app learns what those objects actually look like. And then in the second, finding phase a user at any point in the future can, kind of, say, hey, I want to find my partner&#8217;s sunglasses or my sunglasses. And that will initiate this 3D localization experience, which will help guide them with sound and touch cues to that specific object, wherever it is in the room around them.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE: <\/strong>I\u2019ve heard Find My Things described as a <em>teachable<\/em> AI system. Daniela alluded to this, but, Martin, break it down a bit more for us. What do you and your collaborators mean when you use the term <em>teachable<\/em> AI?&nbsp;<\/p>\n\n\n\n<p><strong>GRAYSON:<\/strong> Something you can say about every person is that we&#8217;re all unique. Unique in the things that we like, whether that&#8217;s music, movies, food; the things we do, whether it&#8217;s at home, at work, or in your hobbies; and of course, the things that we have and own and keep with us. The same applies to accessibility. Everyone has their own unique sets of skills and tools that help them get things done, and we have them set up in just the way that matches us. The other day, I, like, came into the office and I sat in my chair, and I realized immediately that it wasn&#8217;t right. And of course, somebody had borrowed my desk the previous day and changed the height of my chair, but it was no problem because I could just re-personalize the chair back to my liking. When it comes to tools for accessibility, we think that people should have the same ability to personalize those tools to work the very best way for them. Typically, these have been settings like text size, speech, and color display, but AI has become a more and more important component in those tools. And one way we&#8217;re really excited about how to enable that is through teachable AI. So I think for us, teachable AI means that we can take some already really smart AI technology that might have some great general skills, but with a tiny amount of time from a person, that AI can be taught what matters to them and what works for them and become an even better AI to help them get things done.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE: <\/strong>Describe the origins of this work for our listeners, Daniela. What influenced or inspired the Find My Things pursuit? And how does your work build on <em>or differ<\/em> from previous work in the accessible technology space?&nbsp;<\/p>\n\n\n\n<p><strong>MASSICETI:<\/strong> Yeah, great question. And this is going to require me to cast my mind back to around four years ago. Our team at Microsoft Research was developing a system called the PeopleLens. So this was a head-mounted camera device that could be worn by people who are blind or low vision, specifically children who are blind or low vision. And it would help them identify or it would describe to them all the people that are around them in their social scenario\u2014where those people were, were those people looking at them. And I think the team realized very quickly that, as Martin was saying there, each person has a really unique need or a unique view of what they actually want described to them about the social environment around them. And so that got us thinking, well, actually, being able to personalize this system is <em>really<\/em> important. But in complex social environments, personalization is a really hard problem. And so that prompted the team to think, OK, well, we want to study this idea of personalization; let&#8217;s try and find almost the simplest possible example of an AI technology with which we could actually deeply explore this space of personalization. And that led us to object recognizers. Object recognizers are, as I mentioned, a very commonly used technology across the blind and low-vision community, and we know that there is a need for personalization there. And so that really prompted or started this journey along personalizable, or teachable, object recognizers, which we then have been working on for the last three or four years to eventually get us to a point now where we&#8217;re seeing this feature available in Seeing AI.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE:<\/strong> Your team identified few-shot learning and the availability of new datasets as keys to this work. Martin, how have those particular advances helped to make Find My Things possible? And are there other approaches you&#8217;ve incorporated to make sure that it&#8217;s both practical and valuable for people who are blind or have low vision?&nbsp;<\/p>\n\n\n\n<p><strong>GRAYSON:<\/strong> So AI loves data. In fact, data is essential to make AI work. But for AI to work for everyone, it needs to learn from data that somehow <em>represents<\/em> everyone. The earliest challenge for Find My Things was that people who are blind or low vision don&#8217;t often use their cameras to take lots of photos and videos. And this actually gives us two big data gaps. The first is that we don&#8217;t have lots of image data that is representative of their own lives, their environments, and their things. And the second is that if you&#8217;re someone who&#8217;s blind, you may hold your phone differently, or you may use your camera in different ways. And that&#8217;s, too, missing from the data, certainly in the established datasets that exist. So new datasets, like ORBIT, have collected thousands of images and videos by members of the blind and low-vision community, specifically of objects and environments that are really important to them. And this means that we&#8217;ve now addressed those two big data gaps. And the few-shot part is really important, too. Find My Things is not a general object recognizer. It&#8217;s a find <em>my<\/em> things. We want Find My Things to be able to recognize anything you throw at it\u2014whether it&#8217;s your fluffy keyring, your colorful tote bag, or your favorite gadget or toy. However, traditional object detectors, they often need hundreds or thousands of images to learn how to recognize something accurately. Few-shot learning is a super-smart approach that means you only need to trouble our users for a couple of short five-second videos, and then our app will take it from there. Find My Things can use that tiny amount of data and still be able to spot your object from across the room.&nbsp;<\/p>\n\n\n\n<p>Maybe one more thing we did, and this also became so important, was to build and try prototype experiences as soon as we possibly could. And we would try so many models and designs out and then iterate. The team has definitely seen so many videos of me trying to find things around my house. But it&#8217;s actually one of the things we&#8217;re most proud of in the project, is this, kind of, graveyard of interactive prototypes that have all led us to the final experience.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE:<\/strong> Daniela, what have you learned from the Find My Things journey that may help the broader research community create more inclusive and more human-centric AI experiences?&nbsp;<\/p>\n\n\n\n<p><strong>MASSICETI: <\/strong>The first one I would say is the importance of doing participatory research. And what that means is really working with the communities that you are developing technologies for. And the second is really learning how to balance this tension between developing something in a research environment and actually deploying that technology in a real-world environment. To jump to the first learning around participatory research, Martin mentioned the ORBIT dataset.<strong> <\/strong>The ORBIT dataset was collected in partnership with users who are blind or low vision across both the UK and Canada over the years 2020 to 2021. And it was really important for us to actually engage directly with users who are blind as we were collecting that dataset from them to really understand what they wanted from a personalizable object recognition technology, how they would use their cameras, how they would hold their phones, what kinds of objects they would use this technology to find. And all of that was really, really critical in helping us shape what that dataset ended up being. That dataset became such a pivotal part of<strong> <\/strong>the ultimate Find My Things experience. To the second point around this tension between building something in research and deploying something in the real world, I think often as a researcher, we don&#8217;t really have to engage with real-world constraints. But of course, when you build a machine learning model or a machine learning system and you want to deploy it in the real world, suddenly those constraints really hit you in the face. And that was exactly the case with Find My Things. I remember quite distinctly in the model development process, we had a number of different models. They were, sort of, ranging up in size in terms of how much memory they would take on a phone to run. And of course, the larger the model was, the more accurate it was. But when we deployed these models of varying sizes onto a phone, we saw that they each had vastly different reactions to being on this phone. And I think if I recall from memory, some of our largest models ended up basically draining the phone&#8217;s battery in a couple of minutes, which would mean that the experience would be totally unusable to the user. And so one of the key things we had to do there is really find this sweet spot, or this balance, between what is good enough performance that does not end up, kind of, degrading the actual experience of running this model on a phone.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE:<\/strong> You mentioned participatory research, and your team&#8217;s version feels a little different from what we typically encounter. Talk a little bit more about the citizens who helped you build out this app.&nbsp;<\/p>\n\n\n\n<p><strong>MASSICETI:<\/strong> So these were a group of perhaps eight to 10 users who are blind or low vision who we hosted at Microsoft Research a number of times over the course of the development of the Find My Things experience. And they were \u2026 perhaps the best way I can describe them is they were co-designers; they were really helping us design\u2014<em>co-design<\/em>\u2014what the Find My Things experience ultimately turned out to be. We weren&#8217;t coming to them as simply testers of our system. We, kind of, went to them with a blank slate and asked them, well, we have these ideas of what we want to build; what do you think? And from there, we, kind of, iterated upwards and ultimately crafted, co-crafted, the ultimate design of the Find My Things experience, both the teaching part and the finding part.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE:<\/strong> One of the members of that citizen design team, Karolina Pak\u0117nait\u0117, visited the Microsoft Research Podcast back in December with your colleague Cecily Morrison. Martin, talk a bit more about how influential citizen designers like Karolina are to this effort.&nbsp;<\/p>\n\n\n\n<p><strong>GRAYSON:<\/strong> There were so many key ideas and innovations that came from the workshops with Karolina and the rest of the citizen design team. Maybe I can share a couple that have had the biggest impact on the app and the experience. So the first was the process of teaching an object. Our testing of AI models showed that collecting videos of objects from different sides and on different backgrounds was critical. So we developed this thing called the <em>drawback technique<\/em>, where we leaned on the phone&#8217;s augmented reality capabilities to make it possible. We&#8217;d ask the user to start with the phone right next to their object and then slowly draw it away. This meant that we could track all of the different distances the images were,<strong> <\/strong>and the user could really comfortably create a video without leaving their seat. And what&#8217;s more, you can do this so easily without even needing to look at the camera. It&#8217;s really natural. The second big design innovation came later on when you were actually looking for the thing. We called it the <em>last yard<\/em>. So many of the lost-item scenarios that we learned about from the citizen designers \u2026 they shared with us that they had dropped something in a public space. Their wallet fell out of their pocket as they took their phone out, or they knocked their earbud off the table onto the floor of the train on their way to work. And in both of those moments, the last thing anyone wants to be doing is feeling around on the floor, especially on public transport. So we tested these early versions of Find My Things with the design team, and they would get close to their object, overstep it, and then reach down. And they&#8217;d still be feeling around the floor before they found their object, which mostly ended up back behind them. So our last yard design completely changed this. As the user got close to their object, within the last yard, we change the sounds, and the app actually tells them to move down. The phone then responds to the distance to the object exactly like a metal detector. And this meant that when they reached down just at the right moment, they found their object on the floor and it was much easier. No more overstepping. We spent lots of time exploring how the experience and the phone capabilities like AR and AI could work best together, and our citizen design team<strong> <\/strong>gave us all of the key insights that led to us coming to these approaches.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE:<\/strong> So what&#8217;s next for Find My Things? I&#8217;d like you to share a bit about the opportunities or even the obstacles that exist for more widespread adoption of the teachable AI approach.&nbsp;<\/p>\n\n\n\n<p><strong>GRAYSON:<\/strong> So Find My Things was such a great project to work on. It sat right in the center of the triangle of AI innovation, designing with your community, and of course product impact. We&#8217;re taking so much of what we&#8217;ve learned during this project and building it into our research going forwards\u2014how we build and evaluate AI, how to engage with the communities that we want to build for, and of course the value of building lots and lots of prototypes. Teachable AI, I think, is going to be a key approach in addressing the challenge for AI working equally well for everyone. The challenge is how do we ensure that we build these new fantastic models on data that gives representation to all that\u2019ll use it. And so often, the people that might benefit the most from innovations in AI might have the smallest representation in data. And our work with people in the blind and low-vision community have really brought that into focus for us. AI can and will be transformational for them, so long as we can make it work just as well for everyone. And then that creates the opportunity: ensuring that these systems and technologies that we design can learn from and build in all of the diverse and wonderful uniqueness of being a human.&nbsp;<\/p>\n\n\n\n<p><strong>MASSICETI:<\/strong> I think one of the things I&#8217;m most excited about is unlocking this power of personalization. Hopefully, we&#8217;ve convinced you how impactful having personalized AI technologies would be for not only the blind and low-vision community, but for you and I. And so one of the things I&#8217;m most excited about is seeing how we can transplant some of these learnings and ideas that we&#8217;ve had in building Find My Things into now the generative AI era. And so, yeah, I think I&#8217;m really excited to, kind of, bring together these ideas of teachable AI with these new generative models to help really bring to life more useful AI technologies that service not just a small few but all the people across the user distribution.&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE:<\/strong> Daniela and Martin, thank you so much for joining <em>Abstracts<\/em> today.&nbsp;<\/p>\n\n\n\n<p><strong>MASSICETI:<\/strong> Thank you, Amber.&nbsp;<\/p>\n\n\n\n<p><strong>GRAYSON:<\/strong> Thank you for having us.&nbsp;<\/p>\n\n\n\n<p>[MUSIC]&nbsp;<\/p>\n\n\n\n<p><strong>TINGLE:<\/strong> And thanks to our listeners, too. If you&#8217;d like to learn more about Find My Things and teachable AI, visit <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/taix\/\">aka.ms\/TeachableAI<\/a>. Thank you for tuning in. I&#8217;m Amber Tingle. Join us next time for more <em>Abstracts<\/em>.&nbsp;<\/p>\n\n\n\n<p>&nbsp;[MUSIC FADES]&nbsp;<\/p>\n\n\t\t\t\t<\/span>\n\t\t\t<\/div>\n\t\t\t<button\n\t\t\t\tclass=\"action-trigger glyph-prepend mt-2 mb-0 show-more-show-less-toggle\"\n\t\t\t\taria-expanded=\"false\"\n\t\t\t\tdata-show-less-text=\"Show less\"\n\t\t\t\ttype=\"button\"\n\t\t\t\taria-controls=\"show-more-show-less-toggle-1\"\n\t\t\t\taria-label=\"Show more content\"\n\t\t\t\tdata-alternate-aria-label=\"Show less content\">\n\t\t\t\tShow more\t\t\t<\/button>\n\t\t<\/div>\n\t<\/div>\n<\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><a id=\"_ftn1\" href=\"#_ftnref1\">[1]<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> The AI-powered tool Find My Things is not a standalone app. It is available as a feature in the Seeing AI app. <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.seeingai.com\/\" target=\"_blank\" rel=\"noreferrer noopener\">Download the Seeing AI app<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> to try Find My Things.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The personalizable object recognizer Find My Things was recently recognized for accessible design. Researcher Daniela Massiceti and software development engineer Martin Grayson talk about the research project\u2019s origins and the tech advances making it possible.<\/p>\n","protected":false},"author":43518,"featured_media":1091709,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"https:\/\/player.blubrry.com\/id\/136647899","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[240054],"tags":[],"research-area":[13556,13562,13554],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[269148,269142,243990],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[268128],"class_list":["post-1087971","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-msr-podcast","msr-research-area-artificial-intelligence","msr-research-area-computer-vision","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-include-in-river","msr-post-option-podcast-featured","msr-podcast-series-abstracts"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"https:\/\/player.blubrry.com\/id\/136647899","podcast_episode":"","msr_research_lab":[199561,199565],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[283244],"related-projects":[830104],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Amber Tingle","user_id":42681,"display_name":"Amber Tingle","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/ambertingle\/\" aria-label=\"Visit the profile page for Amber Tingle\">Amber Tingle<\/a>","is_active":false,"last_first":"Tingle, Amber","people_section":0,"alias":"ambertingle"},{"type":"user_nicename","value":"Daniela Massiceti","user_id":40408,"display_name":"Daniela Massiceti","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/dmassiceti\/\" aria-label=\"Visit the profile page for Daniela Massiceti\">Daniela Massiceti<\/a>","is_active":false,"last_first":"Massiceti, Daniela","people_section":0,"alias":"dmassiceti"},{"type":"user_nicename","value":"Martin Grayson","user_id":32893,"display_name":"Martin Grayson","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/mgrayson\/\" aria-label=\"Visit the profile page for Martin Grayson\">Martin Grayson<\/a>","is_active":false,"last_first":"Grayson, Martin","people_section":0,"alias":"mgrayson"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-960x540.jpg\" class=\"img-object-cover\" alt=\"Outline illustration of Daniela Massiceti next to Martin Grayson\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/09\/Daniela-and-Martin_Abstracts_Hero_Feature_No_Text_1400x788.jpg 1400w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/ambertingle\/\" title=\"Go to researcher profile for Amber Tingle\" aria-label=\"Go to researcher profile for Amber Tingle\" data-bi-type=\"byline author\" data-bi-cN=\"Amber Tingle\">Amber Tingle<\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/dmassiceti\/\" title=\"Go to researcher profile for Daniela Massiceti\" aria-label=\"Go to researcher profile for Daniela Massiceti\" data-bi-type=\"byline author\" data-bi-cN=\"Daniela Massiceti\">Daniela Massiceti<\/a>, and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/mgrayson\/\" title=\"Go to researcher profile for Martin Grayson\" aria-label=\"Go to researcher profile for Martin Grayson\" data-bi-type=\"byline author\" data-bi-cN=\"Martin Grayson\">Martin Grayson<\/a>","formattedDate":"September 30, 2024","formattedExcerpt":"The personalizable object recognizer Find My Things was recently recognized for accessible design. Researcher Daniela Massiceti and software development engineer Martin Grayson talk about the research project\u2019s origins and the tech advances making it possible.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1087971"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/43518"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1087971"}],"version-history":[{"count":22,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1087971\/revisions"}],"predecessor-version":[{"id":1091718,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1087971\/revisions\/1091718"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1091709"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1087971"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1087971"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1087971"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1087971"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1087971"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1087971"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1087971"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1087971"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1087971"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1087971"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1087971"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}