{"id":747019,"date":"2021-05-21T11:32:44","date_gmt":"2021-05-21T18:32:44","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=747019"},"modified":"2022-11-01T09:01:00","modified_gmt":"2022-11-01T16:01:00","slug":"chi-2021-redefining-accessibility-to-build-more-inclusive-technologies","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/chi-2021-redefining-accessibility-to-build-more-inclusive-technologies\/","title":{"rendered":"CHI 2021: Redefining accessibility to build more inclusive technologies"},"content":{"rendered":"\n<p><\/p>\n\n\n<aside id=accordion-ca3b663a-4eea-457d-8a42-41fa7b01fd50 class=\"msr-table-of-contents-block accordion mb-5 pb-0\" data-bi-aN=\"table-of-contents\">\n\t<button class=\"btn btn-collapse bg-gray-100 mb-0 display-flex justify-content-between\" type=\"button\" data-mount=\"collapse\" data-target=\"#accordion-collapse-ca3b663a-4eea-457d-8a42-41fa7b01fd50\" aria-expanded=\"true\" aria-controls=\"accordion-collapse-ca3b663a-4eea-457d-8a42-41fa7b01fd50\">\n\t\t<span class=\"msr-table-of-contents-block__label subtitle\">In this article<\/span>\n\t\t<span class=\"msr-table-of-contents-block__current mr-4 text-gray-600 font-weight-normal\" aria-hidden=\"true\"><\/span>\n\t<\/button>\n\t<div id=\"accordion-collapse-ca3b663a-4eea-457d-8a42-41fa7b01fd50\" class=\"msr-table-of-contents-block__collapse-wrapper collapse show\" data-parent=\"#accordion-ca3b663a-4eea-457d-8a42-41fa7b01fd50\">\n\t\t<div class=\"accordion-body bg-gray-100 border-top pt-4\">\n\t\t\t<ol class=\"msr-table-of-contents-block__list\">\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"\" class=\"msr-table-of-contents-block__list-item-link\">Using games to collect and label richer data for the creation of signed language systems<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"\" class=\"msr-table-of-contents-block__list-item-link\">The PeopleLens: Using AI to help children who are blind build their social skills<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"\" class=\"msr-table-of-contents-block__list-item-link\">Leveraging the knowledge of experienced users to build more accessible smartphones<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"\" class=\"msr-table-of-contents-block__list-item-link\">Understanding the educational value of digital games in schools for the blind\u2014a teachers\u2019 perspective<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t\t<li class=\"msr-table-of-contents-block__list-item\">\n\t\t\t\t\t\t<a href=\"\" class=\"msr-table-of-contents-block__list-item-link\">More to explore<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<\/ul>\n\t\t<\/div>\n\t<\/div>\n\t<span class=\"msr-table-of-contents-block__progress-bar\"><\/span>\n<\/aside>\n\n\n\n<p>Accessibility and inclusion represent a growing space in the technology landscape, and how research and development are being used to empower people across abilities is expanding in exciting ways.<\/p>\n\n\n\n<p>Instead of treating disabilities as conditions in need of solutions\u2014as has been the case over the years with the medical-based approach to accessibility\u2014research is moving toward a social approach, examining and addressing the societal expectations imposed on people with disabilities and society\u2019s lack of inclusion. The world comprises people of various abilities but accommodates the abilities of the majority. A world in which everyone were blind would be designed very differently from the world we know today.<\/p>\n\n\n\n<p>This leaves a gap between what people with disabilities are capable of and what the environment set up for the majority enables them to do. Technology can play a key role in closing this gap. Starting from one of these gaps and working alongside impacted communities, researchers and technologists can begin pushing beyond making something accessible to providing tools for people to augment their own capabilities, says <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/cecilym\/\">Microsoft Principal Researcher Cecily Morrison<\/a>.<\/p>\n\n\n\n<p>In \u201cSocial Sensemaking with AI: Designing an Open-ended AI experience with a Blind Child\u201d\u2014one of several Microsoft papers in accessibility and inclusion featured at the <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/chi-2021\/\">2021 ACM CHI Virtual Conference on Human Factors in Computing Systems (CHI 2021)<\/a>\u2014Morrison and her coauthors demonstrate how technology can be designed in a way that enables people to define how they live their lives rather than dictating what they can accomplish. Their PeopleLens prototype, an intelligent headset, provides children who are blind with information such as the names of people in a physical space to help them experience social agency and have opportunities to develop their social interaction skills.<\/p>\n\n\n\n<p>\u201cIt\u2019s really shifting the power from a researcher to those users in terms of formulating their own future,\u201d says Morrison.<\/p>\n\n\n\n<p>Two CHI papers from <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/lab\/microsoft-research-india\/\">Microsoft Research India<\/a> explore a divide impacting how well technologists and researchers can meaningfully narrow the ability-environment gap\u2014the inequality among resources.<\/p>\n\n\n\n<p>\u201cEven though the majority of people with disabilities are living in the Global South, most of the work is targeted toward resource-rich Global North environments,\u201d says <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/swmanohmicrosoft-com\/\">Principal Researcher Manohar Swaminathan<\/a>.<\/p>\n\n\n\n<p>In the first paper, researchers gain insight into how experienced smartphone users in India who are blind navigate Android devices specifically, as Android devices are the more affordable option in the country. To help children in India who are blind receive computer science education, authors of the second paper worked directly with teachers of students who are blind, many of whom are blind themselves, to understand the opportunities and challenges of deploying technology in settings lacking technical support and computing infrastructure.<\/p>\n\n\n\n<p>Meanwhile, another CHI paper shows that understanding people\u2019s needs and expectations is only one of many steps; particularly when it comes to AI, data that will support the development of tech that\u2019s effective and helpful in the real world is essential. With the game prototype ASL Sea Battle, researchers aim to provide immediate benefits while also facilitating the collection of much-needed quality data to further help fill the void in tools and services available to signers. <\/p>\n\n\n\n<p>Explore these papers in more detail below, and check out the CHI workshops on accessibility and inclusion co-organized by Microsoft researchers:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/chi2021.acm.org\/for-authors\/interacting-discussing\/workshops-symposia\/accepted-workshops-symposiums#W18\">\u201cAdaptive Accessible AR\/VR Systems\u201d<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/li><li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/chi2021.acm.org\/for-authors\/interacting-discussing\/workshops-symposia\/accepted-workshops-symposiums#W51\">\u201cDisability Design and Innovation in Low-resource Settings: Addressing Inequality through HCI\u201d<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/li><li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/chi2021.acm.org\/for-authors\/interacting-discussing\/workshops-symposia\/accepted-workshops-symposiums#W45\">\u201cRethinking the Senses: A Workshop on Multisensory Embodied Experiences and Disability Interactions\u201d<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/li><\/ul>\n\n\n\n<h2 id=\"using-games-to-collect-and-label-richer-data-for-the-creation-of-signed-language-systems\">Using games to collect and label richer data for the creation of signed language systems<\/h2>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--right\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/asl-sea-battle-gamifying-sign-language-data-collection\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"ASL Sea Battle: Gamifying Sign Language Data Collection\" data-bi-aN=\"margin-callout\" data-bi-cN=\"ASL Sea Battle: Gamifying Sign Language Data Collection\">\n\t\t\t\tASL Sea Battle: Gamifying Sign Language Data Collection&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p><strong>In short: <\/strong>The main barrier to developing intelligent signed language systems is a lack of training data from diverse signers in diverse environments. ASL Sea Battle collects and labels real-world videos from signers while also providing a valuable resource for people who use signed language.<\/p>\n\n\n\n<p><strong>A deeper dive: <\/strong>In <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/asl-sea-battle-gamifying-sign-language-data-collection\/\">\u201cASL Sea Battle: Gamifying Sign Language Data Collection,\u201d<\/a> which received honorable mention at CHI, researchers introduce a smartphone game prototype based on the classic strategy game Battleship. Traditionally using a number-letter pair to guess the position of an opponent\u2019s ship on a grid, players of ASL Sea Battle identify their location selections instead by <em>sign<\/em> labels given to each square. To pick a location, players submit a video of themselves executing the sign of their chosen square; opponents show the success of the choice by selecting the corresponding square on their grid. The exchange facilitates not only the collection of real-world signing data but also the <em>labeling<\/em> of it. The research team, which includes fluent signers who are deaf and American Sign Language (ASL) linguists, engaged in an iterative design process, prototyping a variety of games with members of the signing community.<\/p>\n\n\n\n<p><strong>Forging new ground:<\/strong> The researchers believe their efforts are the first when it comes to leveraging games to collect signed language data.<\/p>\n\n\n\n<p><strong>The inspiration:<\/strong> The current state of the art in signed language systems is far behind that of spoken\/written language systems. For example, there are many resources serving English speakers, ranging from email clients to online articles, but few serving people who prefer to use a signed language. And because signed languages don\u2019t have a standard written form, today\u2019s text-based interfaces typically fail to support them. The inequity exists for a few main reasons: most technologists are spoken\/written language users, signed language support is not typically considered or prioritized, and there\u2019s a lack of sufficient training data.<\/p>\n\n\n\n<p><strong>Areas of impact: <\/strong>Gamifying signed language data collection has the potential to produce larger, more representative signed language datasets. Such datasets would enable the creation of machine learning models that work well for a more diverse set of end users in terms of personal demographics such as race and gender but also in terms of linguistic elements such as regional dialect. For example, no general sign language recognition or translation systems currently exist that are viable for real-world use. Larger, more representative datasets can help change that.<\/p>\n\n\n\n<p><strong>Key takeaway: <\/strong>If you\u2019re a researcher working in a data-scarce domain\u2014and particularly with marginalized communities\u2014explore building resources, like games and educational apps, that can serve the community while also collecting data with participant consent and using secure protocols and servers to help protect privacy. Such methods can provide immediate and direct benefit to a community. They also can remove barriers to participation, expanding inclusion in the data collection initiative itself, which is important to building ML applications that better serve the diverse group of people who will use them.<\/p>\n\n\n\n<p><strong>The research team:<\/strong> <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/dabragg\/\">Danielle Bragg<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/thies\/\">William Thies<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> of Microsoft Research; former Microsoft Research contractors John W. Gallagher and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/oakeyc.github.io\/\">Courtney Oka<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>; and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.bu.edu\/wheelock\/profile\/naomi-caselli\/\">Naomi Caselli<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/sites.bu.edu\/lexlab\/people\/\">Miriam Goldberg<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> of Boston University.<\/p>\n\n\n\n<h2 id=\"the-peoplelens-using-ai-to-help-children-who-are-blind-build-their-social-skills\">The PeopleLens: Using AI to help children who are blind build their social skills<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"406\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Theo_Accessibility_image-1024x406.jpg\" alt=\"A smiling young boy who is blind and wearing a head-mounted augmented reality device with a semicircular LED interface is seated at a kitchen table. A man, also smiling, is seated next to him, a laptop in front of him. In the background, another man is standing behind a kitchen island, working on a laptop. \" class=\"wp-image-747598\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Theo_Accessibility_image-1024x406.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Theo_Accessibility_image-300x119.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Theo_Accessibility_image-768x304.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Theo_Accessibility_image-1536x609.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Theo_Accessibility_image-16x6.jpg 16w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Theo_Accessibility_image.jpg 1885w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>To help children who are blind or have low vision cultivate their social skills and experience social agency, researchers developed the PeopleLens, a head-mounted augmented reality device that identifies the location of people in the wearer\u2019s social environment. Over a seven-month period, the team worked with Theo, a young boy who is blind, to explore the real-world use and potential of the prototype. This photograph was taken by Jonathan Banks.<\/figcaption><\/figure>\n\n\n\n<p><strong>In short:<\/strong> Microsoft has focused on AI that extends human capability rather than replaces it. Researchers worked with people who are blind or have low vision to imagine what that might mean practically for society\u2019s collective future with AI. The outcome is a research prototype called the PeopleLens, an assistive agent running on smart glasses that helps users who are blind or have low vision understand who\u2019s in their immediate social environment.&nbsp;<\/p>\n\n\n\n<p><strong>A deeper dive:<\/strong> In \u201c<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/social-sensemaking-with-ai-designing-an-open-ended-ai-experience-with-a-blind-child\/\">Social Sensemaking with AI: Designing an Open-ended AI experience with a Blind Child<\/a>,\u201d researchers report a seven-month exploration with a boy who is blind and the PeopleLens. The PeopleLens uses a head-mounted augmented reality device designed to enable children who are blind or have low vision to gain a dynamic, real-time understanding of their social environment through spatial audio. For example, whenever the user passes their gaze over another person, they hear the person\u2019s name or, if not identified, a spatialized sound to indicate a person\u2019s presence. This experience demonstrates a new class of AI experiences that goes beyond short tasks.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--left\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/social-sensemaking-with-ai-designing-an-open-ended-ai-experience-with-a-blind-child\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Social Sensemaking with AI: Designing an Open-ended AI experience with a Blind Child\" data-bi-aN=\"margin-callout\" data-bi-cN=\"Social Sensemaking with AI: Designing an Open-ended AI experience with a Blind Child\">\n\t\t\t\tSocial Sensemaking with AI: Designing an Open-ended AI experience with a Blind Child&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p><strong>The inspiration: <\/strong>AI has tremendous opportunity to close the gap between what people can do and what their environment allows them to do and to level the field for people who experience the world in different ways. Deep research powered by deep engagement with communities of varying abilities and their advocates has much to show about effectively closing the gap and building AI tools that empower.&nbsp;<\/p>\n\n\n\n<p><strong>Areas of impact:<\/strong> The researchers\u2019 recent focus has been on enabling children who are born blind to develop social agency. Many children who are blind struggle to interact socially, as it\u2019s a skill learned incidentally through vision. For example, if a child who is blind is unsure where their conversation partner is, they may have difficulty aiming their voice appropriately or they may position their body in a way that conveys disengagement. Providing social information in spatial audio provides children an opportunity to find their friends and build out those skills to direct their attention to others and others\u2019 attention to themselves\u2014the key to social agency.<\/p>\n\n\n\n<p><strong>Key takeaway: <\/strong>When closing the gap between people with different sensory experiences, technologists must consider not only the user but also the people interacting with the user. For example, it\u2019s equally important for a user to \u201csee\u201d a communication partner as it is for the partner to know they\u2019ve been &#8220;seen&#8221;. Researchers can engage more with how technology can create common ground and spread the responsibility of closing the gap among those interacting.<\/p>\n\n\n\n<p><strong>User-experience prototyping:<\/strong> Researchers illustrated a substantial difference between imagined and actual use of the PeopleLens. It\u2019s hard for a person to imagine their lives with a new set of information; testing a prototype and iterating can productively engage the imagination of what\u2019s possible. For example, the researchers\u2019 participant wanted all the information his peers who weren\u2019t blind had. He was overwhelmed and realized he instead needed information that helped him focus on the interactions he wanted to have. Differences in imagined and actual use also suggest that open-ended AI systems can empower users to extend their capabilities in ways technologists can\u2019t necessarily imagine.<\/p>\n\n\n\n<p><strong>The research team:<\/strong> <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.city.ac.uk\/about\/people\/academics\/alex-taylor\">Alex Taylor<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> of City, University of London; <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/cecilym\/\">Cecily Morrison<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/cutrell\/\">Edward Cutrell<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/mgrayson\/\">Martin Grayson<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/anthie\/\">Anja Thieme<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/calongde\/\">Camilla Longden<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/t-rimarq\/\">Rita Faia Marques<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/asellen\/\">Abigail Sellen<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> of Microsoft Research; former Microsoft researcher <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.tschiatschek.net\/\">Sebastian Tschiatschek<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>; and Geert Roumen, who was a Microsoft Research intern at the time of the work. <\/p>\n\n\n\n<h2 id=\"leveraging-the-knowledge-of-experienced-users-to-build-more-accessible-smartphones\">Leveraging the knowledge of experienced users to build more accessible smartphones<\/h2>\n\n\n\n<p><strong>In short:<\/strong> Informing the future of smartphone accessibility via the usage patterns of expert users who are blind.&nbsp;<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--right\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/smartphone-usage-by-expert-blind-users\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Smartphone Usage by Expert Blind Users\" data-bi-aN=\"margin-callout\" data-bi-cN=\"Smartphone Usage by Expert Blind Users\">\n\t\t\t\tSmartphone Usage by Expert Blind Users&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<p><strong>A deeper dive:<\/strong><em> <\/em>In <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/smartphone-usage-by-expert-blind-users\/\">\u201cSmartphone Usage by Expert Blind Users,\u201d<\/a> researchers recruited individuals who are blind and have extensive experience using Android smartphones and its screen reader app TalkBack to inform the design of more accessible devices. Based on logged phone usage over the course of a month and semi-structured interviews, researchers found that the group sought the same <em>interaction speed <\/em>achieved by phone users who aren\u2019t blind and tailored their behavior to that end. Through the examination of a variety of behaviors, from app usage to phone locking to battery charging, and feedback on security concerns and learning to use TalkBack, researchers discovered new engagement styles. They included combining TalkBack\u2019s \u201cexplore by touch\u201d with directional gestures to navigate the phone faster, a preference for voice and external keyboard for text entry, and the selection of different text-to-speech (TTS) engines depending on what participants were reading. Data collected with the consent of participants was anonymized before being uploaded to the cloud for analysis. Contact information, keys pressed, content of the TalkBack speech, and any details related to content consumed were removed.<\/p>\n\n\n\n<p><strong>The inspiration:<\/strong> The researchers were developing a tool to help novice users\u2014mostly young children\u2014learn TalkBack faster and in an intuitive manner. While developing the tool, they realized the details of how people who are blind interact with their smartphones to accomplish what they want were unclear. That lack of knowledge made it difficult to answer a series of important questions: What should the initial set of content they teach novice users be? Which TalkBack gestures are important? How do you use TTS effectively or best enter text? Before continuing with their work, they decided to understand expert users phone usage patterns first, as expert users tend to create efficient workarounds to gain maximum benefit of a system and overcome any limitations.<\/p>\n\n\n\n<p><strong>Areas of impact:<\/strong> The insights from this work can facilitate the design of smartphones that better meet the real-world needs of people who are blind. For example, from a software perspective, optimizing applications specifically for use with screen readers can begin to deliver a faster, more efficient phone experience. On the hardware front, standardization of specifications like touch screen sensitivity across devices could help ensure gesture styles learned and used on one phone transfer over to another. The work also supports the continued use of the tactile fingerprint scanner as an effective and trusted way for those who are blind to secure their devices. Additionally, users could benefit from an app to assist them in mastering TalkBack, perhaps a game-style tool in which new gestures and shortcuts could be introduced once a user reaches a certain skill level in using the screen reader.<\/p>\n\n\n\n<p><strong>Key takeaway:<\/strong> Expert users are an invaluable resource in understanding the real-world potential and limitations of a technology and ways to improve existing technologies. \u201cExpertise\u201d varies depending on the skill, but it\u2019s useful for recruited experts to have continued experience over a long duration. For TalkBack, the researchers considered five-plus years of experience as expert.<\/p>\n\n\n\n<p><strong>The research team: <\/strong><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/mohja\/\">Mohit Jain<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/swmanohmicrosoft-com\/\">Manohar Swaminathan<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> of Microsoft Research India and former Microsoft Research contractor <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.linkedin.com\/in\/nirmalendu-diwakar\/?originalSubdomain=in\">Nirmalendu Diwakar<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. <\/p>\n\n\n\n<h2 id=\"understanding-the-educational-value-of-digital-games-in-schools-for-the-blind-a-teachers-perspective\">Understanding the educational value of digital games in schools for the blind\u2014a teachers\u2019 perspective<\/h2>\n\n\n\n<p><strong>In short:<\/strong> Work aims to discover which attributes of digital games for children teachers find useful and why and explores the perceived challenges in integrating them in schools for children who are blind in India.<\/p>\n\n\n\n<p><strong>A deeper dive:<\/strong> In <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/teachers-perceptions-around-digital-games-for-children-in-low-resource-schools-for-the-blind\/\">\u201cTeachers\u2019 Perceptions around Digital Games for Children in Low-resource Schools for the Blind,\u201d<\/a> researchers partnered with Vision Empower, a nongovernmental organization in India that specializes in accessible resources for children who are blind, and together worked with seven schools for children who are blind in Karnataka, India. The team gauged the workload and skill sets of computer teachers via surveys, then invited the teachers to engage with existing accessible digital games during an informal and playful workshop designed to encourage them to think freely and creatively about the use of these types of games in their classrooms. Semi-structured interviews conducted afterward found teachers to be excited about these games\u2019 potential to increase student engagement and support learning.<\/p>\n\n\n\n<div class=\"annotations \" data-bi-aN=\"margin-callout\">\n\t<ul class=\"annotations__list card depth-16 bg-body p-4 annotations__list--right\">\n\t\t<li class=\"annotations__list-item\">\n\t\t\t\t\t\t<span class=\"annotations__type d-block text-uppercase font-weight-semibold text-neutral-300 small\">Publication<\/span>\n\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/teachers-perceptions-around-digital-games-for-children-in-low-resource-schools-for-the-blind\/\" target=\"_self\" class=\"annotations__link font-weight-semibold text-decoration-none\" data-bi-type=\"annotated-link\" aria-label=\"Teachers\u2019 Perceptions around Digital Games for Children in Low-resource Schools for the Blind\" data-bi-aN=\"margin-callout\" data-bi-cN=\"Teachers\u2019 Perceptions around Digital Games for Children in Low-resource Schools for the Blind\">\n\t\t\t\tTeachers\u2019 Perceptions around Digital Games for Children in Low-resource Schools for the Blind&nbsp;<span class=\"glyph-append glyph-append-chevron-right glyph-append-xsmall\"><\/span>\n\t\t\t<\/a>\n\t\t\t\t\t<\/li>\n\t<\/ul>\n<\/div>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"682\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Teachers-Workshop-view-1024x682.jpg\" alt=\"A woman and man both wearing headphones connected to a laptop on a small table in front of them are seated on a carpeted floor. The woman has her hands on the keyboard. Behind them, two women, one wearing headphones, are seated at a folding table, their attention on a laptop in front of them. A backpack and messenger bag rest on the floor beside them. \" class=\"wp-image-747616\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Teachers-Workshop-view-1024x682.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Teachers-Workshop-view-300x200.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Teachers-Workshop-view-768x512.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Teachers-Workshop-view-16x12.jpg 16w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/Teachers-Workshop-view.jpg 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>To understand how digital games can be used to enhance the classroom and learning experience for children who are blind or have low vision in India, researchers hosted a workshop for teachers at the Microsoft Research India lab. The informal nature of the event aimed to support teachers in exploring the games\u2014and their potential use in classrooms\u2014without limitations.<\/figcaption><\/figure>\n\n\n\n<p><strong>The inspiration: <\/strong>With STEM curricula in India\u2019s early education system limited to students who aren\u2019t blind or don\u2019t have low vision, the country\u2019s nearly 2 million children who are blind or have low vision are at high risk of being left out of the digital world and its myriad opportunities as they continue their education and choose career paths. This work takes the first steps to introducing computational thinking and digital skills to them at a young age.<\/p>\n\n\n\n<p><strong>Areas of impact:<\/strong> The teachers\u2019 positive perceptions of these digital games have the potential to encourage school administrators in India to emphasize the introduction of digital skills to children who are blind or have low vision in early grade levels and with a play-based approach. The work also aims to advance research in understanding and enhancing the learning environments of children with disabilities in low-resource environments.<\/p>\n\n\n\n<p><strong>Key takeaway:<\/strong> This research highlights the need to work with key stakeholders\u2014in this case, teachers\u2014to understand their perceptions about the introduction of new technologies.<\/p>\n\n\n\n<p><strong>Play is fundamental:<\/strong> This work is part of the application of a new methodology called <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/ludicdesign\/\">Ludic Design for Accessibility<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> developed at Microsoft Research India that calls for technology design and application that is motivated by play and playfulness. The focus on digital games is to enable learning of multiple skills through play. A key design element in developing these games is the autonomy of players\u2014children <em>and<\/em> teachers\u2014to define all aspects of the play and the experience. Learn more about <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.ludicdesign.org\/\">Ludic Design for Accessibility, the partners involved, and other projects leveraging the approach<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n<p><strong>The research team: <\/strong><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/mohja\/\">Mohit Jain<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/swmanohmicrosoft-com\/\">Manohar Swaminathan<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> of Microsoft Research India; Microsoft Research Fellow <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.linkedin.com\/in\/gesuindia\/?originalSubdomain=in\">Gesu India<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>; former Microsoft Research contractor <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.linkedin.com\/in\/nirmalendu-diwakar\/?originalSubdomain=in\">Nirmalendu Diwakar<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>; <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/visionempowertrust.in\/?team=vidhya-y\">Vidhya Y<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/visionempowertrust.in\/?team=aishwarya\">Aishwarya O<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> of Vision Empower, Bangalore, India; and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/infosci.cornell.edu\/content\/vashistha\">Aditya Vashistha<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> of Cornell University. <\/p>\n\n\n\n<h2 id=\"more-to-explore\">More to explore<\/h2>\n\n\n\n<h3 id=\"locomotion-inside-virtual-reality\">Locomotion inside virtual reality<\/h3>\n\n\n\n<p>As new and existing technologies progress and become a more regular part of our lives, new challenges and opportunities around accessibility and inclusion will present themselves. Virtual reality is a great example. Gains in software and hardware that lead to more immersive experiences only make the technology more attractive for use in professional and social scenarios. How can VR be designed to accommodate the variety of capabilities represented by those who want to use it?<\/p>\n\n\n\n<p>At CHI, researchers <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.youtube.com\/watch?v=KVF_V6P1698\">presented Locomotion Vault<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, an interactive database and analysis of VR locomotion techniques (LTs) being used across industry and academia. Not only was accessibility\u2014defined by the authors as the \u201cextent of motor ability that the LT requires\u201d\u2014an attribute considered during analysis, but this library of methods itself allows developers to include different locomotion options for users to choose from, making the VR experiences they\u2019re creating more accessible to different people, says <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/margon\/\">Principal Researcher Mar Gonzalez Franco<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. To use the database or suggest a new locomotion method, visit the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/locomotionvault.github.io\/about.html\">Locomotion Vault GitHub repo<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. Locomotion Vault is a collaborative effort between Microsoft Research and researchers from the University of Copenhagen and the University of Birmingham.&nbsp;<\/p>\n\n\n\n<h3 id=\"capturing-the-remote-work-experience\">Capturing the remote work experience<\/h3>\n\n\n\n<p>Outside of CHI and very relevant is a pair of papers studying remote work as experienced by people with disabilities. In\u202f\u201c<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/understanding-the-telework-experience-of-people-with-disabilities-2\/\">Understanding the Telework Experience\u202fof People with Disabilities<\/a>,\u201d\u202f<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/johntang\/\">Principal Researcher John Tang<\/a>\u202finterviewed\u202fpeople\u202fwith a range of abilities who regularly work remotely.\u202fParticipants\u202fdescribed productivity challenges and opportunities around the use of telework tools and their\u202fsudden transition to interacting with an increased number of colleagues also working remotely because of the pandemic. The\u202f<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/towards-accessible-remote-work-understanding-work-from-home-practices-of-neurodivergent-professionals\/\">second paper<\/a>\u202ffocuses on the impact remote work during the pandemic has had on\u202fprofessionals\u202fwho are\u202fneurodivergent,\u202fincluding\u202fpeople who have\u202fautism spectrum disorder, attention deficit hyperactivity disorder (ADHD), and dyslexia.\u202fBoth\u202fpapers\u202fidentify design implications for how to make remote work technologies accessible to people of all abilities.\u202f\u202f<\/p>\n\n\n\n<p>The scope of capabilities in the world is expansive, and taking a more inclusive approach and broadening the idea of ability can open doors to some truly innovative advancements.<\/p>\n\n\n\n<p>\u201cThey all require different kinds of creative ways of thinking about how we could use what people have and what their capabilities are now to more effectively allow them to do what they want to do and to live in the world,\u201d says <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/cutrell\/\">Senior Principal Research Manager Ed Cutrell<\/a>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Accessibility and inclusion represent a growing space in the technology landscape, and how research and development are being used to empower people across abilities is expanding in exciting ways. Instead of treating disabilities as conditions in need of solutions\u2014as has been the case over the years with the medical-based approach to accessibility\u2014research is moving toward [&hellip;]<\/p>\n","protected":false},"author":38838,"featured_media":747634,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243984],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-747019","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-blog-homepage-featured"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[830104,295553],"related-events":[725128],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-960x540.jpg\" class=\"img-object-cover\" alt=\"A close-up of a man wearing casual clothing, he has his smartphone in his hand and he is using an assistive mobile app for people with vision disabilities to assist him.\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-1024x577.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-768x433.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-16x9.jpg 16w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2021\/05\/1400x788_Chi_accessibility_no_logo_still.jpg 1400w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"May 21, 2021","formattedExcerpt":"Accessibility and inclusion represent a growing space in the technology landscape, and how research and development are being used to empower people across abilities is expanding in exciting ways. Instead of treating disabilities as conditions in need of solutions\u2014as has been the case over the&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/747019"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/38838"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=747019"}],"version-history":[{"count":19,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/747019\/revisions"}],"predecessor-version":[{"id":894504,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/747019\/revisions\/894504"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/747634"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=747019"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=747019"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=747019"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=747019"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=747019"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=747019"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=747019"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=747019"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=747019"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=747019"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=747019"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}