Planning for Natural Language Failures with the AI Playbook

2021 CHI Conference on Human Factors in Computing Systems |

Download BibTex

Prototyping AI user experiences is challenging due in part to probabilistic AI models making it difficult to anticipate, test, and mitigate AI failures before deployment. In this work, we set out to support practitioners with early AI prototyping, with a focus on natural language (NL)-based technologies. Our interviews with 12 NL practitioners from a large technology company revealed that, in addition to challenges prototyping AI, prototyping was often not happening at all or focused only on idealized scenarios due to a lack of tools and tight timelines. These findings informed our design of the AI Playbook, an interactive and low-cost tool we developed to encourage proactive and systematic consideration of AI errors before deployment. Our evaluation of the AI Playbook demonstrates its potential to 1) encourage product teams to prioritize both ideal and failure scenarios, 2) standardize the articulation of AI failures from a user experience perspective, and 3) act as a boundary object between user experience designers, data scientists, and engineers.

Launch the Playbook (opens in new tab)

Visit our GitHub page (opens in new tab) to learn more.

Create human-centered AI with the Human-AI eXperience (HAX) Toolkit webinar

The tech industry is being called upon to develop and deploy AI technologies more responsibly. Yet many organizations that create AI technologies report being unprepared to address AI risks and failures.

To meet these challenges, Microsoft is striving to take a human-centered approach to AI, designing and building technologies that benefit people and society while also mitigating potential harms. This includes understanding human needs and using that insight to drive development decisions from beginning to end.

To assist AI practitioners in building human-centered AI, we are introducing the Human-AI eXperience (HAX) Toolkit, launching on July 19. This suite of tools spans the end-to-end product development lifecycle, providing support where AI practitioners have requested it.

In this webinar, join Saleema Amershi, Senior Principal Research Manager, and Mihaela Vorvoreanu, Aether Director of UX Research and RAI education, to learn how and when to use each tool in the HAX Toolkit to create human-centered AI.

Together, you’ll explore:

  • Guidelines for Human-AI Interaction – best practices for how AI systems should behave during user interactions that synthesize more than 20 years of guidance on this topic
  • The HAX Workbook – a tool to guide teams through planning and implementing human-AI interaction best practices
  • The HAX Design Patterns – a set of flexible solutions to recurring human-AI interaction problems
  • The HAX Playbook – an interactive tool for generating scenarios to test based on likely human-AI interaction failures

Resource list:

*This on-demand webinar features a previously recorded Q&A session and open captioning.

Explore more Microsoft Research webinars: https://aka.ms/msrwebinars (opens in new tab)