UX Matters: The Critical Role of UX in Responsible AI

Let’s imagine a scenario—inspired by true events—in which a company has deployed an AI-powered system in a hospital. The system provides recommendations for treatment plans. Some clinicians find that the new system requires them to change their routine and significantly adds to their workload, so they start resisting the use of the system. Other clinicians are amazed by this new, powerful technology and overly rely on the AI by accepting its recommendations even when they are incorrect, resulting in medical errors with negative effects on patients—especially women, for whom the AI system tends to underperform. Unfortunately, this is a scenario that happens too often as AI technologies are being deployed in various domains. A responsible approach to AI development and deployment should have aimed to prevent these issues. UX practitioners could have been involved to play an instrumental role. For example:

• UX researchers could have identified stakeholder needs, concerns, and values, including those of clinicians and patients, to inform better choices of AI use cases, datasets, model parameters, evaluation criteria, and so on.
• UX designers could have taken a leading role in designing the system interactions to be compatible with current work processes, and created interface features that help mitigate overreliance on AI