Human-Centered Transparency & Intelligibility in AI

“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust

Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan

7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024

Publication

AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

Q. Vera Liao, Jennifer Wortman Vaughan

Harvard Data Science Review | February 2024

Publication

Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal

Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7

Article 370

PDF

Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023

PDF

GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse

Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023

PDF

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project

Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI

Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar

AAAI Conference on Human Computation and Crowdsourcing | October 2022

Best Paper Award, Honorable Mention

Publication

Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values

Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana

KDD 2022 | July 2022

Publication

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan

FAccT 2022 | May 2022

DOI

A Human-Centered Agenda for Intelligible Machine Learning

Jennifer Wortman Vaughan, Hanna Wallach

In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021

Author's Version

Datasheets for Datasets

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford

Communications of the ACM | December 2021, Vol 64(12): pp. 86-92

Publication Project

Summarize with Caution: Comparing Global Feature Attributions

Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan

Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021

Publication

From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence

David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan

HCOMP 2021 | September 2021

Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication

The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons

Solon Barocas, Andrew Selbst, Manish Raghavan

FAT* 2020 | August 2020

Publication

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication

“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust

Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan

7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024

Publication

AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

Q. Vera Liao, Jennifer Wortman Vaughan

Harvard Data Science Review | February 2024

Publication

Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal

Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7

Article 370

PDF

Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023

PDF

GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse

Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023

PDF

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project

Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI

Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar

AAAI Conference on Human Computation and Crowdsourcing | October 2022

Best Paper Award, Honorable Mention

Publication

Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values

Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana

KDD 2022 | July 2022

Publication

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan

FAccT 2022 | May 2022

DOI

A Human-Centered Agenda for Intelligible Machine Learning

Jennifer Wortman Vaughan, Hanna Wallach

In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021

Author's Version

Summarize with Caution: Comparing Global Feature Attributions

Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan

Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021

Publication

Datasheets for Datasets

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford

Communications of the ACM | December 2021, Vol 64(12): pp. 86-92

Publication Project

From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence

David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan

HCOMP 2021 | September 2021

Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication

The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons

Solon Barocas, Andrew Selbst, Manish Raghavan

FAT* 2020 | August 2020

Publication

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication

AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

Q. Vera Liao, Jennifer Wortman Vaughan

Harvard Data Science Review | February 2024

Publication

Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal

Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7

Article 370

PDF

Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023

PDF

GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse

Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023

PDF

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project

Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI

Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar

AAAI Conference on Human Computation and Crowdsourcing | October 2022

Best Paper Award, Honorable Mention

Publication

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication

“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust

Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan

7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024

Publication

Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023

PDF

GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse

Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023

PDF

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project

Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI

Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar

AAAI Conference on Human Computation and Crowdsourcing | October 2022

Best Paper Award, Honorable Mention

Publication

Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values

Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana

KDD 2022 | July 2022

Publication

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan

FAccT 2022 | May 2022

DOI

From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence

David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan

HCOMP 2021 | September 2021

Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication

The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons

Solon Barocas, Andrew Selbst, Manish Raghavan

FAT* 2020 | August 2020

Publication

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication

AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

Q. Vera Liao, Jennifer Wortman Vaughan

Harvard Data Science Review | February 2024

Publication

Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal

Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7

Article 370

PDF

Summarize with Caution: Comparing Global Feature Attributions

Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan

Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021

Publication

Datasheets for Datasets

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford

Communications of the ACM | December 2021, Vol 64(12): pp. 86-92

Publication Project

A Human-Centered Agenda for Intelligible Machine Learning

Jennifer Wortman Vaughan, Hanna Wallach

In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021

Author's Version