FATE: Fairness, Accountability, Transparency, and Ethics in AI

Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work

Wesley Hanwen Deng, Solon Barocas, Jennifer Wortman Vaughan

ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2025) | November 2025

Preprint

Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

January 2025

Publication

A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts

Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach

December 2024

Publication

Challenges in Human-Agent Communication

Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld

MSR-TR-2024-53 | December 2024

Published by Microsoft

Publication Project

Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems

Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach

November 2024

Publication

Dimensions of Generative AI Evaluation Design

P. A. Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach

November 2024

Publication

Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

November 2024

Publication

(De)Noise: Moderating the Inconsistency Between Human Decision-Makers

Nina Grgić-Hlača, Junaid Ali, Krishna P. Gummadi, Jennifer Wortman Vaughan

ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2024) | November 2024

Publication

Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions

Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan

ToCHI | August 2024

Publication

UX Matters: The Critical Role of UX in Responsible AI

Q. Vera Liao, Mihaela Vorvoreanu, Hariharan Subramonyam, Lauren Wilcox

ACM Interactions | July 2024, Vol 31: pp. 22-27

DOI Publication

“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust

Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan

7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024

Publication Project

The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels

Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat

NAACL 2024 | June 2024

Publication

“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors

Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu

NAACL 2024 | June 2024

Publication

The CLeAR Documentation Framework for AI Transparency: Recommendations for Practitioners & Context for Policymakers

Kasia Chmielinski, Sarah Newman, Chris N. Kranzinger, Michael Hind, Jennifer Wortman Vaughan, Margaret Mitchell, Julia Stoyanovich, Angelina McMillan-Major, Emily McReynolds, Kathleen Esfahany, Mary L. Gray, Audrey Chang, Maui Hudson

April 2024

Publication Project

Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists

Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan

Computer-Supported Cooperative Work | April 2024

Publication Project

A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers

Anna Gausen, Bhaskar Mitra, Siân Lindley

ACM Conference on Fairness, Accountability, and Transparency | March 2024

Publication Project Project

AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

Q. Vera Liao, Jennifer Wortman Vaughan

Harvard Data Science Review | February 2024

Publication Project

Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems

Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra

January 2024

Publication Project Project

Responsible AI Considerations in Text Summarization Research: A Review of Current Practices

Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler

Findings of EMNLP 2023 | December 2023

Publication

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

ArXiv | October 2023, Vol abs/2310.17750

DOI Publication Publication

Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal

Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7

Article 370

PDF Project

Patterns of gender-specializing query reformulation

Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand

2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023

Publication Publication

FairPrism: Evaluating Fairness-Related Harms in Text Generation

Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach

ACL 2023 | July 2023

Publication

This Prompt is Measuring : Evaluating Bias Evaluation in Language Models

Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett

Findings of ACL 2023 | July 2023

Publication

Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023

PDF Project

fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks

Steven Moore, Q. Vera Liao, Hariharan Subramonyam

CHI 2023 | April 2023

Preprint

GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse

Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023

PDF Project

Taxonomizing and Measuring Representational Harms: A Look at Image Tagging

Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas

AAAI 2022 | February 2023

Publication

De-Biasing Relevance Judgements for Fair Ranking

Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri

Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023

Publication

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project Project

Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions

Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan

NeurIPS Workshop on Human-Centered AI | October 2022

Publication

Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI

Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar

AAAI Conference on Human Computation and Crowdsourcing | October 2022

Best Paper Award, Honorable Mention

Publication Project

Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems

Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley

Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022

Publication Publication Project Project

Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews

David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad

AIES 2022 | July 2022

Publication

Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values

Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana

KDD 2022 | July 2022

Publication Project

How Different Groups Prioritize Ethical Values for Responsible AI

Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu

FAccT 2022 | May 2022

Publication DOI

Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications

Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu

NAACL 2022 | May 2022

Publication

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan

FAccT 2022 | May 2022

DOI Project

Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation

Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao, Yunfeng Zhang, Chenhao Tan

CHI 2022 | May 2022

Publication

Designing for Responsible Trust in AI Systems: A Communication Perspective

Q. Vera Liao, S. Shyam Sundar

FAccT 2022 | April 2022

Preprint

Publication

Joint Multisided Exposure Fairness for Recommendation

Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu

2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022

Publication

Responsible Language Technologies: Foreseeing and Mitigating Harms

Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang

CHI 2022 | April 2022

Publication

Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support

Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach

25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022

Publication

Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases

Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri

Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022

Publication

Exposing Query Identification for Search Transparency

Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega

2022 The Web Conference | January 2022

Publication PDF

A Human-Centered Agenda for Intelligible Machine Learning

Jennifer Wortman Vaughan, Hanna Wallach

In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021

Author's Version Project

Datasheets for Datasets

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford

Communications of the ACM | December 2021, Vol 64(12): pp. 86-92

Publication Project Project

Summarize with Caution: Comparing Global Feature Attributions

Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan

Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021

Publication Project

Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness

Nicola Neophytou, Bhaskar Mitra, Catherine Stinson

European Conference on Information Retrieval | November 2021

Publication Publication

Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets

Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach

ACL-IJCNLP 2021 | August 2021

Publication

Responsible computing during COVID-19 and beyond

Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf

Communications of the ACM | July 2021, Vol 67(7): pp. 30-32

https://doi.org/10.1145/3466612

Publication

Measuring algorithmically infused societies

Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad

Nature | June 2021, Vol 595: pp. 197-204

Publication Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication Project

“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions

Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey

CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021

Publication Video

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach

AIES 2021 | May 2021

Publication

Measurement and Fairness

Abigail Z. Jacobs, Hanna Wallach

Fairness, Accountability, and Transparency (FAccT ’21) | March 2021

Publication DOI

Overcoming Failures of Imagination in AI Infused System Development and Deployment

Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford

In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020

Publication Video

Assessing and mitigating unfairness in credit models with the Fairlearn toolkit

Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach

MSR-TR-2020-34 | September 2020

Published by Microsoft

Publication

Mitigating bias in algorithmic hiring: evaluating claims and practices

Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy

FAT* 2020 | August 2020

Publication

The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons

Solon Barocas, Andrew Selbst, Manish Raghavan

FAT* 2020 | August 2020

Publication Project

When Are Search Completion Suggestions Problematic?

Alexandra Olteanu, Fernando Diaz, Gabriella Kazai

Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020

CSCW 2020 Honorable Mention

CSCW 2020 Honorable Mention

Publication Video

Operationalizing the Legal Principle of Data Minimization for Personalization

Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck

SIGIR 2020 | July 2020

PDF

Evaluating Stochastic Rankings with Expected Exposure

Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette

Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020

Best Long Research Paper Nominee

Publication

On the Social and Technical Challenges of Web Search Autosuggestion Moderation

T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski

July 2020

Publication

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

ACL | June 2020

Publication Video

Privacy Dependencies

Solon Barocas, Karen Levy

Washington Law Review | June 2020, Vol 95(2): pp. 555-615

Publication

Fairlearn: A toolkit for assessing and improving fairness in AI

Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker

MSR-TR-2020-32 | May 2020

Published by Microsoft

Publication

Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions

Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy

ECIR 2020 | April 2020

Publication

A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition

Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach

April 2020

PDF

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github Project

“I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance

Luke Stark, Amanda Stanhaus, Denise L. Anthony

Journal of the Association for Information Science and Technology | March 2020

Publication

Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI

Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach

CHI Conference on Human Factors in Computing Systems | March 2020

CHI 2020 Best Paper Award

PDF Project

Roles for computing in social change

Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David Robinson

FAT* | January 2020

Publication

When not to design, build, or deploy

Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark

2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020

Publication Author's Version

FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval

Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand

ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20

Publication

Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer

Zach Harned, Hanna Wallach

Florida State University Law Review, Forthcoming | December 2019

Publication

Overview of the TREC 2019 Fair Ranking Track

Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier

TREC 2019 | November 2019

Publication Publication

Toward Fairness in AI for People with Disabilities: A Research Roadmap

Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris

ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019

Publication Video Project

Fair Regression: Quantitative Definitions and Reduction-Based Algorithms

Alekh Agarwal, Miro Dudík, Zhiwei Steven Wu

36th International Conference on Machine Learning | September 2019

PDF Github

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries

Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman

Frontiers in Big Data | July 2019, Vol 2

DOI Publication

Improving fairness in machine learning systems: What do industry practitioners need?

Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019

Publication

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication Project

What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai

NAACL 2019 | April 2019

Best Paper Award, NAACL 2019

Publication

The Disparate Effects of Strategic Manipulation

Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan

2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019

Publication

The Externalities of Exploration and How Data Diversity Helps Exploitation

Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu

31st Annual Conference on Learning Theory | February 2018

Publication

Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work

Wesley Hanwen Deng, Solon Barocas, Jennifer Wortman Vaughan

ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2025) | November 2025

Preprint

Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

January 2025

Publication

A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts

Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach

December 2024

Publication

Challenges in Human-Agent Communication

Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld

MSR-TR-2024-53 | December 2024

Published by Microsoft

Publication Project

Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems

Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach

November 2024

Publication

Dimensions of Generative AI Evaluation Design

P. A. Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach

November 2024

Publication

Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

November 2024

Publication

(De)Noise: Moderating the Inconsistency Between Human Decision-Makers

Nina Grgić-Hlača, Junaid Ali, Krishna P. Gummadi, Jennifer Wortman Vaughan

ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2024) | November 2024

Publication

Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions

Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan

ToCHI | August 2024

Publication

UX Matters: The Critical Role of UX in Responsible AI

Q. Vera Liao, Mihaela Vorvoreanu, Hariharan Subramonyam, Lauren Wilcox

ACM Interactions | July 2024, Vol 31: pp. 22-27

DOI Publication

“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust

Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan

7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024

Publication Project

The CLeAR Documentation Framework for AI Transparency: Recommendations for Practitioners & Context for Policymakers

Kasia Chmielinski, Sarah Newman, Chris N. Kranzinger, Michael Hind, Jennifer Wortman Vaughan, Margaret Mitchell, Julia Stoyanovich, Angelina McMillan-Major, Emily McReynolds, Kathleen Esfahany, Mary L. Gray, Audrey Chang, Maui Hudson

April 2024

Publication Project

Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists

Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan

Computer-Supported Cooperative Work | April 2024

Publication Project

A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers

Anna Gausen, Bhaskar Mitra, Siân Lindley

ACM Conference on Fairness, Accountability, and Transparency | March 2024

Publication Project Project

AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

Q. Vera Liao, Jennifer Wortman Vaughan

Harvard Data Science Review | February 2024

Publication Project

Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems

Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra

January 2024

Publication Project Project

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

ArXiv | October 2023, Vol abs/2310.17750

DOI Publication Publication

Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal

Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7

Article 370

PDF Project

FairPrism: Evaluating Fairness-Related Harms in Text Generation

Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach

ACL 2023 | July 2023

Publication

Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023

PDF Project

Taxonomizing and Measuring Representational Harms: A Look at Image Tagging

Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas

AAAI 2022 | February 2023

Publication

GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse

Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023

PDF Project

De-Biasing Relevance Judgements for Fair Ranking

Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri

Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023

Publication

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project Project

Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions

Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan

NeurIPS Workshop on Human-Centered AI | October 2022

Publication

Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI

Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar

AAAI Conference on Human Computation and Crowdsourcing | October 2022

Best Paper Award, Honorable Mention

Publication Project

Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems

Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley

Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022

Publication Publication Project Project

Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews

David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad

AIES 2022 | July 2022

Publication

Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values

Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana

KDD 2022 | July 2022

Publication Project

How Different Groups Prioritize Ethical Values for Responsible AI

Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu

FAccT 2022 | May 2022

Publication DOI

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan

FAccT 2022 | May 2022

DOI Project

Designing for Responsible Trust in AI Systems: A Communication Perspective

Q. Vera Liao, S. Shyam Sundar

FAccT 2022 | April 2022

Preprint

Publication

Joint Multisided Exposure Fairness for Recommendation

Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu

2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022

Publication

Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support

Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach

25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022

Publication

Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases

Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri

Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022

Publication

Exposing Query Identification for Search Transparency

Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega

2022 The Web Conference | January 2022

Publication PDF

A Human-Centered Agenda for Intelligible Machine Learning

Jennifer Wortman Vaughan, Hanna Wallach

In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021

Author's Version Project

Summarize with Caution: Comparing Global Feature Attributions

Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan

Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021

Publication Project

Datasheets for Datasets

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford

Communications of the ACM | December 2021, Vol 64(12): pp. 86-92

Publication Project Project

Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness

Nicola Neophytou, Bhaskar Mitra, Catherine Stinson

European Conference on Information Retrieval | November 2021

Publication Publication

Responsible computing during COVID-19 and beyond

Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf

Communications of the ACM | July 2021, Vol 67(7): pp. 30-32

https://doi.org/10.1145/3466612

Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication Project

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach

AIES 2021 | May 2021

Publication

Measurement and Fairness

Abigail Z. Jacobs, Hanna Wallach

Fairness, Accountability, and Transparency (FAccT ’21) | March 2021

Publication DOI

Overcoming Failures of Imagination in AI Infused System Development and Deployment

Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford

In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020

Publication Video

Assessing and mitigating unfairness in credit models with the Fairlearn toolkit

Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach

MSR-TR-2020-34 | September 2020

Published by Microsoft

Publication

Mitigating bias in algorithmic hiring: evaluating claims and practices

Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy

FAT* 2020 | August 2020

Publication

The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons

Solon Barocas, Andrew Selbst, Manish Raghavan

FAT* 2020 | August 2020

Publication Project

Operationalizing the Legal Principle of Data Minimization for Personalization

Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck

SIGIR 2020 | July 2020

PDF

Evaluating Stochastic Rankings with Expected Exposure

Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette

Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020

Best Long Research Paper Nominee

Publication

Privacy Dependencies

Solon Barocas, Karen Levy

Washington Law Review | June 2020, Vol 95(2): pp. 555-615

Publication

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

ACL | June 2020

Publication Video

Fairlearn: A toolkit for assessing and improving fairness in AI

Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker

MSR-TR-2020-32 | May 2020

Published by Microsoft

Publication

Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions

Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy

ECIR 2020 | April 2020

Publication

A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition

Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach

April 2020

PDF

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github Project

“I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance

Luke Stark, Amanda Stanhaus, Denise L. Anthony

Journal of the Association for Information Science and Technology | March 2020

Publication

Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI

Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach

CHI Conference on Human Factors in Computing Systems | March 2020

CHI 2020 Best Paper Award

PDF Project

When not to design, build, or deploy

Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark

2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020

Publication Author's Version

FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval

Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand

ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20

Publication

Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer

Zach Harned, Hanna Wallach

Florida State University Law Review, Forthcoming | December 2019

Publication

Overview of the TREC 2019 Fair Ranking Track

Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier

TREC 2019 | November 2019

Publication Publication

Toward Fairness in AI for People with Disabilities: A Research Roadmap

Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris

ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019

Publication Video Project

Fair Regression: Quantitative Definitions and Reduction-Based Algorithms

Alekh Agarwal, Miro Dudík, Zhiwei Steven Wu

36th International Conference on Machine Learning | September 2019

PDF Github

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries

Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman

Frontiers in Big Data | July 2019, Vol 2

DOI Publication

Improving fairness in machine learning systems: What do industry practitioners need?

Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019

Publication

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication Project

What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai

NAACL 2019 | April 2019

Best Paper Award, NAACL 2019

Publication

The Disparate Effects of Strategic Manipulation

Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan

2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019

Publication

The Externalities of Exploration and How Data Diversity Helps Exploitation

Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu

31st Annual Conference on Learning Theory | February 2018

Publication

Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

January 2025

Publication

A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts

Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach

December 2024

Publication

Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems

Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach

November 2024

Publication

Dimensions of Generative AI Evaluation Design

P. A. Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach

November 2024

Publication

Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

November 2024

Publication

A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers

Anna Gausen, Bhaskar Mitra, Siân Lindley

ACM Conference on Fairness, Accountability, and Transparency | March 2024

Publication Project Project

Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems

Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra

January 2024

Publication Project Project

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

ArXiv | October 2023, Vol abs/2310.17750

DOI Publication Publication

Responsible computing during COVID-19 and beyond

Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf

Communications of the ACM | July 2021, Vol 67(7): pp. 30-32

https://doi.org/10.1145/3466612

Publication

Measuring algorithmically infused societies

Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad

Nature | June 2021, Vol 595: pp. 197-204

Publication Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication Project

Measurement and Fairness

Abigail Z. Jacobs, Hanna Wallach

Fairness, Accountability, and Transparency (FAccT ’21) | March 2021

Publication DOI

Overcoming Failures of Imagination in AI Infused System Development and Deployment

Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford

In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020

Publication Video

Operationalizing the Legal Principle of Data Minimization for Personalization

Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck

SIGIR 2020 | July 2020

PDF

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

ACL | June 2020

Publication Video

When not to design, build, or deploy

Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark

2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020

Publication Author's Version

Overview of the TREC 2019 Fair Ranking Track

Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier

TREC 2019 | November 2019

Publication Publication

What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai

NAACL 2019 | April 2019

Best Paper Award, NAACL 2019

Publication

The CLeAR Documentation Framework for AI Transparency: Recommendations for Practitioners & Context for Policymakers

Kasia Chmielinski, Sarah Newman, Chris N. Kranzinger, Michael Hind, Jennifer Wortman Vaughan, Margaret Mitchell, Julia Stoyanovich, Angelina McMillan-Major, Emily McReynolds, Kathleen Esfahany, Mary L. Gray, Audrey Chang, Maui Hudson

April 2024

Publication Project

Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work

Wesley Hanwen Deng, Solon Barocas, Jennifer Wortman Vaughan

ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2025) | November 2025

Preprint

Challenges in Human-Agent Communication

Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld

MSR-TR-2024-53 | December 2024

Published by Microsoft

Publication Project

(De)Noise: Moderating the Inconsistency Between Human Decision-Makers

Nina Grgić-Hlača, Junaid Ali, Krishna P. Gummadi, Jennifer Wortman Vaughan

ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2024) | November 2024

Publication

Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions

Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan

ToCHI | August 2024

Publication

A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers

Anna Gausen, Bhaskar Mitra, Siân Lindley

ACM Conference on Fairness, Accountability, and Transparency | March 2024

Publication Project Project

AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

Q. Vera Liao, Jennifer Wortman Vaughan

Harvard Data Science Review | February 2024

Publication Project

Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems

Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra

January 2024

Publication Project Project

Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal

Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7

Article 370

PDF Project

Patterns of gender-specializing query reformulation

Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand

2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023

Publication Publication

fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks

Steven Moore, Q. Vera Liao, Hariharan Subramonyam

CHI 2023 | April 2023

Preprint

Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023

PDF Project

GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse

Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023

PDF Project

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project Project

Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions

Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan

NeurIPS Workshop on Human-Centered AI | October 2022

Publication

Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI

Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar

AAAI Conference on Human Computation and Crowdsourcing | October 2022

Best Paper Award, Honorable Mention

Publication Project

Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems

Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley

Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022

Publication Publication Project Project

How Different Groups Prioritize Ethical Values for Responsible AI

Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu

FAccT 2022 | May 2022

Publication DOI

Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation

Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao, Yunfeng Zhang, Chenhao Tan

CHI 2022 | May 2022

Publication

Designing for Responsible Trust in AI Systems: A Communication Perspective

Q. Vera Liao, S. Shyam Sundar

FAccT 2022 | April 2022

Preprint

Publication

Responsible Language Technologies: Foreseeing and Mitigating Harms

Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang

CHI 2022 | April 2022

Publication

Responsible computing during COVID-19 and beyond

Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf

Communications of the ACM | July 2021, Vol 67(7): pp. 30-32

https://doi.org/10.1145/3466612

Publication

Measuring algorithmically infused societies

Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad

Nature | June 2021, Vol 595: pp. 197-204

Publication Publication

“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions

Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey

CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021

Publication Video

When Are Search Completion Suggestions Problematic?

Alexandra Olteanu, Fernando Diaz, Gabriella Kazai

Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020

CSCW 2020 Honorable Mention

CSCW 2020 Honorable Mention

Publication Video

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github Project

Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI

Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach

CHI Conference on Human Factors in Computing Systems | March 2020

CHI 2020 Best Paper Award

PDF Project

When not to design, build, or deploy

Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark

2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020

Publication Author's Version

Toward Fairness in AI for People with Disabilities: A Research Roadmap

Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris

ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019

Publication Video Project

Improving fairness in machine learning systems: What do industry practitioners need?

Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019

Publication

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication Project

The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels

Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat

NAACL 2024 | June 2024

Publication

“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors

Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu

NAACL 2024 | June 2024

Publication

Responsible AI Considerations in Text Summarization Research: A Review of Current Practices

Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler

Findings of EMNLP 2023 | December 2023

Publication

This Prompt is Measuring : Evaluating Bias Evaluation in Language Models

Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett

Findings of ACL 2023 | July 2023

Publication

Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications

Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu

NAACL 2022 | May 2022

Publication

Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets

Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach

ACL-IJCNLP 2021 | August 2021

Publication

“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions

Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey

CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021

Publication Video

When Are Search Completion Suggestions Problematic?

Alexandra Olteanu, Fernando Diaz, Gabriella Kazai

Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020

CSCW 2020 Honorable Mention

CSCW 2020 Honorable Mention

Publication Video

On the Social and Technical Challenges of Web Search Autosuggestion Moderation

T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski

July 2020

Publication

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

ACL | June 2020

Publication Video

Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions

Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy

ECIR 2020 | April 2020

Publication

Overview of the TREC 2019 Fair Ranking Track

Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier

TREC 2019 | November 2019

Publication Publication

What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai

NAACL 2019 | April 2019

Best Paper Award, NAACL 2019

Publication

A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers

Anna Gausen, Bhaskar Mitra, Siân Lindley

ACM Conference on Fairness, Accountability, and Transparency | March 2024

Publication Project Project

Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems

Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra

January 2024

Publication Project Project

Patterns of gender-specializing query reformulation

Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand

2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023

Publication Publication

De-Biasing Relevance Judgements for Fair Ranking

Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri

Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023

Publication

Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems

Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley

Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022

Publication Publication Project Project

Joint Multisided Exposure Fairness for Recommendation

Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu

2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022

Publication

Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases

Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri

Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022

Publication

Exposing Query Identification for Search Transparency

Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega

2022 The Web Conference | January 2022

Publication PDF

Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness

Nicola Neophytou, Bhaskar Mitra, Catherine Stinson

European Conference on Information Retrieval | November 2021

Publication Publication

When Are Search Completion Suggestions Problematic?

Alexandra Olteanu, Fernando Diaz, Gabriella Kazai

Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020

CSCW 2020 Honorable Mention

CSCW 2020 Honorable Mention

Publication Video

Operationalizing the Legal Principle of Data Minimization for Personalization

Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck

SIGIR 2020 | July 2020

PDF

On the Social and Technical Challenges of Web Search Autosuggestion Moderation

T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski

July 2020

Publication

Evaluating Stochastic Rankings with Expected Exposure

Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette

Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020

Best Long Research Paper Nominee

Publication

Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions

Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy

ECIR 2020 | April 2020

Publication

FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval

Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand

ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20

Publication

Overview of the TREC 2019 Fair Ranking Track

Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier

TREC 2019 | November 2019

Publication Publication

Overcoming Failures of Imagination in AI Infused System Development and Deployment

Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford

In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020

Publication Video

Roles for computing in social change

Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David Robinson

FAT* | January 2020

Publication

The Externalities of Exploration and How Data Diversity Helps Exploitation

Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu

31st Annual Conference on Learning Theory | February 2018

Publication

Operationalizing the Legal Principle of Data Minimization for Personalization

Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck

SIGIR 2020 | July 2020

PDF

Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions

Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy

ECIR 2020 | April 2020

Publication

“I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance

Luke Stark, Amanda Stanhaus, Denise L. Anthony

Journal of the Association for Information Science and Technology | March 2020

Publication

Toward Fairness in AI for People with Disabilities: A Research Roadmap

Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris

ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019

Publication Video Project

The Disparate Effects of Strategic Manipulation

Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan

2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019

Publication

The Externalities of Exploration and How Data Diversity Helps Exploitation

Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu

31st Annual Conference on Learning Theory | February 2018

Publication

Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

January 2025

Publication

A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts

Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach

December 2024

Publication

Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems

Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach

November 2024

Publication

Dimensions of Generative AI Evaluation Design

P. A. Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach

November 2024

Publication

Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

November 2024

Publication

The CLeAR Documentation Framework for AI Transparency: Recommendations for Practitioners & Context for Policymakers

Kasia Chmielinski, Sarah Newman, Chris N. Kranzinger, Michael Hind, Jennifer Wortman Vaughan, Margaret Mitchell, Julia Stoyanovich, Angelina McMillan-Major, Emily McReynolds, Kathleen Esfahany, Mary L. Gray, Audrey Chang, Maui Hudson

April 2024

Publication Project

A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition

Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach

April 2020

PDF

Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work

Wesley Hanwen Deng, Solon Barocas, Jennifer Wortman Vaughan

ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2025) | November 2025

Preprint

(De)Noise: Moderating the Inconsistency Between Human Decision-Makers

Nina Grgić-Hlača, Junaid Ali, Krishna P. Gummadi, Jennifer Wortman Vaughan

ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2024) | November 2024

Publication

The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels

Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat

NAACL 2024 | June 2024

Publication

“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors

Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu

NAACL 2024 | June 2024

Publication

“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust

Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan

7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024

Publication Project

Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists

Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan

Computer-Supported Cooperative Work | April 2024

Publication Project

A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers

Anna Gausen, Bhaskar Mitra, Siân Lindley

ACM Conference on Fairness, Accountability, and Transparency | March 2024

Publication Project Project

Responsible AI Considerations in Text Summarization Research: A Review of Current Practices

Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler

Findings of EMNLP 2023 | December 2023

Publication

Patterns of gender-specializing query reformulation

Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand

2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023

Publication Publication

FairPrism: Evaluating Fairness-Related Harms in Text Generation

Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach

ACL 2023 | July 2023

Publication

This Prompt is Measuring : Evaluating Bias Evaluation in Language Models

Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett

Findings of ACL 2023 | July 2023

Publication

fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks

Steven Moore, Q. Vera Liao, Hariharan Subramonyam

CHI 2023 | April 2023

Preprint

Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023

PDF Project

Taxonomizing and Measuring Representational Harms: A Look at Image Tagging

Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas

AAAI 2022 | February 2023

Publication

GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse

Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau

2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023

PDF Project

De-Biasing Relevance Judgements for Fair Ranking

Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri

Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023

Publication

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project Project

Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions

Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan

NeurIPS Workshop on Human-Centered AI | October 2022

Publication

Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI

Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar

AAAI Conference on Human Computation and Crowdsourcing | October 2022

Best Paper Award, Honorable Mention

Publication Project

Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems

Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley

Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022

Publication Publication Project Project

Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews

David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad

AIES 2022 | July 2022

Publication

Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values

Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana

KDD 2022 | July 2022

Publication Project

How Different Groups Prioritize Ethical Values for Responsible AI

Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu

FAccT 2022 | May 2022

Publication DOI

Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications

Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu

NAACL 2022 | May 2022

Publication

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan

FAccT 2022 | May 2022

DOI Project

Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation

Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao, Yunfeng Zhang, Chenhao Tan

CHI 2022 | May 2022

Publication

Designing for Responsible Trust in AI Systems: A Communication Perspective

Q. Vera Liao, S. Shyam Sundar

FAccT 2022 | April 2022

Preprint

Publication

Joint Multisided Exposure Fairness for Recommendation

Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu

2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022

Publication

Responsible Language Technologies: Foreseeing and Mitigating Harms

Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang

CHI 2022 | April 2022

Publication

Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support

Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach

25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022

Publication

Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases

Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri

Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022

Publication

Exposing Query Identification for Search Transparency

Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega

2022 The Web Conference | January 2022

Publication PDF

Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness

Nicola Neophytou, Bhaskar Mitra, Catherine Stinson

European Conference on Information Retrieval | November 2021

Publication Publication

Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets

Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach

ACL-IJCNLP 2021 | August 2021

Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication Project

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach

AIES 2021 | May 2021

Publication

“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions

Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey

CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021

Publication Video

Measurement and Fairness

Abigail Z. Jacobs, Hanna Wallach

Fairness, Accountability, and Transparency (FAccT ’21) | March 2021

Publication DOI

Overcoming Failures of Imagination in AI Infused System Development and Deployment

Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford

In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020

Publication Video

Mitigating bias in algorithmic hiring: evaluating claims and practices

Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy

FAT* 2020 | August 2020

Publication

The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons

Solon Barocas, Andrew Selbst, Manish Raghavan

FAT* 2020 | August 2020

Publication Project

When Are Search Completion Suggestions Problematic?

Alexandra Olteanu, Fernando Diaz, Gabriella Kazai

Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020

CSCW 2020 Honorable Mention

CSCW 2020 Honorable Mention

Publication Video

Operationalizing the Legal Principle of Data Minimization for Personalization

Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck

SIGIR 2020 | July 2020

PDF

Evaluating Stochastic Rankings with Expected Exposure

Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette

Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020

Best Long Research Paper Nominee

Publication

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

ACL | June 2020

Publication Video

Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions

Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy

ECIR 2020 | April 2020

Publication

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github Project

Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI

Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach

CHI Conference on Human Factors in Computing Systems | March 2020

CHI 2020 Best Paper Award

PDF Project

Roles for computing in social change

Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David Robinson

FAT* | January 2020

Publication

When not to design, build, or deploy

Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark

2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020

Publication Author's Version

Overview of the TREC 2019 Fair Ranking Track

Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier

TREC 2019 | November 2019

Publication Publication

Toward Fairness in AI for People with Disabilities: A Research Roadmap

Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris

ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019

Publication Video Project

Fair Regression: Quantitative Definitions and Reduction-Based Algorithms

Alekh Agarwal, Miro Dudík, Zhiwei Steven Wu

36th International Conference on Machine Learning | September 2019

PDF Github

Improving fairness in machine learning systems: What do industry practitioners need?

Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019

Publication

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication Project

What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai

NAACL 2019 | April 2019

Best Paper Award, NAACL 2019

Publication

The Disparate Effects of Strategic Manipulation

Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan

2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019

Publication

The Externalities of Exploration and How Data Diversity Helps Exploitation

Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu

31st Annual Conference on Learning Theory | February 2018

Publication

Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions

Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan

ToCHI | August 2024

Publication

UX Matters: The Critical Role of UX in Responsible AI

Q. Vera Liao, Mihaela Vorvoreanu, Hariharan Subramonyam, Lauren Wilcox

ACM Interactions | July 2024, Vol 31: pp. 22-27

DOI Publication

AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap

Q. Vera Liao, Jennifer Wortman Vaughan

Harvard Data Science Review | February 2024

Publication Project

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

ArXiv | October 2023, Vol abs/2310.17750

DOI Publication Publication

Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations

Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal

Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7

Article 370

PDF Project

Summarize with Caution: Comparing Global Feature Attributions

Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan

Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021

Publication Project

Datasheets for Datasets

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford

Communications of the ACM | December 2021, Vol 64(12): pp. 86-92

Publication Project Project

Responsible computing during COVID-19 and beyond

Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf

Communications of the ACM | July 2021, Vol 67(7): pp. 30-32

https://doi.org/10.1145/3466612

Publication

Measuring algorithmically infused societies

Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad

Nature | June 2021, Vol 595: pp. 197-204

Publication Publication

Privacy Dependencies

Solon Barocas, Karen Levy

Washington Law Review | June 2020, Vol 95(2): pp. 555-615

Publication

“I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance

Luke Stark, Amanda Stanhaus, Denise L. Anthony

Journal of the Association for Information Science and Technology | March 2020

Publication

FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval

Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand

ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20

Publication

Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer

Zach Harned, Hanna Wallach

Florida State University Law Review, Forthcoming | December 2019

Publication

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries

Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman

Frontiers in Big Data | July 2019, Vol 2

DOI Publication

Challenges in Human-Agent Communication

Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld

MSR-TR-2024-53 | December 2024

Published by Microsoft

Publication Project

Assessing and mitigating unfairness in credit models with the Fairlearn toolkit

Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach

MSR-TR-2020-34 | September 2020

Published by Microsoft

Publication

Fairlearn: A toolkit for assessing and improving fairness in AI

Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker

MSR-TR-2020-32 | May 2020

Published by Microsoft

Publication

Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems

Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra

January 2024

Publication Project Project

On the Social and Technical Challenges of Web Search Autosuggestion Moderation

T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski

July 2020

Publication

A Human-Centered Agenda for Intelligible Machine Learning

Jennifer Wortman Vaughan, Hanna Wallach

In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021

Author's Version Project