Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | August 2024
Publications
Publications
Publications by Year
-
UX Matters: The Critical Role of UX in Responsible AI
Q. Vera Liao, Mihaela Vorvoreanu, Hariharan Subramonyam, Lauren Wilcox
ACM Interactions | July 2024, Vol 31: pp. 22-27
“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
NAACL 2024 | June 2024
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
Computer-Supported Cooperative Work | April 2024
A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers
Anna Gausen, Bhaskar Mitra, Siân Lindley
ACM Conference on Fairness, Accountability, and Transparency | March 2024
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
January 2024
-
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
Findings of EMNLP 2023 | December 2023
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
Patterns of gender-specializing query reformulation
Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand
2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann (HE/HIM), Hanna Wallach
ACL 2023 | July 2023
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
Findings of ACL 2023 | July 2023
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
Steven Moore, Q. Vera Liao, Hariharan Subramonyam
CHI 2023 | April 2023
GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse
Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
De-Biasing Relevance Judgements for Fair Ranking
Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023
-
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems
Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley
Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022
Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
AIES 2022 | July 2022
Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values
Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana
KDD 2022 | July 2022
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
NAACL 2022 | May 2022
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
FAccT 2022 | May 2022
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao, Yunfeng Zhang, Chenhao Tan
CHI 2022 | May 2022
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
Joint Multisided Exposure Fairness for Recommendation
Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu
2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022
Responsible Language Technologies: Foreseeing and Mitigating Harms
Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
CHI 2022 | April 2022
Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases
Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022
Exposing Query Identification for Search Transparency
Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega
2022 The Web Conference | January 2022
-
A Human-Centered Agenda for Intelligible Machine Learning
Jennifer Wortman Vaughan, Hanna Wallach
In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021
Datasheets for Datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
Communications of the ACM | December 2021, Vol 64(12): pp. 86-92
Summarize with Caution: Comparing Global Feature Attributions
Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness
Nicola Neophytou, Bhaskar Mitra, Catherine Stinson
European Conference on Information Retrieval | November 2021
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
AIES 2021 | May 2021
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
-
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
MSR-TR-2020-34 | September 2020
Published by Microsoft
Mitigating bias in algorithmic hiring: evaluating claims and practices
Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy
FAT* 2020 | August 2020
The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
Solon Barocas, Andrew Selbst, Manish Raghavan
FAT* 2020 | August 2020
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
Privacy Dependencies
Solon Barocas, Karen Levy
Washington Law Review | June 2020, Vol 95(2): pp. 555-615
Fairlearn: A toolkit for assessing and improving fairness in AI
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
MSR-TR-2020-32 | May 2020
Published by Microsoft
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
April 2020
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
“I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance
Luke Stark, Amanda Stanhaus, Denise L. Anthony
Journal of the Association for Information Science and Technology | March 2020
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
Roles for computing in social change
Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David Robinson
FAT* | January 2020
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
-
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
Zach Harned, Hanna Wallach
Florida State University Law Review, Forthcoming | December 2019
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
Fair Regression: Quantitative Definitions and Reduction-Based Algorithms
Alekh Agarwal, Miro Dudík, Zhiwei Steven Wu
36th International Conference on Machine Learning | September 2019
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
Frontiers in Big Data | July 2019, Vol 2
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
The Disparate Effects of Strategic Manipulation
Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan
2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019
-
The Externalities of Exploration and How Data Diversity Helps Exploitation
Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu
31st Annual Conference on Learning Theory | February 2018
Publications by Research Area
-
Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | August 2024
UX Matters: The Critical Role of UX in Responsible AI
Q. Vera Liao, Mihaela Vorvoreanu, Hariharan Subramonyam, Lauren Wilcox
ACM Interactions | July 2024, Vol 31: pp. 22-27
“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
Computer-Supported Cooperative Work | April 2024
A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers
Anna Gausen, Bhaskar Mitra, Siân Lindley
ACM Conference on Fairness, Accountability, and Transparency | March 2024
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
January 2024
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann (HE/HIM), Hanna Wallach
ACL 2023 | July 2023
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse
Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023
De-Biasing Relevance Judgements for Fair Ranking
Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems
Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley
Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022
Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
AIES 2022 | July 2022
Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values
Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana
KDD 2022 | July 2022
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
FAccT 2022 | May 2022
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
Joint Multisided Exposure Fairness for Recommendation
Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu
2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022
Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases
Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022
Exposing Query Identification for Search Transparency
Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega
2022 The Web Conference | January 2022
A Human-Centered Agenda for Intelligible Machine Learning
Jennifer Wortman Vaughan, Hanna Wallach
In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021
Summarize with Caution: Comparing Global Feature Attributions
Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
Datasheets for Datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
Communications of the ACM | December 2021, Vol 64(12): pp. 86-92
Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness
Nicola Neophytou, Bhaskar Mitra, Catherine Stinson
European Conference on Information Retrieval | November 2021
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
AIES 2021 | May 2021
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
MSR-TR-2020-34 | September 2020
Published by Microsoft
Mitigating bias in algorithmic hiring: evaluating claims and practices
Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy
FAT* 2020 | August 2020
The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
Solon Barocas, Andrew Selbst, Manish Raghavan
FAT* 2020 | August 2020
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
Privacy Dependencies
Solon Barocas, Karen Levy
Washington Law Review | June 2020, Vol 95(2): pp. 555-615
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
Fairlearn: A toolkit for assessing and improving fairness in AI
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
MSR-TR-2020-32 | May 2020
Published by Microsoft
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
April 2020
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
“I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance
Luke Stark, Amanda Stanhaus, Denise L. Anthony
Journal of the Association for Information Science and Technology | March 2020
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
Zach Harned, Hanna Wallach
Florida State University Law Review, Forthcoming | December 2019
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
Fair Regression: Quantitative Definitions and Reduction-Based Algorithms
Alekh Agarwal, Miro Dudík, Zhiwei Steven Wu
36th International Conference on Machine Learning | September 2019
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
Frontiers in Big Data | July 2019, Vol 2
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
The Disparate Effects of Strategic Manipulation
Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan
2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019
The Externalities of Exploration and How Data Diversity Helps Exploitation
Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu
31st Annual Conference on Learning Theory | February 2018
-
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
NAACL 2024 | June 2024
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
Findings of EMNLP 2023 | December 2023
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
Findings of ACL 2023 | July 2023
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
NAACL 2022 | May 2022
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
-
Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | August 2024
A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers
Anna Gausen, Bhaskar Mitra, Siân Lindley
ACM Conference on Fairness, Accountability, and Transparency | March 2024
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
January 2024
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
Patterns of gender-specializing query reformulation
Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand
2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023
fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
Steven Moore, Q. Vera Liao, Hariharan Subramonyam
CHI 2023 | April 2023
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse
Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems
Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley
Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao, Yunfeng Zhang, Chenhao Tan
CHI 2022 | May 2022
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
Responsible Language Technologies: Foreseeing and Mitigating Harms
Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
CHI 2022 | April 2022
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
-
A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers
Anna Gausen, Bhaskar Mitra, Siân Lindley
ACM Conference on Fairness, Accountability, and Transparency | March 2024
Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
January 2024
Patterns of gender-specializing query reformulation
Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand
2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023
De-Biasing Relevance Judgements for Fair Ranking
Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023
Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems
Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley
Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022
Joint Multisided Exposure Fairness for Recommendation
Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu
2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022
Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases
Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022
Exposing Query Identification for Search Transparency
Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega
2022 The Web Conference | January 2022
Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness
Nicola Neophytou, Bhaskar Mitra, Catherine Stinson
European Conference on Information Retrieval | November 2021
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
-
A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers
Anna Gausen, Bhaskar Mitra, Siân Lindley
ACM Conference on Fairness, Accountability, and Transparency | March 2024
Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
January 2024
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
-
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
-
Roles for computing in social change
Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David Robinson
FAT* | January 2020
The Externalities of Exploration and How Data Diversity Helps Exploitation
Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu
31st Annual Conference on Learning Theory | February 2018
-
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
“I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance
Luke Stark, Amanda Stanhaus, Denise L. Anthony
Journal of the Association for Information Science and Technology | March 2020
-
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
-
The Disparate Effects of Strategic Manipulation
Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan
2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019
The Externalities of Exploration and How Data Diversity Helps Exploitation
Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu
31st Annual Conference on Learning Theory | February 2018
Publications by Type
-
Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | August 2024
UX Matters: The Critical Role of UX in Responsible AI
Q. Vera Liao, Mihaela Vorvoreanu, Hariharan Subramonyam, Lauren Wilcox
ACM Interactions | July 2024, Vol 31: pp. 22-27
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
Summarize with Caution: Comparing Global Feature Attributions
Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
Datasheets for Datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
Communications of the ACM | December 2021, Vol 64(12): pp. 86-92
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
Privacy Dependencies
Solon Barocas, Karen Levy
Washington Law Review | June 2020, Vol 95(2): pp. 555-615
“I Don’t Want Someone to Watch Me While I’m Working”: Gendered Views of Facial Recognition Technology in Workplace Surveillance
Luke Stark, Amanda Stanhaus, Denise L. Anthony
Journal of the Association for Information Science and Technology | March 2020
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
Zach Harned, Hanna Wallach
Florida State University Law Review, Forthcoming | December 2019
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
Frontiers in Big Data | July 2019, Vol 2
-
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
NAACL 2024 | June 2024
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
Computer-Supported Cooperative Work | April 2024
A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to Workers
Anna Gausen, Bhaskar Mitra, Siân Lindley
ACM Conference on Fairness, Accountability, and Transparency | March 2024
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
Findings of EMNLP 2023 | December 2023
Patterns of gender-specializing query reformulation
Amifa Raj, Bhaskar Mitra, Nick Craswell, Michael D. Ekstrand
2023 International ACM SIGIR Conference on Research and Development in Information Retrieval | July 2023
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann (HE/HIM), Hanna Wallach
ACL 2023 | July 2023
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
Findings of ACL 2023 | July 2023
fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
Steven Moore, Q. Vera Liao, Hariharan Subramonyam
CHI 2023 | April 2023
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
GAM Coach: Towards Interactive and User-Centered Algorithmic Recourse
Zijie J. Wang, Jennifer Wortman Vaughan, Rich Caruana, Duen Horng Chau
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | February 2023
De-Biasing Relevance Judgements for Fair Ranking
Amin Bigdeli, Negar Arabzadeh, Shirin Seyedsalehi, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
Proceedings of the 45th European Conference on Information Retrieval (ECIR) | January 2023
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems
Ida Larsen-Ledet, Bhaskar Mitra, Siân Lindley
Proceedings of the 5th FAccTRec Workshop on Responsible Recommendation at RecSys 2022 | September 2022
Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
AIES 2022 | July 2022
Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values
Zijie J. Wang, Alex Kale, Harsha Nori, Peter Stella, Mark E. Nunnally, Duen Horng Chau, Mihaela Vorvoreanu, Jennifer Wortman Vaughan, Rich Caruana
KDD 2022 | July 2022
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
NAACL 2022 | May 2022
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
FAccT 2022 | May 2022
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao, Yunfeng Zhang, Chenhao Tan
CHI 2022 | May 2022
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
Joint Multisided Exposure Fairness for Recommendation
Bhaskar Mitra, Haolun Wu, Chen Ma, Fernando Diaz, Xue Liu
2022 International ACM SIGIR Conference on Research and Development in Information Retrieval | April 2022
Responsible Language Technologies: Foreseeing and Mitigating Harms
Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
CHI 2022 | April 2022
Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
Bias-aware Fair Neural Ranking for Addressing Stereotypical Gender Biases
Shirin SeyedSalehi, Amin Bigdeli, Negar Arabzadeh, Bhaskar Mitra, Morteza Zihayat, Ebrahim Bagheri
Proceedings of the 25th International Conference on Extending Database Technology (EDBT) | February 2022
Exposing Query Identification for Search Transparency
Ruohan Li, Jianxiang Li, Bhaskar Mitra, Fernando Diaz, Asia Biega
2022 The Web Conference | January 2022
Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness
Nicola Neophytou, Bhaskar Mitra, Catherine Stinson
European Conference on Information Retrieval | November 2021
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
AIES 2021 | May 2021
“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
Mitigating bias in algorithmic hiring: evaluating claims and practices
Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy
FAT* 2020 | August 2020
The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
Solon Barocas, Andrew Selbst, Manish Raghavan
FAT* 2020 | August 2020
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
Roles for computing in social change
Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David Robinson
FAT* | January 2020
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
Fair Regression: Quantitative Definitions and Reduction-Based Algorithms
Alekh Agarwal, Miro Dudík, Zhiwei Steven Wu
36th International Conference on Machine Learning | September 2019
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
The Disparate Effects of Strategic Manipulation
Lily Hu, Nicole Immorlica, Jennifer Wortman Vaughan
2nd ACM Conference on Fairness, Accountability, and Transparency | January 2019
The Externalities of Exploration and How Data Diversity Helps Exploitation
Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, Zhiwei Steven Wu
31st Annual Conference on Learning Theory | February 2018
-
Through the Looking-Glass: Transparency Implications and Challenges in Enterprise AI Knowledge Systems
Karina Cortiñas-Lorenzo, Siân Lindley, Ida Larsen-Ledet, Bhaskar Mitra
January 2024
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
-
Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
MSR-TR-2020-34 | September 2020
Published by Microsoft
Fairlearn: A toolkit for assessing and improving fairness in AI
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
MSR-TR-2020-32 | May 2020
Published by Microsoft
-
A Human-Centered Agenda for Intelligible Machine Learning
Jennifer Wortman Vaughan, Hanna Wallach
In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021
-
A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
April 2020