Hanna Wallach

Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

January 2025

Publication

A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts

Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach

December 2024

Publication

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems

Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach

November 2024

Publication

Dimensions of Generative AI Evaluation Design

P. A. Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach

November 2024

Publication

Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

November 2024

Publication

“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors

Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu

NAACL 2024 | June 2024

Publication

Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists

Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan

Computer-Supported Cooperative Work | April 2024

Publication Project

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

ArXiv | October 2023, Vol abs/2310.17750

DOI Publication Publication

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

October 2023

Publication

FairPrism: Evaluating Fairness-Related Harms in Text Generation

Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach

ACL 2023 | July 2023

Publication

Taxonomizing and Measuring Representational Harms: A Look at Image Tagging

Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas

AAAI 2022 | February 2023

Publication

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project Project

Measuring Representational Harms in Image Captioning

Kristen Laird, Solon Barocas, Hanna Wallach, Angelina Wang

June 2022

PDF

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan

FAccT 2022 | May 2022

DOI Project

Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support

Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach

25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022

Publication

A Human-Centered Agenda for Intelligible Machine Learning

Jennifer Wortman Vaughan, Hanna Wallach

In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021

Author's Version Project

Datasheets for Datasets

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford

Communications of the ACM | December 2021, Vol 64(12): pp. 86-92

Publication Project Project

Summarize with Caution: Comparing Global Feature Attributions

Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan

Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021

Publication Project

From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence

David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan

HCOMP 2021 | September 2021

Publication Project

Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets

Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach

ACL-IJCNLP 2021 | August 2021

Publication

Responsible computing during COVID-19 and beyond

Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf

Communications of the ACM | July 2021, Vol 67(7): pp. 30-32

https://doi.org/10.1145/3466612

Publication

Doubly Non-Central Beta Matrix Factorization for DNA Methylation Data

Aaron Schein, Anjali Nagulpally, Hanna Wallach, Patrick Flaherty

Conference on Uncertainty in Artificial Intelligence (UAI) 2021 | June 2021

Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication Project

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach

AIES 2021 | May 2021

Publication

Bridging Human–Computer Interaction and Natural Language Processing

Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang

EACL 2021 Workshop | April 2021

Publication

Measurement and Fairness

Abigail Z. Jacobs, Hanna Wallach

Fairness, Accountability, and Transparency (FAccT ’21) | March 2021

Publication DOI

Assessing and mitigating unfairness in credit models with the Fairlearn toolkit

Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach

MSR-TR-2020-34 | September 2020

Published by Microsoft

Publication

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

ACL | June 2020

Publication Video

Fairlearn: A toolkit for assessing and improving fairness in AI

Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker

MSR-TR-2020-32 | May 2020

Published by Microsoft

Publication

On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs

Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach

May 2020

Publication

A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition

Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach

April 2020

PDF

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github Project

Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI

Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach

CHI Conference on Human Factors in Computing Systems | March 2020

CHI 2020 Best Paper Award

PDF Project

Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer

Zach Harned, Hanna Wallach

Florida State University Law Review, Forthcoming | December 2019

Publication

Weight of Evidence as a Basis for Human-Oriented Explanations

David Alvarez-Melis, Hal Daumé III, Jennifer Wortman Vaughan, Hanna Wallach

Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. | October 2019

Publication

Toward Fairness in AI for People with Disabilities: A Research Roadmap

Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris

ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019

Publication Video Project

Improving fairness in machine learning systems: What do industry practitioners need?

Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019

Publication

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Yin, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2019 | May 2019

Honorable Mention Award at CHI 2019

PDF

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication Project

What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai

NAACL 2019 | April 2019

Best Paper Award, NAACL 2019

Publication

Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting

Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai

FAT* ’19: Conference on Fairness, Accountability, and Transparency | January 2019

PDF DOI Github

The Hyperedge Event Model

Bomin Kim, Aaron Schein, Bruce A. Desmarais, Hanna Wallach

July 2018

PDF Publication

A Reductions Approach to Fair Classification

Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, Hanna Wallach

FATML’17 | March 2018

Publication Github

Auditing Search Engines for Differential Satisfaction Across Demographics

Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna Wallach, Emine Yilmaz

Proceedings of the 26th International Conference on World Wide Web (Industry Track) | April 2017

Publication PDF

Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments

James ben-Aaron, Matthew Denny, Bruce Desmarais, Hanna Wallach

Public Administration Review | February 2017, Vol 77: pp. 68-77

Publication Publication Publication

Detecting and Characterizing Events

Allison June-Barlow Chaney, Hanna Wallach, Matthew Connelly, David M. Blei

EMNLP | November 2016

Publication

Flexible Models for Microclustering with Application to Entity Resolution

Glacomo Zanella, Brenda Betancourt, Hanna Wallach, Jeffrey Miller, Abbas Zaidi, Rebecca C. Steorts

30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | October 2016

Publication Publication

The Social Dynamics of Language Change in Online Networks

Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Fernando Diaz, Hanna Wallach, Jacob Eisenstein

International Conference on Social Informatics: SocInfo 2016 | October 2016

Publication Publication

Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI

Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach

Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016

Publication Publication

Poisson–Gamma Dynamical Systems

Aaron Schein, Hanna Wallach, Mingyuan Zhou

Advances in Neural Information Processing Systems 29 (NIPS 2016) | May 2016

Publication

Bag of What? Simple Noun Phrase Extraction for Text Analysis

Abram Handler, Matthew Denny, Hanna Wallach, Brendan T. O'Connor

NLP+CSS@EMNLP | April 2016

Publication

Bayesian Poisson Tucker Decomposition for Learning the Structure of International Relations

Aaron Schein, Mingyuan Zhou, David M. Blei, Hanna Wallach

33rd International Conference on Machine Learning (ICML 2016) | April 2016

Publication

Computational Social Science: Toward a Collaborative Future

Hanna Wallach

Computational Social Science: Discovery and Prediction | Published by Cambridge University Press | 2016 | Computational Social Science: Discovery and Prediction edition

Publication

Bayesian Poisson Tensor Factorization for Inferring Multilateral Relations from Sparse Dyadic Event Counts

Aaron Schein, John Paisley, David M. Blei, Hanna Wallach

Proceedings of the Twenty-First ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2015

Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015

Publication

The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation

F. Guo, C. Blundell, Hanna Wallach, K. Heller

Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015

Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014

Publication

Big Data, Machine Learning, and the Social Sciences: Fairness, Accountability, and Transparency

Hanna Wallach

Medium | December 2014

Publication

Computational Social Science: Computer-Supported Cooperative Work in the Social Media Era

Scott Counts, Munmun De Choudhury, Jana Diesner, Eric Gilbert, Marta C. Gonzalez, Brian Keegan, Mor Naaman, Hanna Wallach

CSCW'14 | February 2014

Publication

Summarizing Topics: From Word Lists to Phrases

Lauren A. Hannah, Hanna Wallach

NIPS Workshop on Modern Machine Learning and Natural Language Processing | January 2014

Publication

Computational social science and social computing

Winter Mason, Jennifer Wortman Vaughan, Hanna Wallach

Mach Learn | November 2013, Vol 95: pp. 257-260

Publication DOI

Efficient Nearest-Neighbor Search in the Probability Simplex

Kriste Krstovski, David A. Smith, Hanna Wallach, Andrew McGregor

2013 Conference on the Theory of Information Retrieval | September 2013

Publication DOI

Inferring Multilateral Relations from Dynamic Pairwise Interactions

Aaron Schein, Juston Moore, Hanna Wallach

NIPS 2013 Workshop on Frontiers of Network Analysis | September 2013

Publication

Seven Things I Wish I’d Known About Derby

Hanna Wallach

DerbyLife | March 2013

Publication

The Benefits of Double-Blind Review

Hanna Wallach

ICML 2013 | January 2013

Publication

Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

January 2025

Publication

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts

Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach

December 2024

Publication

Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems

Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach

November 2024

Publication

Dimensions of Generative AI Evaluation Design

P. A. Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach

November 2024

Publication

Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

November 2024

Publication

Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists

Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan

Computer-Supported Cooperative Work | April 2024

Publication Project

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

ArXiv | October 2023, Vol abs/2310.17750

DOI Publication Publication

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

October 2023

Publication

FairPrism: Evaluating Fairness-Related Harms in Text Generation

Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach

ACL 2023 | July 2023

Publication

Taxonomizing and Measuring Representational Harms: A Look at Image Tagging

Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas

AAAI 2022 | February 2023

Publication

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project Project

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan

FAccT 2022 | May 2022

DOI Project

Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support

Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach

25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022

Publication

A Human-Centered Agenda for Intelligible Machine Learning

Jennifer Wortman Vaughan, Hanna Wallach

In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021

Author's Version Project

Summarize with Caution: Comparing Global Feature Attributions

Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan

Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021

Publication Project

Datasheets for Datasets

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford

Communications of the ACM | December 2021, Vol 64(12): pp. 86-92

Publication Project Project

From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence

David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan

HCOMP 2021 | September 2021

Publication Project

Responsible computing during COVID-19 and beyond

Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf

Communications of the ACM | July 2021, Vol 67(7): pp. 30-32

https://doi.org/10.1145/3466612

Publication

Doubly Non-Central Beta Matrix Factorization for DNA Methylation Data

Aaron Schein, Anjali Nagulpally, Hanna Wallach, Patrick Flaherty

Conference on Uncertainty in Artificial Intelligence (UAI) 2021 | June 2021

Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication Project

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach

AIES 2021 | May 2021

Publication

Measurement and Fairness

Abigail Z. Jacobs, Hanna Wallach

Fairness, Accountability, and Transparency (FAccT ’21) | March 2021

Publication DOI

Assessing and mitigating unfairness in credit models with the Fairlearn toolkit

Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach

MSR-TR-2020-34 | September 2020

Published by Microsoft

Publication

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

ACL | June 2020

Publication Video

Fairlearn: A toolkit for assessing and improving fairness in AI

Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker

MSR-TR-2020-32 | May 2020

Published by Microsoft

Publication

A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition

Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach

April 2020

PDF

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github Project

Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI

Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach

CHI Conference on Human Factors in Computing Systems | March 2020

CHI 2020 Best Paper Award

PDF Project

Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer

Zach Harned, Hanna Wallach

Florida State University Law Review, Forthcoming | December 2019

Publication

Weight of Evidence as a Basis for Human-Oriented Explanations

David Alvarez-Melis, Hal Daumé III, Jennifer Wortman Vaughan, Hanna Wallach

Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. | October 2019

Publication

Toward Fairness in AI for People with Disabilities: A Research Roadmap

Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris

ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019

Publication Video Project

Improving fairness in machine learning systems: What do industry practitioners need?

Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019

Publication

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication Project

What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai

NAACL 2019 | April 2019

Best Paper Award, NAACL 2019

Publication

A Reductions Approach to Fair Classification

Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, Hanna Wallach

FATML’17 | March 2018

Publication Github

Flexible Models for Microclustering with Application to Entity Resolution

Glacomo Zanella, Brenda Betancourt, Hanna Wallach, Jeffrey Miller, Abbas Zaidi, Rebecca C. Steorts

30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | October 2016

Publication Publication

The Social Dynamics of Language Change in Online Networks

Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Fernando Diaz, Hanna Wallach, Jacob Eisenstein

International Conference on Social Informatics: SocInfo 2016 | October 2016

Publication Publication

Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI

Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach

Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016

Publication Publication

Poisson–Gamma Dynamical Systems

Aaron Schein, Hanna Wallach, Mingyuan Zhou

Advances in Neural Information Processing Systems 29 (NIPS 2016) | May 2016

Publication

Summarizing Topics: From Word Lists to Phrases

Lauren A. Hannah, Hanna Wallach

NIPS Workshop on Modern Machine Learning and Natural Language Processing | January 2014

Publication

Efficient Nearest-Neighbor Search in the Probability Simplex

Kriste Krstovski, David A. Smith, Hanna Wallach, Andrew McGregor

2013 Conference on the Theory of Information Retrieval | September 2013

Publication DOI

Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

January 2025

Publication

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts

Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach

December 2024

Publication

Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems

Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach

November 2024

Publication

Dimensions of Generative AI Evaluation Design

P. A. Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach

November 2024

Publication

Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

November 2024

Publication

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

ArXiv | October 2023, Vol abs/2310.17750

DOI Publication Publication

Responsible computing during COVID-19 and beyond

Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf

Communications of the ACM | July 2021, Vol 67(7): pp. 30-32

https://doi.org/10.1145/3466612

Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication Project

Measurement and Fairness

Abigail Z. Jacobs, Hanna Wallach

Fairness, Accountability, and Transparency (FAccT ’21) | March 2021

Publication DOI

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

ACL | June 2020

Publication Video

What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai

NAACL 2019 | April 2019

Best Paper Award, NAACL 2019

Publication

Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting

Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai

FAT* ’19: Conference on Fairness, Accountability, and Transparency | January 2019

PDF DOI Github

Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments

James ben-Aaron, Matthew Denny, Bruce Desmarais, Hanna Wallach

Public Administration Review | February 2017, Vol 77: pp. 68-77

Publication Publication Publication

Detecting and Characterizing Events

Allison June-Barlow Chaney, Hanna Wallach, Matthew Connelly, David M. Blei

EMNLP | November 2016

Publication

Bayesian Poisson Tucker Decomposition for Learning the Structure of International Relations

Aaron Schein, Mingyuan Zhou, David M. Blei, Hanna Wallach

33rd International Conference on Machine Learning (ICML 2016) | April 2016

Publication

Bag of What? Simple Noun Phrase Extraction for Text Analysis

Abram Handler, Matthew Denny, Hanna Wallach, Brendan T. O'Connor

NLP+CSS@EMNLP | April 2016

Publication

Computational Social Science: Toward a Collaborative Future

Hanna Wallach

Computational Social Science: Discovery and Prediction | Published by Cambridge University Press | 2016 | Computational Social Science: Discovery and Prediction edition

Publication

Bayesian Poisson Tensor Factorization for Inferring Multilateral Relations from Sparse Dyadic Event Counts

Aaron Schein, John Paisley, David M. Blei, Hanna Wallach

Proceedings of the Twenty-First ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2015

Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015

Publication

The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation

F. Guo, C. Blundell, Hanna Wallach, K. Heller

Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015

Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014

Publication

Big Data, Machine Learning, and the Social Sciences: Fairness, Accountability, and Transparency

Hanna Wallach

Medium | December 2014

Publication

Computational Social Science: Computer-Supported Cooperative Work in the Social Media Era

Scott Counts, Munmun De Choudhury, Jana Diesner, Eric Gilbert, Marta C. Gonzalez, Brian Keegan, Mor Naaman, Hanna Wallach

CSCW'14 | February 2014

Publication

Computational social science and social computing

Winter Mason, Jennifer Wortman Vaughan, Hanna Wallach

Mach Learn | November 2013, Vol 95: pp. 257-260

Publication DOI

Inferring Multilateral Relations from Dynamic Pairwise Interactions

Aaron Schein, Juston Moore, Hanna Wallach

NIPS 2013 Workshop on Frontiers of Network Analysis | September 2013

Publication

Seven Things I Wish I’d Known About Derby

Hanna Wallach

DerbyLife | March 2013

Publication

The Benefits of Double-Blind Review

Hanna Wallach

ICML 2013 | January 2013

Publication

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

The Hyperedge Event Model

Bomin Kim, Aaron Schein, Bruce A. Desmarais, Hanna Wallach

July 2018

PDF Publication

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project Project

Responsible computing during COVID-19 and beyond

Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf

Communications of the ACM | July 2021, Vol 67(7): pp. 30-32

https://doi.org/10.1145/3466612

Publication

Bridging Human–Computer Interaction and Natural Language Processing

Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang

EACL 2021 Workshop | April 2021

Publication

On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs

Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach

May 2020

Publication

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github Project

Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI

Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach

CHI Conference on Human Factors in Computing Systems | March 2020

CHI 2020 Best Paper Award

PDF Project

Toward Fairness in AI for People with Disabilities: A Research Roadmap

Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris

ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019

Publication Video Project

Improving fairness in machine learning systems: What do industry practitioners need?

Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019

Publication

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Yin, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2019 | May 2019

Honorable Mention Award at CHI 2019

PDF

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication Project

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors

Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu

NAACL 2024 | June 2024

Publication

Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets

Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach

ACL-IJCNLP 2021 | August 2021

Publication

Bridging Human–Computer Interaction and Natural Language Processing

Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang

EACL 2021 Workshop | April 2021

Publication

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

ACL | June 2020

Publication Video

On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs

Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach

May 2020

Publication

What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai

NAACL 2019 | April 2019

Best Paper Award, NAACL 2019

Publication

The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation

F. Guo, C. Blundell, Hanna Wallach, K. Heller

Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015

Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014

Publication

Measuring Representational Harms in Image Captioning

Kristen Laird, Solon Barocas, Hanna Wallach, Angelina Wang

June 2022

PDF

Toward Fairness in AI for People with Disabilities: A Research Roadmap

Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris

ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019

Publication Video Project

Auditing Search Engines for Differential Satisfaction Across Demographics

Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna Wallach, Emine Yilmaz

Proceedings of the 26th International Conference on World Wide Web (Industry Track) | April 2017

Publication PDF

Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI

Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach

Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016

Publication Publication

Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

January 2025

Publication

A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts

Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach

December 2024

Publication

Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems

Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach

November 2024

Publication

Dimensions of Generative AI Evaluation Design

P. A. Dow, Jennifer Wortman Vaughan, Solon Barocas, Chad Atalla, Alex Chouldechova, Hanna Wallach

November 2024

Publication

Evaluating Generative AI Systems is a Social Science Measurement Challenge

Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, P. A. Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs

November 2024

Publication

On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs

Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach

May 2020

Publication

A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition

Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach

April 2020

PDF

The Hyperedge Event Model

Bomin Kim, Aaron Schein, Bruce A. Desmarais, Hanna Wallach

July 2018

PDF Publication

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

ArXiv | October 2023, Vol abs/2310.17750

DOI Publication Publication

Summarize with Caution: Comparing Global Feature Attributions

Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan

Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021

Publication Project

Datasheets for Datasets

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford

Communications of the ACM | December 2021, Vol 64(12): pp. 86-92

Publication Project Project

Responsible computing during COVID-19 and beyond

Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf

Communications of the ACM | July 2021, Vol 67(7): pp. 30-32

https://doi.org/10.1145/3466612

Publication

Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer

Zach Harned, Hanna Wallach

Florida State University Law Review, Forthcoming | December 2019

Publication

Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments

James ben-Aaron, Matthew Denny, Bruce Desmarais, Hanna Wallach

Public Administration Review | February 2017, Vol 77: pp. 68-77

Publication Publication Publication

Big Data, Machine Learning, and the Social Sciences: Fairness, Accountability, and Transparency

Hanna Wallach

Medium | December 2014

Publication

Computational social science and social computing

Winter Mason, Jennifer Wortman Vaughan, Hanna Wallach

Mach Learn | November 2013, Vol 95: pp. 257-260

Publication DOI

Seven Things I Wish I’d Known About Derby

Hanna Wallach

DerbyLife | March 2013

Publication

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

Assessing and mitigating unfairness in credit models with the Fairlearn toolkit

Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach

MSR-TR-2020-34 | September 2020

Published by Microsoft

Publication

Fairlearn: A toolkit for assessing and improving fairness in AI

Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker

MSR-TR-2020-32 | May 2020

Published by Microsoft

Publication

“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors

Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu

NAACL 2024 | June 2024

Publication

Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists

Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan

Computer-Supported Cooperative Work | April 2024

Publication Project

FairPrism: Evaluating Fairness-Related Harms in Text Generation

Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach

ACL 2023 | July 2023

Publication

Taxonomizing and Measuring Representational Harms: A Look at Image Tagging

Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas

AAAI 2022 | February 2023

Publication

Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata

Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan

2022 Conference on Computer Supported Cooperative Work | November 2022

Publication Project Project

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan

FAccT 2022 | May 2022

DOI Project

Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support

Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach

25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022

Publication

From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence

David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan

HCOMP 2021 | September 2021

Publication Project

Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets

Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach

ACL-IJCNLP 2021 | August 2021

Publication

Doubly Non-Central Beta Matrix Factorization for DNA Methylation Data

Aaron Schein, Anjali Nagulpally, Hanna Wallach, Patrick Flaherty

Conference on Uncertainty in Artificial Intelligence (UAI) 2021 | June 2021

Publication

Manipulating and Measuring Model Interpretability

Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2021 | May 2021

Publication Project

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach

AIES 2021 | May 2021

Publication

Bridging Human–Computer Interaction and Natural Language Processing

Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang

EACL 2021 Workshop | April 2021

Publication

Measurement and Fairness

Abigail Z. Jacobs, Hanna Wallach

Fairness, Accountability, and Transparency (FAccT ’21) | March 2021

Publication DOI

Language (Technology) is Power: A Critical Survey of “Bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach

ACL | June 2020

Publication Video

Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning

Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan

CHI 2020 | April 2020

CHI 2020 Honorable Mention Award

Publication Github Project

Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI

Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach

CHI Conference on Human Factors in Computing Systems | March 2020

CHI 2020 Best Paper Award

PDF Project

Weight of Evidence as a Basis for Human-Oriented Explanations

David Alvarez-Melis, Hal Daumé III, Jennifer Wortman Vaughan, Hanna Wallach

Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. | October 2019

Publication

Toward Fairness in AI for People with Disabilities: A Research Roadmap

Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris

ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019

Publication Video Project

Improving fairness in machine learning systems: What do industry practitioners need?

Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019

Publication

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Yin, Jennifer Wortman Vaughan, Hanna Wallach

CHI 2019 | May 2019

Honorable Mention Award at CHI 2019

PDF

Understanding the Effect of Accuracy on Trust in Machine Learning Models

Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach

2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019

Honorable Mention Award at CHI 2019

Publication Project

What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes

Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai

NAACL 2019 | April 2019

Best Paper Award, NAACL 2019

Publication

Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting

Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai

FAT* ’19: Conference on Fairness, Accountability, and Transparency | January 2019

PDF DOI Github

A Reductions Approach to Fair Classification

Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, Hanna Wallach

FATML’17 | March 2018

Publication Github

Auditing Search Engines for Differential Satisfaction Across Demographics

Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna Wallach, Emine Yilmaz

Proceedings of the 26th International Conference on World Wide Web (Industry Track) | April 2017

Publication PDF

Detecting and Characterizing Events

Allison June-Barlow Chaney, Hanna Wallach, Matthew Connelly, David M. Blei

EMNLP | November 2016

Publication

Flexible Models for Microclustering with Application to Entity Resolution

Glacomo Zanella, Brenda Betancourt, Hanna Wallach, Jeffrey Miller, Abbas Zaidi, Rebecca C. Steorts

30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | October 2016

Publication Publication

The Social Dynamics of Language Change in Online Networks

Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Fernando Diaz, Hanna Wallach, Jacob Eisenstein

International Conference on Social Informatics: SocInfo 2016 | October 2016

Publication Publication

Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI

Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach

Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016

Publication Publication

Poisson–Gamma Dynamical Systems

Aaron Schein, Hanna Wallach, Mingyuan Zhou

Advances in Neural Information Processing Systems 29 (NIPS 2016) | May 2016

Publication

Bayesian Poisson Tucker Decomposition for Learning the Structure of International Relations

Aaron Schein, Mingyuan Zhou, David M. Blei, Hanna Wallach

33rd International Conference on Machine Learning (ICML 2016) | April 2016

Publication

Bag of What? Simple Noun Phrase Extraction for Text Analysis

Abram Handler, Matthew Denny, Hanna Wallach, Brendan T. O'Connor

NLP+CSS@EMNLP | April 2016

Publication

Bayesian Poisson Tensor Factorization for Inferring Multilateral Relations from Sparse Dyadic Event Counts

Aaron Schein, John Paisley, David M. Blei, Hanna Wallach

Proceedings of the Twenty-First ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2015

Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015

Publication

The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation

F. Guo, C. Blundell, Hanna Wallach, K. Heller

Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015

Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014

Publication

Computational Social Science: Computer-Supported Cooperative Work in the Social Media Era

Scott Counts, Munmun De Choudhury, Jana Diesner, Eric Gilbert, Marta C. Gonzalez, Brian Keegan, Mor Naaman, Hanna Wallach

CSCW'14 | February 2014

Publication

Summarizing Topics: From Word Lists to Phrases

Lauren A. Hannah, Hanna Wallach

NIPS Workshop on Modern Machine Learning and Natural Language Processing | January 2014

Publication

Efficient Nearest-Neighbor Search in the Probability Simplex

Kriste Krstovski, David A. Smith, Hanna Wallach, Andrew McGregor

2013 Conference on the Theory of Information Retrieval | September 2013

Publication DOI

Inferring Multilateral Relations from Dynamic Pairwise Interactions

Aaron Schein, Juston Moore, Hanna Wallach

NIPS 2013 Workshop on Frontiers of Network Analysis | September 2013

Publication

The Benefits of Double-Blind Review

Hanna Wallach

ICML 2013 | January 2013

Publication

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

October 2023

Publication

Measuring Representational Harms in Image Captioning

Kristen Laird, Solon Barocas, Hanna Wallach, Angelina Wang

June 2022

PDF

A Human-Centered Agenda for Intelligible Machine Learning

Jennifer Wortman Vaughan, Hanna Wallach

In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021

Author's Version Project

Computational Social Science: Toward a Collaborative Future

Hanna Wallach

Computational Social Science: Discovery and Prediction | Published by Cambridge University Press | 2016 | Computational Social Science: Discovery and Prediction edition

Publication