“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
Hanna Wallach
Partner Research Manager
Publications
Publications by Year
-
Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
Computer-Supported Cooperative Work | April 2024
-
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann (HE/HIM), Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
October 2023
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann (HE/HIM), Hanna Wallach
ACL 2023 | July 2023
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
-
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
Measuring Representational Harms in Image Captioning
Kristen Laird, Solon Barocas, Hanna Wallach, Angelina Wang
June 2022
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
FAccT 2022 | May 2022
Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
-
A Human-Centered Agenda for Intelligible Machine Learning
Jennifer Wortman Vaughan, Hanna Wallach
In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021
Datasheets for Datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
Communications of the ACM | December 2021, Vol 64(12): pp. 86-92
Summarize with Caution: Comparing Global Feature Attributions
Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence
David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan
HCOMP 2021 | September 2021
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Doubly Non-Central Beta Matrix Factorization for DNA Methylation Data
Aaron Schein, Anjali Nagulpally, Hanna Wallach, Patrick Flaherty
Conference on Uncertainty in Artificial Intelligence (UAI) 2021 | June 2021
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
AIES 2021 | May 2021
Bridging Human–Computer Interaction and Natural Language Processing
Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang
EACL 2021 Workshop | April 2021
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
-
Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
MSR-TR-2020-34 | September 2020
Published by Microsoft
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
Fairlearn: A toolkit for assessing and improving fairness in AI
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
MSR-TR-2020-32 | May 2020
Published by Microsoft
On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs
Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach
May 2020
A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
April 2020
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
-
Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
Zach Harned, Hanna Wallach
Florida State University Law Review, Forthcoming | December 2019
Weight of Evidence as a Basis for Human-Oriented Explanations
David Alvarez-Melis, Hal Daumé III, Jennifer Wortman Vaughan, Hanna Wallach
Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. | October 2019
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Yin, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2019 | May 2019
Honorable Mention Award at CHI 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai
FAT* ’19: Conference on Fairness, Accountability, and Transparency | January 2019
-
The Hyperedge Event Model
Bomin Kim, Aaron Schein, Bruce A. Desmarais, Hanna Wallach
July 2018
A Reductions Approach to Fair Classification
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, Hanna Wallach
FATML’17 | March 2018
-
Auditing Search Engines for Differential Satisfaction Across Demographics
Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna Wallach, Emine Yilmaz
Proceedings of the 26th International Conference on World Wide Web (Industry Track) | April 2017
Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments
James ben-Aaron, Matthew Denny, Bruce Desmarais, Hanna Wallach
Public Administration Review | February 2017, Vol 77: pp. 68-77
-
Detecting and Characterizing Events
Allison June-Barlow Chaney, Hanna Wallach, Matthew Connelly, David M. Blei
EMNLP | November 2016
Flexible Models for Microclustering with Application to Entity Resolution
Glacomo Zanella, Brenda Betancourt, Hanna Wallach, Jeffrey Miller, Abbas Zaidi, Rebecca C. Steorts
30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | October 2016
The Social Dynamics of Language Change in Online Networks
Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Fernando Diaz, Hanna Wallach, Jacob Eisenstein
International Conference on Social Informatics: SocInfo 2016 | October 2016
Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI
Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach
Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016
Poisson–Gamma Dynamical Systems
Aaron Schein, Hanna Wallach, Mingyuan Zhou
Advances in Neural Information Processing Systems 29 (NIPS 2016) | May 2016
Bag of What? Simple Noun Phrase Extraction for Text Analysis
Abram Handler, Matthew Denny, Hanna Wallach, Brendan T. O'Connor
NLP+CSS@EMNLP | April 2016
Bayesian Poisson Tucker Decomposition for Learning the Structure of International Relations
Aaron Schein, Mingyuan Zhou, David M. Blei, Hanna Wallach
33rd International Conference on Machine Learning (ICML 2016) | April 2016
Computational Social Science: Toward a Collaborative Future
Computational Social Science: Discovery and Prediction | Published by Cambridge University Press | 2016 | Computational Social Science: Discovery and Prediction edition
-
Bayesian Poisson Tensor Factorization for Inferring Multilateral Relations from Sparse Dyadic Event Counts
Aaron Schein, John Paisley, David M. Blei, Hanna Wallach
Proceedings of the Twenty-First ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015
The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation
F. Guo, C. Blundell, Hanna Wallach, K. Heller
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014
-
Big Data, Machine Learning, and the Social Sciences: Fairness, Accountability, and Transparency
Medium | December 2014
Computational Social Science: Computer-Supported Cooperative Work in the Social Media Era
Scott Counts, Munmun De Choudhury, Jana Diesner, Eric Gilbert, Marta C. Gonzalez, Brian Keegan, Mor Naaman, Hanna Wallach
CSCW'14 | February 2014
Summarizing Topics: From Word Lists to Phrases
Lauren A. Hannah, Hanna Wallach
NIPS Workshop on Modern Machine Learning and Natural Language Processing | January 2014
-
Computational social science and social computing
Winter Mason, Jennifer Wortman Vaughan, Hanna Wallach
Mach Learn | November 2013, Vol 95: pp. 257-260
Efficient Nearest-Neighbor Search in the Probability Simplex
Kriste Krstovski, David A. Smith, Hanna Wallach, Andrew McGregor
2013 Conference on the Theory of Information Retrieval | September 2013
Inferring Multilateral Relations from Dynamic Pairwise Interactions
Aaron Schein, Juston Moore, Hanna Wallach
NIPS 2013 Workshop on Frontiers of Network Analysis | September 2013
Seven Things I Wish I’d Known About Derby
DerbyLife | March 2013
The Benefits of Double-Blind Review
ICML 2013 | January 2013
Publications by Research Area
-
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
Bridging Human–Computer Interaction and Natural Language Processing
Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang
EACL 2021 Workshop | April 2021
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs
Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach
May 2020
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation
F. Guo, C. Blundell, Hanna Wallach, K. Heller
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014
-
Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
Computer-Supported Cooperative Work | April 2024
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann (HE/HIM), Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
October 2023
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann (HE/HIM), Hanna Wallach
ACL 2023 | July 2023
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
FAccT 2022 | May 2022
Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
A Human-Centered Agenda for Intelligible Machine Learning
Jennifer Wortman Vaughan, Hanna Wallach
In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021
Summarize with Caution: Comparing Global Feature Attributions
Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
Datasheets for Datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
Communications of the ACM | December 2021, Vol 64(12): pp. 86-92
From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence
David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan
HCOMP 2021 | September 2021
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Doubly Non-Central Beta Matrix Factorization for DNA Methylation Data
Aaron Schein, Anjali Nagulpally, Hanna Wallach, Patrick Flaherty
Conference on Uncertainty in Artificial Intelligence (UAI) 2021 | June 2021
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
AIES 2021 | May 2021
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
MSR-TR-2020-34 | September 2020
Published by Microsoft
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
Fairlearn: A toolkit for assessing and improving fairness in AI
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
MSR-TR-2020-32 | May 2020
Published by Microsoft
A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
April 2020
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
Zach Harned, Hanna Wallach
Florida State University Law Review, Forthcoming | December 2019
Weight of Evidence as a Basis for Human-Oriented Explanations
David Alvarez-Melis, Hal Daumé III, Jennifer Wortman Vaughan, Hanna Wallach
Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. | October 2019
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
A Reductions Approach to Fair Classification
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, Hanna Wallach
FATML’17 | March 2018
Flexible Models for Microclustering with Application to Entity Resolution
Glacomo Zanella, Brenda Betancourt, Hanna Wallach, Jeffrey Miller, Abbas Zaidi, Rebecca C. Steorts
30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | October 2016
The Social Dynamics of Language Change in Online Networks
Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Fernando Diaz, Hanna Wallach, Jacob Eisenstein
International Conference on Social Informatics: SocInfo 2016 | October 2016
Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI
Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach
Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016
Poisson–Gamma Dynamical Systems
Aaron Schein, Hanna Wallach, Mingyuan Zhou
Advances in Neural Information Processing Systems 29 (NIPS 2016) | May 2016
Summarizing Topics: From Word Lists to Phrases
Lauren A. Hannah, Hanna Wallach
NIPS Workshop on Modern Machine Learning and Natural Language Processing | January 2014
Efficient Nearest-Neighbor Search in the Probability Simplex
Kriste Krstovski, David A. Smith, Hanna Wallach, Andrew McGregor
2013 Conference on the Theory of Information Retrieval | September 2013
-
Measuring Representational Harms in Image Captioning
Kristen Laird, Solon Barocas, Hanna Wallach, Angelina Wang
June 2022
-
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Bridging Human–Computer Interaction and Natural Language Processing
Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang
EACL 2021 Workshop | April 2021
On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs
Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach
May 2020
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Yin, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2019 | May 2019
Honorable Mention Award at CHI 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
-
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai
FAT* ’19: Conference on Fairness, Accountability, and Transparency | January 2019
Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments
James ben-Aaron, Matthew Denny, Bruce Desmarais, Hanna Wallach
Public Administration Review | February 2017, Vol 77: pp. 68-77
Detecting and Characterizing Events
Allison June-Barlow Chaney, Hanna Wallach, Matthew Connelly, David M. Blei
EMNLP | November 2016
Bayesian Poisson Tucker Decomposition for Learning the Structure of International Relations
Aaron Schein, Mingyuan Zhou, David M. Blei, Hanna Wallach
33rd International Conference on Machine Learning (ICML 2016) | April 2016
Bag of What? Simple Noun Phrase Extraction for Text Analysis
Abram Handler, Matthew Denny, Hanna Wallach, Brendan T. O'Connor
NLP+CSS@EMNLP | April 2016
Computational Social Science: Toward a Collaborative Future
Computational Social Science: Discovery and Prediction | Published by Cambridge University Press | 2016 | Computational Social Science: Discovery and Prediction edition
Bayesian Poisson Tensor Factorization for Inferring Multilateral Relations from Sparse Dyadic Event Counts
Aaron Schein, John Paisley, David M. Blei, Hanna Wallach
Proceedings of the Twenty-First ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015
The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation
F. Guo, C. Blundell, Hanna Wallach, K. Heller
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014
Big Data, Machine Learning, and the Social Sciences: Fairness, Accountability, and Transparency
Medium | December 2014
Computational Social Science: Computer-Supported Cooperative Work in the Social Media Era
Scott Counts, Munmun De Choudhury, Jana Diesner, Eric Gilbert, Marta C. Gonzalez, Brian Keegan, Mor Naaman, Hanna Wallach
CSCW'14 | February 2014
Computational social science and social computing
Winter Mason, Jennifer Wortman Vaughan, Hanna Wallach
Mach Learn | November 2013, Vol 95: pp. 257-260
Inferring Multilateral Relations from Dynamic Pairwise Interactions
Aaron Schein, Juston Moore, Hanna Wallach
NIPS 2013 Workshop on Frontiers of Network Analysis | September 2013
Seven Things I Wish I’d Known About Derby
DerbyLife | March 2013
The Benefits of Double-Blind Review
ICML 2013 | January 2013
-
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
-
Auditing Search Engines for Differential Satisfaction Across Demographics
Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna Wallach, Emine Yilmaz
Proceedings of the 26th International Conference on World Wide Web (Industry Track) | April 2017
-
The Hyperedge Event Model
Bomin Kim, Aaron Schein, Bruce A. Desmarais, Hanna Wallach
July 2018
-
Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI
Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach
Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016
Publications by Type
-
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
Tinker, Tailor, Configure, Customize: The Articulation Work of Customizing AI Fairness Checklists
Michael Madaio, Jingya Chen, Hanna Wallach, Jennifer Wortman Vaughan
Computer-Supported Cooperative Work | April 2024
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann (HE/HIM), Hanna Wallach
ACL 2023 | July 2023
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata
Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
2022 Conference on Computer Supported Cooperative Work | November 2022
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
FAccT 2022 | May 2022
Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support
Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022) | February 2022
From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence
David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan
HCOMP 2021 | September 2021
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
Doubly Non-Central Beta Matrix Factorization for DNA Methylation Data
Aaron Schein, Anjali Nagulpally, Hanna Wallach, Patrick Flaherty
Conference on Uncertainty in Artificial Intelligence (UAI) 2021 | June 2021
Manipulating and Measuring Model Interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake Hofman, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2021 | May 2021
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
AIES 2021 | May 2021
Bridging Human–Computer Interaction and Natural Language Processing
Su Lin Blodgett, Michael Madaio, Brendan O’Connor, Hanna Wallach, Qian Yang
EACL 2021 Workshop | April 2021
Measurement and Fairness
Abigail Z. Jacobs, Hanna Wallach
Fairness, Accountability, and Transparency (FAccT ’21) | March 2021
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
ACL | June 2020
Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning
Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, Jennifer Wortman Vaughan
CHI 2020 | April 2020
CHI 2020 Honorable Mention Award
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
Weight of Evidence as a Basis for Human-Oriented Explanations
David Alvarez-Melis, Hal Daumé III, Jennifer Wortman Vaughan, Hanna Wallach
Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. | October 2019
Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
ASSETS 2019 Workshop on AI Fairness for People with Disabilities | October 2019
Improving fairness in machine learning systems: What do industry practitioners need?
Ken Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | June 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Yin, Jennifer Wortman Vaughan, Hanna Wallach
CHI 2019 | May 2019
Honorable Mention Award at CHI 2019
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Ming Ying, Jennifer Wortman Vaughan, Hanna Wallach
2019 ACM CHI Conference on Human Factors in Computing Systems | May 2019
Honorable Mention Award at CHI 2019
What’s in a Name? Reducing Bias in Bios without Access to Protected Attributes
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
NAACL 2019 | April 2019
Best Paper Award, NAACL 2019
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alex Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai
FAT* ’19: Conference on Fairness, Accountability, and Transparency | January 2019
A Reductions Approach to Fair Classification
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, Hanna Wallach
FATML’17 | March 2018
Auditing Search Engines for Differential Satisfaction Across Demographics
Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna Wallach, Emine Yilmaz
Proceedings of the 26th International Conference on World Wide Web (Industry Track) | April 2017
Detecting and Characterizing Events
Allison June-Barlow Chaney, Hanna Wallach, Matthew Connelly, David M. Blei
EMNLP | November 2016
Flexible Models for Microclustering with Application to Entity Resolution
Glacomo Zanella, Brenda Betancourt, Hanna Wallach, Jeffrey Miller, Abbas Zaidi, Rebecca C. Steorts
30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | October 2016
The Social Dynamics of Language Change in Online Networks
Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Fernando Diaz, Hanna Wallach, Jacob Eisenstein
International Conference on Social Informatics: SocInfo 2016 | October 2016
Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI
Sarah Bird, Solon Barocas, Kate Crawford, Hanna Wallach
Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), New York University | October 2016
Poisson–Gamma Dynamical Systems
Aaron Schein, Hanna Wallach, Mingyuan Zhou
Advances in Neural Information Processing Systems 29 (NIPS 2016) | May 2016
Bayesian Poisson Tucker Decomposition for Learning the Structure of International Relations
Aaron Schein, Mingyuan Zhou, David M. Blei, Hanna Wallach
33rd International Conference on Machine Learning (ICML 2016) | April 2016
Bag of What? Simple Noun Phrase Extraction for Text Analysis
Abram Handler, Matthew Denny, Hanna Wallach, Brendan T. O'Connor
NLP+CSS@EMNLP | April 2016
Bayesian Poisson Tensor Factorization for Inferring Multilateral Relations from Sparse Dyadic Event Counts
Aaron Schein, John Paisley, David M. Blei, Hanna Wallach
Proceedings of the Twenty-First ACM SIGKDD Conference on Knowledge Discovery and Data Mining | August 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015
The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation
F. Guo, C. Blundell, Hanna Wallach, K. Heller
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics | May 2015
Also presented at the Tenth Conference on Bayesian Nonparametrics, 2015; the Fifth Annual Text as Data Conference, 2014
Computational Social Science: Computer-Supported Cooperative Work in the Social Media Era
Scott Counts, Munmun De Choudhury, Jana Diesner, Eric Gilbert, Marta C. Gonzalez, Brian Keegan, Mor Naaman, Hanna Wallach
CSCW'14 | February 2014
Summarizing Topics: From Word Lists to Phrases
Lauren A. Hannah, Hanna Wallach
NIPS Workshop on Modern Machine Learning and Natural Language Processing | January 2014
Efficient Nearest-Neighbor Search in the Probability Simplex
Kriste Krstovski, David A. Smith, Hanna Wallach, Andrew McGregor
2013 Conference on the Theory of Information Retrieval | September 2013
Inferring Multilateral Relations from Dynamic Pairwise Interactions
Aaron Schein, Juston Moore, Hanna Wallach
NIPS 2013 Workshop on Frontiers of Network Analysis | September 2013
The Benefits of Double-Blind Review
ICML 2013 | January 2013
-
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann (HE/HIM), Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
October 2023
Measuring Representational Harms in Image Captioning
Kristen Laird, Solon Barocas, Hanna Wallach, Angelina Wang
June 2022
-
Summarize with Caution: Comparing Global Feature Attributions
Alex Okeson, Rich Caruana, Nick Craswell, Kori Inkpen, Scott M. Lundberg, Harsha Nori, Hanna Wallach, Jennifer Wortman Vaughan
Bulletin of the IEEE Computer Society Technical Committee on Data Engineering | December 2021
Datasheets for Datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
Communications of the ACM | December 2021, Vol 64(12): pp. 86-92
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Stretching Human Laws to Apply to Machines: The Dangers of a ‘Colorblind’ Computer
Zach Harned, Hanna Wallach
Florida State University Law Review, Forthcoming | December 2019
Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments
James ben-Aaron, Matthew Denny, Bruce Desmarais, Hanna Wallach
Public Administration Review | February 2017, Vol 77: pp. 68-77
Big Data, Machine Learning, and the Social Sciences: Fairness, Accountability, and Transparency
Medium | December 2014
Computational social science and social computing
Winter Mason, Jennifer Wortman Vaughan, Hanna Wallach
Mach Learn | November 2013, Vol 95: pp. 257-260
Seven Things I Wish I’d Known About Derby
DerbyLife | March 2013
-
Assessing and mitigating unfairness in credit models with the Fairlearn toolkit
Miro Dudík, William Chen, Solon Barocas, Mario Inchiosa, Nick Lewins, Miruna Oprescu, Joy Qiao, Mehrnoosh Sameki, Mario Schlener, Jason Tuo, Hanna Wallach
MSR-TR-2020-34 | September 2020
Published by Microsoft
Fairlearn: A toolkit for assessing and improving fairness in AI
Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, Kathleen Walker
MSR-TR-2020-32 | May 2020
Published by Microsoft
-
A Human-Centered Agenda for Intelligible Machine Learning
Jennifer Wortman Vaughan, Hanna Wallach
In Machines We Trust: Perspectives on Dependable AI | Published by MIT Press | 2021
Computational Social Science: Toward a Collaborative Future
Computational Social Science: Discovery and Prediction | Published by Cambridge University Press | 2016 | Computational Social Science: Discovery and Prediction edition
-
On the Relationships Between the Grammatical Genders of Inanimate Nouns and their Co-Occurring Adjectives and Verbs
Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damian Blasi, Hanna Wallach
May 2020
A Human in the Loop is Not Enough: The Need for Human-Subject Experiments in Facial Recognition
Forough Poursabzi-Sangdeh, Samira Samadi, Jennifer Wortman Vaughan, Hanna Wallach
April 2020
The Hyperedge Event Model
Bomin Kim, Aaron Schein, Bruce A. Desmarais, Hanna Wallach
July 2018