Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | August 2024
Return to Microsoft Research Lab – Montréal
FATE | Montréal
Publications
Publications by Year
-
“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
NAACL 2024 | June 2024
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
-
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
Findings of EMNLP 2023 | December 2023
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann (HE/HIM), Hanna Wallach
ACL 2023 | July 2023
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
Findings of ACL 2023 | July 2023
AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
June 2023
Can Workers Meaningfully Consent To Workplace Wellbeing Technologies?
Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
2023 ACM Conference on Fairness, Accountability, and Transparency | June 2023
The Kitmus Test: Evaluating Knowledge Integration From Multiple Sources In Natural Language Understanding Systems
Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung
ACL 2023 | May 2023
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
Steven Moore, Q. Vera Liao, Hariharan Subramonyam
CHI 2023 | April 2023
Sensing Wellbeing in the Workplace, Why and For Whom? Envisioning Impacts with Organizational Stakeholders
Anna Kawakami, Shreya Chowdhary, Shamsi T. Iqbal, Q. Liao, Alexandra Olteanu, J. Suh, Koustuv Saha, Alexandra Olteanu, Jina Suh
Proceedings of the ACM on Human-Computer Interaction | March 2023, Vol 7: pp. 1-33
Powering an AI Chatbot with Expert Sourcing to Support Credible Information Access
Ziang Xiao, Q. Vera Liao, Michelle X. Zhou, Tyrone Grandison, Yunyao Li
IUI 2023 | March 2023
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
-
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
AIES 2022 | July 2022
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
NAACL 2022 | May 2022
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
Responsible Language Technologies: Foreseeing and Mitigating Harms
Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
CHI 2022 | April 2022
-
ADEPT: An Adjective-Dependent Plausibility Task
Ali Emami, Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung
ACL-IJCNLP 2021 | August 2021
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
-
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
External Information Sharing on Health Forums: An Exploration
Dana Nguyen, Alexandra Olteanu, Emre Kiciman
Intl. Conf. on Web and Social Media (ICWSM) | June 2020
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
Privacy Norms and Preferences for Photos Posted Online
Roberto Hoyle, Luke Stark, Qatrunnada Ismail, David Crandall, Apu Kapadia, Denise Anthony
ACM Transactions on Computer-Human Interaction | March 2020
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
-
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
Report on the First HIPstIR Workshop on the Future of Information Retrieval
Laura Dietz, Bhaskar Mitra, Jeremy Pickens, Hana Anber, Sandeep Avula, Asia J. Biega, Adrian Boteanu, Shubham Chatterjee, Jeff Dalton, Shiri Dori-Hacohen, John Foley, Henry Feild, Ben Gamari, Rosie Jones, Pallika Kanani, Sumanta Kashyapi, Widad Machmouchi, Matthew Mitsui, Steve Nole, Alexandre Tachard Passos, Jordan Ramsdell, Adam Roegiest, David Smith, Alessandro Sordoni
SIGIR Forum | December 2019, Vol 53(2): pp. 62-75
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
Frontiers in Big Data | July 2019, Vol 2
Gaps in Information Access in Social Networks
Benjamin Fish, Ashkan Bashardoust, danah boyd, Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian
The Web Conference 2019 | May 2019
Publications by Research Area
-
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
NAACL 2024 | June 2024
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
Findings of EMNLP 2023 | December 2023
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
Findings of ACL 2023 | July 2023
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
NAACL 2022 | May 2022
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
-
Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | August 2024
“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann (HE/HIM), Hanna Wallach
ACL 2023 | July 2023
AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
June 2023
Can Workers Meaningfully Consent To Workplace Wellbeing Technologies?
Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
2023 ACM Conference on Fairness, Accountability, and Transparency | June 2023
The Kitmus Test: Evaluating Knowledge Integration From Multiple Sources In Natural Language Understanding Systems
Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung
ACL 2023 | May 2023
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
AIES 2022 | July 2022
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
ADEPT: An Adjective-Dependent Plausibility Task
Ali Emami, Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung
ACL-IJCNLP 2021 | August 2021
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
External Information Sharing on Health Forums: An Exploration
Dana Nguyen, Alexandra Olteanu, Emre Kiciman
Intl. Conf. on Web and Social Media (ICWSM) | June 2020
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
Report on the First HIPstIR Workshop on the Future of Information Retrieval
Laura Dietz, Bhaskar Mitra, Jeremy Pickens, Hana Anber, Sandeep Avula, Asia J. Biega, Adrian Boteanu, Shubham Chatterjee, Jeff Dalton, Shiri Dori-Hacohen, John Foley, Henry Feild, Ben Gamari, Rosie Jones, Pallika Kanani, Sumanta Kashyapi, Widad Machmouchi, Matthew Mitsui, Steve Nole, Alexandre Tachard Passos, Jordan Ramsdell, Adam Roegiest, David Smith, Alessandro Sordoni
SIGIR Forum | December 2019, Vol 53(2): pp. 62-75
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
Frontiers in Big Data | July 2019, Vol 2
Gaps in Information Access in Social Networks
Benjamin Fish, Ashkan Bashardoust, danah boyd, Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian
The Web Conference 2019 | May 2019
-
Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | August 2024
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
June 2023
Can Workers Meaningfully Consent To Workplace Wellbeing Technologies?
Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
2023 ACM Conference on Fairness, Accountability, and Transparency | June 2023
The Kitmus Test: Evaluating Knowledge Integration From Multiple Sources In Natural Language Understanding Systems
Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung
ACL 2023 | May 2023
fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
Steven Moore, Q. Vera Liao, Hariharan Subramonyam
CHI 2023 | April 2023
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
Sensing Wellbeing in the Workplace, Why and For Whom? Envisioning Impacts with Organizational Stakeholders
Anna Kawakami, Shreya Chowdhary, Shamsi T. Iqbal, Q. Liao, Alexandra Olteanu, J. Suh, Koustuv Saha, Alexandra Olteanu, Jina Suh
Proceedings of the ACM on Human-Computer Interaction | March 2023, Vol 7: pp. 1-33
Powering an AI Chatbot with Expert Sourcing to Support Credible Information Access
Ziang Xiao, Q. Vera Liao, Michelle X. Zhou, Tyrone Grandison, Yunyao Li
IUI 2023 | March 2023
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
Responsible Language Technologies: Foreseeing and Mitigating Harms
Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
CHI 2022 | April 2022
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
Privacy Norms and Preferences for Photos Posted Online
Roberto Hoyle, Luke Stark, Qatrunnada Ismail, David Crandall, Apu Kapadia, Denise Anthony
ACM Transactions on Computer-Human Interaction | March 2020
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
-
Can Workers Meaningfully Consent To Workplace Wellbeing Technologies?
Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
2023 ACM Conference on Fairness, Accountability, and Transparency | June 2023
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
-
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
-
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
Report on the First HIPstIR Workshop on the Future of Information Retrieval
Laura Dietz, Bhaskar Mitra, Jeremy Pickens, Hana Anber, Sandeep Avula, Asia J. Biega, Adrian Boteanu, Shubham Chatterjee, Jeff Dalton, Shiri Dori-Hacohen, John Foley, Henry Feild, Ben Gamari, Rosie Jones, Pallika Kanani, Sumanta Kashyapi, Widad Machmouchi, Matthew Mitsui, Steve Nole, Alexandre Tachard Passos, Jordan Ramsdell, Adam Roegiest, David Smith, Alessandro Sordoni
SIGIR Forum | December 2019, Vol 53(2): pp. 62-75
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
-
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
Privacy Norms and Preferences for Photos Posted Online
Roberto Hoyle, Luke Stark, Qatrunnada Ismail, David Crandall, Apu Kapadia, Denise Anthony
ACM Transactions on Computer-Human Interaction | March 2020
Publications by Type
-
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
NAACL 2024 | June 2024
“One-Size-Fits-All”? Examining Expectations around What Constitute “Fair” or “Good” NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
NAACL 2024 | June 2024
“I’m Not Sure, But…”: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust
Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, Jennifer Wortman Vaughan
7th ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024) | June 2024
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Yu Lu Liu, Meng Cao, Su Lin Blodgett, Jackie Chi Kit Cheung, Alexandra Olteanu, Adam Trischler
Findings of EMNLP 2023 | December 2023
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann (HE/HIM), Hanna Wallach
ACL 2023 | July 2023
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models
Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin Blodgett
Findings of ACL 2023 | July 2023
Can Workers Meaningfully Consent To Workplace Wellbeing Technologies?
Shreya Chowdhary, Anna Kawakami, Mary L. Gray, Jina Suh, Alexandra Olteanu, Koustuv Saha
2023 ACM Conference on Fairness, Accountability, and Transparency | June 2023
The Kitmus Test: Evaluating Knowledge Integration From Multiple Sources In Natural Language Understanding Systems
Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung
ACL 2023 | May 2023
fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
Steven Moore, Q. Vera Liao, Hariharan Subramonyam
CHI 2023 | April 2023
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
Q. Vera Liao, Hariharan Subramonyam, Jennifer Wang, Jennifer Wortman Vaughan
2023 ACM CHI Conference on Human Factors in Computing Systems (CHI 2023) | April 2023
Powering an AI Chatbot with Expert Sourcing to Support Credible Information Access
Ziang Xiao, Q. Vera Liao, Michelle X. Zhou, Tyrone Grandison, Yunyao Li
IUI 2023 | March 2023
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging
Jared Katzman, Angelina Wang, Morgan Scheuerman, Su Lin Blodgett, Kristen Laird, Hanna Wallach, Solon Barocas
AAAI 2022 | February 2023
Generation Probabilities are Not Enough: Improving Error Highlighting for AI Code Suggestions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
NeurIPS Workshop on Human-Centered AI | October 2022
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, Amit Dhurandhar
AAAI Conference on Human Computation and Crowdsourcing | October 2022
Best Paper Award, Honorable Mention
Examining Responsibility and Deliberation in AI Impact Statements and Ethics Reviews
David Liu, Priyanka Nanayakkara, Sarah Ariyan Sakha, Grace Abuhamad, Su Lin Blodgett, Nicholas Diakopoulos, Jessica R. Hullman, Tina Eliassi-Rad
AIES 2022 | July 2022
How Different Groups Prioritize Ethical Values for Responsible AI
Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
FAccT 2022 | May 2022
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, Alexandra Olteanu
NAACL 2022 | May 2022
Designing for Responsible Trust in AI Systems: A Communication Perspective
Q. Vera Liao, S. Shyam Sundar
FAccT 2022 | April 2022
Preprint
Responsible Language Technologies: Foreseeing and Mitigating Harms
Su Lin Blodgett, Q. Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, Qian Yang
CHI 2022 | April 2022
ADEPT: An Adjective-Dependent Plausibility Task
Ali Emami, Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung
ACL-IJCNLP 2021 | August 2021
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
ACL-IJCNLP 2021 | August 2021
“I Can’t Reply with That”: Characterizing Problematic Email Reply Suggestions
Ronald Robertson, Alexandra Olteanu, Fernando Diaz, Milad Shokouhi, Peter Bailey
CHI Conference on Human Factors in Computing Systems (CHI ’21) | May 2021
Overcoming Failures of Imagination in AI Infused System Development and Deployment
Margarita Boyarskaya, Alexandra Olteanu, Kate Crawford
In the Navigating the Broader Impacts of AI Research Workshop at NeurIPS 2020 | December 2020
When Are Search Completion Suggestions Problematic?
Alexandra Olteanu, Fernando Diaz, Gabriella Kazai
Computer Supported Collaborative Work and Social Computing (CSCW) | August 2020
CSCW 2020 Honorable Mention
CSCW 2020 Honorable Mention
Operationalizing the Legal Principle of Data Minimization for Personalization
Asia J. Biega, Peter Potash, Hal Daumé III, Fernando Diaz, Michèle Finck
SIGIR 2020 | July 2020
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, Ben Carterette
Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) | July 2020
Best Long Research Paper Nominee
External Information Sharing on Health Forums: An Exploration
Dana Nguyen, Alexandra Olteanu, Emre Kiciman
Intl. Conf. on Web and Social Media (ICWSM) | June 2020
Towards Query Logs For Privacy Studies: On Deriving Search Queries From Questions
Asia J. Biega, Jana Schmidt, Rishiraj Saha Roy
ECIR 2020 | April 2020
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI
Michael Madaio, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach
CHI Conference on Human Factors in Computing Systems | March 2020
CHI 2020 Best Paper Award
When not to design, build, or deploy
Solon Barocas, Asia J. Biega, Benjamin Fish, Jędrzej Niklas, Luke Stark
2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) | January 2020
Overview of the TREC 2019 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, Sebastian Kohlmeier
TREC 2019 | November 2019
Gaps in Information Access in Social Networks
Benjamin Fish, Ashkan Bashardoust, danah boyd, Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian
The Web Conference 2019 | May 2019
-
Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, Jennifer Wortman Vaughan
ToCHI | August 2024
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Q. Vera Liao, Jennifer Wortman Vaughan
Harvard Data Science Review | February 2024
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Valerie Chen, Q. Vera Liao, Jennifer Wortman Vaughan, Gagan Bansal
Proc. ACM Hum.-Comput. Interact. | October 2023, Vol 7
Article 370
Sensing Wellbeing in the Workplace, Why and For Whom? Envisioning Impacts with Organizational Stakeholders
Anna Kawakami, Shreya Chowdhary, Shamsi T. Iqbal, Q. Liao, Alexandra Olteanu, J. Suh, Koustuv Saha, Alexandra Olteanu, Jina Suh
Proceedings of the ACM on Human-Computer Interaction | March 2023, Vol 7: pp. 1-33
Responsible computing during COVID-19 and beyond
Solon Barocas, Asia J. Biega, Margarita Boyarskaya, Kate Crawford, Hal Daumé III, Miro Dudík, Benjamin Fish, Mary L. Gray, Brent Hecht, Alexandra Olteanu, Forough Poursabzi-Sangdeh, Luke Stark, Jennifer Wortman Vaughan, Hanna Wallach, Marion Zepf
Communications of the ACM | July 2021, Vol 67(7): pp. 30-32
https://doi.org/10.1145/3466612
Measuring algorithmically infused societies
Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kiciman, Noshir Contractor, Tina Eliassi-Rad
Nature | June 2021, Vol 595: pp. 197-204
Privacy Norms and Preferences for Photos Posted Online
Roberto Hoyle, Luke Stark, Qatrunnada Ismail, David Crandall, Apu Kapadia, Denise Anthony
ACM Transactions on Computer-Human Interaction | March 2020
FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval
Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand
ACM SIGIR Forum | December 2019, Vol 53(2): pp. 20
Report on the First HIPstIR Workshop on the Future of Information Retrieval
Laura Dietz, Bhaskar Mitra, Jeremy Pickens, Hana Anber, Sandeep Avula, Asia J. Biega, Adrian Boteanu, Shubham Chatterjee, Jeff Dalton, Shiri Dori-Hacohen, John Foley, Henry Feild, Ben Gamari, Rosie Jones, Pallika Kanani, Sumanta Kashyapi, Widad Machmouchi, Matthew Mitsui, Steve Nole, Alexandre Tachard Passos, Jordan Ramsdell, Adam Roegiest, David Smith, Alessandro Sordoni
SIGIR Forum | December 2019, Vol 53(2): pp. 62-75
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Alexandra Olteanu, Carlos Castillo, Fernando Diaz, Emre Kiciman
Frontiers in Big Data | July 2019, Vol 2
-
AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
June 2023
-
On the Social and Technical Challenges of Web Search Autosuggestion Moderation
T. J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, Michael Golebiewski
July 2020