AI-Assisted Systematization for Evaluating GenAI Systems
Dhruv Agarwal, Emily Sheng, Chad Atalla, Jean Garcia-Gathright, Hussein Mozannar, Hannah Washington, Alexandra Chouldechova, Solon Barocas, Hanna Wallach
May 2026
Jean Garcia-Gathright
Principal Applied Scientist
Publications
-
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Taxonomizing Representational Harms using Speech Act Theory
Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
March 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
-
-
A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
December 2024
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
-
-
AI-Assisted Systematization for Evaluating GenAI Systems
Dhruv Agarwal, Emily Sheng, Chad Atalla, Jean Garcia-Gathright, Hussein Mozannar, Hannah Washington, Alexandra Chouldechova, Solon Barocas, Hanna Wallach
May 2026
-
Taxonomizing Representational Harms using Speech Act Theory
Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
March 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
December 2024
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
-
-
AI-Assisted Systematization for Evaluating GenAI Systems
Dhruv Agarwal, Emily Sheng, Chad Atalla, Jean Garcia-Gathright, Hussein Mozannar, Hannah Washington, Alexandra Chouldechova, Solon Barocas, Hanna Wallach
May 2026
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Taxonomizing Representational Harms using Speech Act Theory
Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
March 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-
A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
December 2024
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
-
-
AI-Assisted Systematization for Evaluating GenAI Systems
Dhruv Agarwal, Emily Sheng, Chad Atalla, Jean Garcia-Gathright, Hussein Mozannar, Hannah Washington, Alexandra Chouldechova, Solon Barocas, Hanna Wallach
May 2026
-
Taxonomizing Representational Harms using Speech Act Theory
Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alex Chouldechova, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
March 2025
-
A Shared Standard for Valid Measurement of Generative AI Systems’ Capabilities, Risks, and Impacts
Alex Chouldechova, Chad Atalla, Solon Barocas, A. Feder Cooper, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Matthew Vogel, Hannah Washington, Hanna Wallach
December 2024
-
Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
November 2024
-
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, Nick Pangakis, A. Feder Cooper, Angelina Wang, Solon Barocas, Alex Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
November 2024
-
-
-
Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
Emma Harvey, Emily Sheng, Su Lin Blodgett, Alex Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, Hanna Wallach
Findings of the Association for Computational Linguistics: ACL 2025 | July 2025
-
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alex Chouldechova, Emily Corvi, Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nick Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Hannah Washington, Abigail Z. Jacobs
ICML 2025 | January 2025
-