Besmira Nushi

EUREKA: Evaluating and Understanding Large Foundation Models

Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey, Safoora Yousefi

MSR-TR-2024-33 | September 2024

Published by Microsoft

Preprint Github

Understanding Information Storage and Transfer in Multi-modal Large Language Models

Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, S. Feizi, Daniela Massiceti

NeurIPS 2024 | June 2024

DOI Publication Publication

KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval

Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Peshawaria, Ranjita Naik, Besmira Nushi

October 2023

DOI Publication Video

Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models

Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, Besmira Nushi

September 2023

DOI Publication Video

Social Biases through the Text-to-Image Generation Lens

Ranjita Naik, Besmira Nushi

AIES 2023 | August 2023

Publication Video

Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning

Yu Yang, Besmira Nushi, Hamid Palangi, Baharan Mirzasoleiman

ICML 2023 | June 2023

Publication

Benchmarking Spatial Relationships in Text-to-Image Generation

Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, Yezhou Yang

MSR-TR-2023-44 | May 2023

Published by Microsoft

Publication Github

Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging

Riccardo Fogliato, Shreya Chappidi, Matthew Lungren, Michael Fitzke, Mark Parkinson, Diane Wilson, Paul Fisher, Eric Horvitz, Kori Inkpen, Besmira Nushi

FAccT 2022 | May 2022

Publication

HINT: Integration Testing for AI-based Features with Humans in the Loop

Quanze Chen, Tobias Schnabel, Besmira Nushi, Saleema Amershi

International Conference on Intelligent User Interfaces | March 2022

Publication Project

Investigations of Performance and Bias in Human-AI Teamwork in Hiring

Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, Ece Kamar

AAAI 2022 | February 2022

Publication Download

Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs

Shruthi Bannur, Ozan Oktay, Melanie Bernhardt, Anton Schwaighofer, Rajesh Jena, Besmira Nushi, Sharan Wadhwani, Aditya Nori, Kal Natarajan, Shazad Ashraf, Javier Alvarez-Valle, Daniel Coelho de Castro

ICML 2021 Workshop on Interpretable Machine Learning in Healthcare | July 2021

PDF Preprint Project Project Project Project

Understanding Failures of Deep Networks via Robust Feature Extraction

Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, Eric Horvitz

CVPR 2021 | June 2021

Publication Project

Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld

CHI | May 2021

Publication

Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork

Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, Dan Weld

AAAI 2021 | February 2021

Publication

An Empirical Analysis of Backward Compatibility in Machine Learning Systems

Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, Eric Horvitz

KDD | August 2020

Publication PDF Github Project

SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions

Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, Ece Kamar

CVPR 2020 | June 2020

Publication Download Project

Characterizing Search-Engine Traffic to Internet Research Agency Web Properties

Alexander Spangher, Gireeja Ranade, Besmira Nushi, Adam Fourney, Eric Horvitz

The Web Conference (TheWebConf '20) | April 2020

Publication

Metareasoning in Modular Software Systems: On-the-Fly Configuration using Reinforcement Learning with Rich Contextual Representations

Aditya Modi, Debadeepta Dey, Alekh Agarwal, Adith Swaminathan, Besmira Nushi, Sean Andrist, Eric Horvitz

The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) | February 2020

PDF Publication

Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance

Gagan Bansal, Besmira Nushi, Ece Kamar, Walter Lasecki, Dan Weld, Eric Horvitz

HCOMP 2019 | October 2019

PDF

What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring

Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, Siddharth Suri, Ece Kamar

7th Conference on Human Computation and Crowdsourcing (HCOMP 2019) | September 2019

Publication PDF

Software Engineering for Machine Learning: A Case Study

Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLIne, Harald Gall, Ece Kamar, Nachi Nagappan, Besmira Nushi, Tom Zimmermann

International Conference on Software Engineering (ICSE 2019) - Software Engineering in Practice track | May 2019

ICSE 2019 Best Paper Award

PDF

Guidelines for Human-AI Interaction

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz

CHI 2019 | May 2019

CHI 2019 Honorable Mention Award

PDF Video Project Project

Overcoming Blind Spots in the Real World: Leveraging Complementary Abilities for Joint Execution

Ramya Ramakrishnan, Ece Kamar, Besmira Nushi, Debadeepta Dey, Julie Shah, Eric Horvitz

AAAI Conference on Artificial Intelligence | January 2019

Publication

Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff

Gagan Bansal, Besmira Nushi, Ece Kamar, Dan Weld, Walter Lasecki, Eric Horvitz

AAAI Conference on Artificial Intelligence | January 2019

Publication Github Project

Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure

Besmira Nushi, Ece Kamar, Eric Horvitz

HCOMP 2018 | July 2018

Publication Github Project

On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems.

Besmira Nushi, Ece Kamar, Eric Horvitz, Donald Kossmann

AAAI 2017 | January 2017

Publication

Quality Control and Optimization for Hybrid Crowd-Machine Learning Systems

Besmira Nushi

December 2016

December 2016

Publication

On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems

Besmira Nushi, Ece Kamar, Eric Horvitz, Donald Kossmann

Thirty-First AAAI Conference on Artificial Intelligence | November 2016

Publication Publication

Learning and Feature Selection under Budget Constraints in Crowdsourcing.

Besmira Nushi, Adish Singla, Andreas Krause, Donald Kossmann

HCOMP 2016 | January 2016

Publication

Crowd-STAR: A Social Task Routing Framework for Online Communities.

Besmira Nushi, Omar Alonso, Martin Hentschel, Vasileios Kandylas

ICWE 2015 | November 2015

Publication

Crowd Access Path Optimization: Diversity matters.

Besmira Nushi, Adish Singla, Anja Gruenheid, Erfan Zamanian, Andreas Krause, Donald Kossmann

HCOMP 2015 | January 2015

Publication

When is A = B?

Donald Kossmann, Anja Gruenheid, Besmira Nushi

Bulletin of the EATCS | November 2013

Publication

Uncertain time-series similarity: Return to the basics.

Michele Dallachiesa, Besmira Nushi, Katsiaryna Mirylenka, Themis Palpanas

Very Large Databases | November 2012

Publication Publication

Similarity matching for uncertain time series: analytical and experimental comparison.

Michele Dallachiesa, Besmira Nushi, Katsiaryna Mirylenka, Themis Palpanas

Proceedings of the 2nd ACM SIGSPATIAL International Workshop on Querying and Mining Uncertain Spatio-Temporal Data | November 2011

Publication

EUREKA: Evaluating and Understanding Large Foundation Models

Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey, Safoora Yousefi

MSR-TR-2024-33 | September 2024

Published by Microsoft

Preprint Github

Understanding Information Storage and Transfer in Multi-modal Large Language Models

Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, S. Feizi, Daniela Massiceti

NeurIPS 2024 | June 2024

DOI Publication Publication

KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval

Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Peshawaria, Ranjita Naik, Besmira Nushi

October 2023

DOI Publication Video

Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models

Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, Besmira Nushi

September 2023

DOI Publication Video

Social Biases through the Text-to-Image Generation Lens

Ranjita Naik, Besmira Nushi

AIES 2023 | August 2023

Publication Video

Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning

Yu Yang, Besmira Nushi, Hamid Palangi, Baharan Mirzasoleiman

ICML 2023 | June 2023

Publication

Benchmarking Spatial Relationships in Text-to-Image Generation

Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, Yezhou Yang

MSR-TR-2023-44 | May 2023

Published by Microsoft

Publication Github

Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging

Riccardo Fogliato, Shreya Chappidi, Matthew Lungren, Michael Fitzke, Mark Parkinson, Diane Wilson, Paul Fisher, Eric Horvitz, Kori Inkpen, Besmira Nushi

FAccT 2022 | May 2022

Publication

HINT: Integration Testing for AI-based Features with Humans in the Loop

Quanze Chen, Tobias Schnabel, Besmira Nushi, Saleema Amershi

International Conference on Intelligent User Interfaces | March 2022

Publication Project

Investigations of Performance and Bias in Human-AI Teamwork in Hiring

Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, Ece Kamar

AAAI 2022 | February 2022

Publication Download

Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs

Shruthi Bannur, Ozan Oktay, Melanie Bernhardt, Anton Schwaighofer, Rajesh Jena, Besmira Nushi, Sharan Wadhwani, Aditya Nori, Kal Natarajan, Shazad Ashraf, Javier Alvarez-Valle, Daniel Coelho de Castro

ICML 2021 Workshop on Interpretable Machine Learning in Healthcare | July 2021

PDF Preprint Project Project Project Project

Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld

CHI | May 2021

Publication

Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork

Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, Dan Weld

AAAI 2021 | February 2021

Publication

An Empirical Analysis of Backward Compatibility in Machine Learning Systems

Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, Eric Horvitz

KDD | August 2020

Publication PDF Github Project

SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions

Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, Ece Kamar

CVPR 2020 | June 2020

Publication Download Project

Metareasoning in Modular Software Systems: On-the-Fly Configuration using Reinforcement Learning with Rich Contextual Representations

Aditya Modi, Debadeepta Dey, Alekh Agarwal, Adith Swaminathan, Besmira Nushi, Sean Andrist, Eric Horvitz

The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) | February 2020

PDF Publication

Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance

Gagan Bansal, Besmira Nushi, Ece Kamar, Walter Lasecki, Dan Weld, Eric Horvitz

HCOMP 2019 | October 2019

PDF

What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring

Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, Siddharth Suri, Ece Kamar

7th Conference on Human Computation and Crowdsourcing (HCOMP 2019) | September 2019

Publication PDF

Software Engineering for Machine Learning: A Case Study

Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLIne, Harald Gall, Ece Kamar, Nachi Nagappan, Besmira Nushi, Tom Zimmermann

International Conference on Software Engineering (ICSE 2019) - Software Engineering in Practice track | May 2019

ICSE 2019 Best Paper Award

PDF

Guidelines for Human-AI Interaction

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz

CHI 2019 | May 2019

CHI 2019 Honorable Mention Award

PDF Video Project Project

Overcoming Blind Spots in the Real World: Leveraging Complementary Abilities for Joint Execution

Ramya Ramakrishnan, Ece Kamar, Besmira Nushi, Debadeepta Dey, Julie Shah, Eric Horvitz

AAAI Conference on Artificial Intelligence | January 2019

Publication

Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff

Gagan Bansal, Besmira Nushi, Ece Kamar, Dan Weld, Walter Lasecki, Eric Horvitz

AAAI Conference on Artificial Intelligence | January 2019

Publication Github Project

Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure

Besmira Nushi, Ece Kamar, Eric Horvitz

HCOMP 2018 | July 2018

Publication Github Project

On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems.

Besmira Nushi, Ece Kamar, Eric Horvitz, Donald Kossmann

AAAI 2017 | January 2017

Publication

Quality Control and Optimization for Hybrid Crowd-Machine Learning Systems

Besmira Nushi

December 2016

December 2016

Publication

On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems

Besmira Nushi, Ece Kamar, Eric Horvitz, Donald Kossmann

Thirty-First AAAI Conference on Artificial Intelligence | November 2016

Publication Publication

Learning and Feature Selection under Budget Constraints in Crowdsourcing.

Besmira Nushi, Adish Singla, Andreas Krause, Donald Kossmann

HCOMP 2016 | January 2016

Publication

Crowd Access Path Optimization: Diversity matters.

Besmira Nushi, Adish Singla, Anja Gruenheid, Erfan Zamanian, Andreas Krause, Donald Kossmann

HCOMP 2015 | January 2015

Publication

EUREKA: Evaluating and Understanding Large Foundation Models

Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey, Safoora Yousefi

MSR-TR-2024-33 | September 2024

Published by Microsoft

Preprint Github

Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs

Shruthi Bannur, Ozan Oktay, Melanie Bernhardt, Anton Schwaighofer, Rajesh Jena, Besmira Nushi, Sharan Wadhwani, Aditya Nori, Kal Natarajan, Shazad Ashraf, Javier Alvarez-Valle, Daniel Coelho de Castro

ICML 2021 Workshop on Interpretable Machine Learning in Healthcare | July 2021

PDF Preprint Project Project Project Project

Understanding Failures of Deep Networks via Robust Feature Extraction

Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, Eric Horvitz

CVPR 2021 | June 2021

Publication Project

On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems.

Besmira Nushi, Ece Kamar, Eric Horvitz, Donald Kossmann

AAAI 2017 | January 2017

Publication

EUREKA: Evaluating and Understanding Large Foundation Models

Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey, Safoora Yousefi

MSR-TR-2024-33 | September 2024

Published by Microsoft

Preprint Github

KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval

Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Peshawaria, Ranjita Naik, Besmira Nushi

October 2023

DOI Publication Video

Characterizing Search-Engine Traffic to Internet Research Agency Web Properties

Alexander Spangher, Gireeja Ranade, Besmira Nushi, Adam Fourney, Eric Horvitz

The Web Conference (TheWebConf '20) | April 2020

Publication

Crowd-STAR: A Social Task Routing Framework for Online Communities.

Besmira Nushi, Omar Alonso, Martin Hentschel, Vasileios Kandylas

ICWE 2015 | November 2015

Publication

Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging

Riccardo Fogliato, Shreya Chappidi, Matthew Lungren, Michael Fitzke, Mark Parkinson, Diane Wilson, Paul Fisher, Eric Horvitz, Kori Inkpen, Besmira Nushi

FAccT 2022 | May 2022

Publication

HINT: Integration Testing for AI-based Features with Humans in the Loop

Quanze Chen, Tobias Schnabel, Besmira Nushi, Saleema Amershi

International Conference on Intelligent User Interfaces | March 2022

Publication Project

Investigations of Performance and Bias in Human-AI Teamwork in Hiring

Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, Ece Kamar

AAAI 2022 | February 2022

Publication Download

Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld

CHI | May 2021

Publication

Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance

Gagan Bansal, Besmira Nushi, Ece Kamar, Walter Lasecki, Dan Weld, Eric Horvitz

HCOMP 2019 | October 2019

PDF

What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring

Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, Siddharth Suri, Ece Kamar

7th Conference on Human Computation and Crowdsourcing (HCOMP 2019) | September 2019

Publication PDF

Guidelines for Human-AI Interaction

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz

CHI 2019 | May 2019

CHI 2019 Honorable Mention Award

PDF Video Project Project

Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff

Gagan Bansal, Besmira Nushi, Ece Kamar, Dan Weld, Walter Lasecki, Eric Horvitz

AAAI Conference on Artificial Intelligence | January 2019

Publication Github Project

Quality Control and Optimization for Hybrid Crowd-Machine Learning Systems

Besmira Nushi

December 2016

December 2016

Publication

Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging

Riccardo Fogliato, Shreya Chappidi, Matthew Lungren, Michael Fitzke, Mark Parkinson, Diane Wilson, Paul Fisher, Eric Horvitz, Kori Inkpen, Besmira Nushi

FAccT 2022 | May 2022

Publication

What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring

Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, Siddharth Suri, Ece Kamar

7th Conference on Human Computation and Crowdsourcing (HCOMP 2019) | September 2019

Publication PDF

Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs

Shruthi Bannur, Ozan Oktay, Melanie Bernhardt, Anton Schwaighofer, Rajesh Jena, Besmira Nushi, Sharan Wadhwani, Aditya Nori, Kal Natarajan, Shazad Ashraf, Javier Alvarez-Valle, Daniel Coelho de Castro

ICML 2021 Workshop on Interpretable Machine Learning in Healthcare | July 2021

PDF Preprint Project Project Project Project

Software Engineering for Machine Learning: A Case Study

Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLIne, Harald Gall, Ece Kamar, Nachi Nagappan, Besmira Nushi, Tom Zimmermann

International Conference on Software Engineering (ICSE 2019) - Software Engineering in Practice track | May 2019

ICSE 2019 Best Paper Award

PDF

When is A = B?

Donald Kossmann, Anja Gruenheid, Besmira Nushi

Bulletin of the EATCS | November 2013

Publication

Uncertain time-series similarity: Return to the basics.

Michele Dallachiesa, Besmira Nushi, Katsiaryna Mirylenka, Themis Palpanas

Very Large Databases | November 2012

Publication Publication

When is A = B?

Donald Kossmann, Anja Gruenheid, Besmira Nushi

Bulletin of the EATCS | November 2013

Publication

Uncertain time-series similarity: Return to the basics.

Michele Dallachiesa, Besmira Nushi, Katsiaryna Mirylenka, Themis Palpanas

Very Large Databases | November 2012

Publication Publication

Similarity matching for uncertain time series: analytical and experimental comparison.

Michele Dallachiesa, Besmira Nushi, Katsiaryna Mirylenka, Themis Palpanas

Proceedings of the 2nd ACM SIGSPATIAL International Workshop on Querying and Mining Uncertain Spatio-Temporal Data | November 2011

Publication

Understanding Information Storage and Transfer in Multi-modal Large Language Models

Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, S. Feizi, Daniela Massiceti

NeurIPS 2024 | June 2024

DOI Publication Publication

Social Biases through the Text-to-Image Generation Lens

Ranjita Naik, Besmira Nushi

AIES 2023 | August 2023

Publication Video

Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning

Yu Yang, Besmira Nushi, Hamid Palangi, Baharan Mirzasoleiman

ICML 2023 | June 2023

Publication

Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging

Riccardo Fogliato, Shreya Chappidi, Matthew Lungren, Michael Fitzke, Mark Parkinson, Diane Wilson, Paul Fisher, Eric Horvitz, Kori Inkpen, Besmira Nushi

FAccT 2022 | May 2022

Publication

HINT: Integration Testing for AI-based Features with Humans in the Loop

Quanze Chen, Tobias Schnabel, Besmira Nushi, Saleema Amershi

International Conference on Intelligent User Interfaces | March 2022

Publication Project

Investigations of Performance and Bias in Human-AI Teamwork in Hiring

Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, Ece Kamar

AAAI 2022 | February 2022

Publication Download

Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs

Shruthi Bannur, Ozan Oktay, Melanie Bernhardt, Anton Schwaighofer, Rajesh Jena, Besmira Nushi, Sharan Wadhwani, Aditya Nori, Kal Natarajan, Shazad Ashraf, Javier Alvarez-Valle, Daniel Coelho de Castro

ICML 2021 Workshop on Interpretable Machine Learning in Healthcare | July 2021

PDF Preprint Project Project Project Project

Understanding Failures of Deep Networks via Robust Feature Extraction

Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, Eric Horvitz

CVPR 2021 | June 2021

Publication Project

Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld

CHI | May 2021

Publication

Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork

Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, Dan Weld

AAAI 2021 | February 2021

Publication

An Empirical Analysis of Backward Compatibility in Machine Learning Systems

Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, Eric Horvitz

KDD | August 2020

Publication PDF Github Project

SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions

Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, Ece Kamar

CVPR 2020 | June 2020

Publication Download Project

Characterizing Search-Engine Traffic to Internet Research Agency Web Properties

Alexander Spangher, Gireeja Ranade, Besmira Nushi, Adam Fourney, Eric Horvitz

The Web Conference (TheWebConf '20) | April 2020

Publication

Metareasoning in Modular Software Systems: On-the-Fly Configuration using Reinforcement Learning with Rich Contextual Representations

Aditya Modi, Debadeepta Dey, Alekh Agarwal, Adith Swaminathan, Besmira Nushi, Sean Andrist, Eric Horvitz

The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) | February 2020

PDF Publication

Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance

Gagan Bansal, Besmira Nushi, Ece Kamar, Walter Lasecki, Dan Weld, Eric Horvitz

HCOMP 2019 | October 2019

PDF

What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring

Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, Siddharth Suri, Ece Kamar

7th Conference on Human Computation and Crowdsourcing (HCOMP 2019) | September 2019

Publication PDF

Software Engineering for Machine Learning: A Case Study

Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLIne, Harald Gall, Ece Kamar, Nachi Nagappan, Besmira Nushi, Tom Zimmermann

International Conference on Software Engineering (ICSE 2019) - Software Engineering in Practice track | May 2019

ICSE 2019 Best Paper Award

PDF

Guidelines for Human-AI Interaction

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz

CHI 2019 | May 2019

CHI 2019 Honorable Mention Award

PDF Video Project Project

Overcoming Blind Spots in the Real World: Leveraging Complementary Abilities for Joint Execution

Ramya Ramakrishnan, Ece Kamar, Besmira Nushi, Debadeepta Dey, Julie Shah, Eric Horvitz

AAAI Conference on Artificial Intelligence | January 2019

Publication

Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff

Gagan Bansal, Besmira Nushi, Ece Kamar, Dan Weld, Walter Lasecki, Eric Horvitz

AAAI Conference on Artificial Intelligence | January 2019

Publication Github Project

Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure

Besmira Nushi, Ece Kamar, Eric Horvitz

HCOMP 2018 | July 2018

Publication Github Project

On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems.

Besmira Nushi, Ece Kamar, Eric Horvitz, Donald Kossmann

AAAI 2017 | January 2017

Publication

On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems

Besmira Nushi, Ece Kamar, Eric Horvitz, Donald Kossmann

Thirty-First AAAI Conference on Artificial Intelligence | November 2016

Publication Publication

Learning and Feature Selection under Budget Constraints in Crowdsourcing.

Besmira Nushi, Adish Singla, Andreas Krause, Donald Kossmann

HCOMP 2016 | January 2016

Publication

Crowd-STAR: A Social Task Routing Framework for Online Communities.

Besmira Nushi, Omar Alonso, Martin Hentschel, Vasileios Kandylas

ICWE 2015 | November 2015

Publication

Crowd Access Path Optimization: Diversity matters.

Besmira Nushi, Adish Singla, Anja Gruenheid, Erfan Zamanian, Andreas Krause, Donald Kossmann

HCOMP 2015 | January 2015

Publication

Similarity matching for uncertain time series: analytical and experimental comparison.

Michele Dallachiesa, Besmira Nushi, Katsiaryna Mirylenka, Themis Palpanas

Proceedings of the 2nd ACM SIGSPATIAL International Workshop on Querying and Mining Uncertain Spatio-Temporal Data | November 2011

Publication

EUREKA: Evaluating and Understanding Large Foundation Models

Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey, Safoora Yousefi

MSR-TR-2024-33 | September 2024

Published by Microsoft

Preprint Github

Benchmarking Spatial Relationships in Text-to-Image Generation

Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, Yezhou Yang

MSR-TR-2023-44 | May 2023

Published by Microsoft

Publication Github

KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval

Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Peshawaria, Ranjita Naik, Besmira Nushi

October 2023

DOI Publication Video

Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models

Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, Besmira Nushi

September 2023

DOI Publication Video

Quality Control and Optimization for Hybrid Crowd-Machine Learning Systems

Besmira Nushi

December 2016

December 2016

Publication

When is A = B?

Donald Kossmann, Anja Gruenheid, Besmira Nushi

Bulletin of the EATCS | November 2013

Publication

Uncertain time-series similarity: Return to the basics.

Michele Dallachiesa, Besmira Nushi, Katsiaryna Mirylenka, Themis Palpanas

Very Large Databases | November 2012

Publication Publication