Sunayana Sitaram

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

Bridging the Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs

Somnath Kumar, Vaibhav Balloli, Mercy Ranjit, Kabir Ahuja, Tanuja Ganu, Sunayana Sitaram, Kalika Bali, Akshay Nambi

May 2024

Preprint

PARIKSHA: A Scalable, Democratic, Transparent Evaluation Platform for Assessing Indic Large Language Models

Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Swami Manohar, Sunayana Sitaram

May 2024

Preprint Video Project

TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs

Nishanth Chandran, Sunayana Sitaram, Divya Gupta, Rahul Sharma, Kashish Mittal, Swami Manohar

February 2024

DOI Publication Publication Github

DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures

Agrima Seth, Sanchit Ahuja, Kalika Bali, Sunayana Sitaram

February 2024

DOI Publication Publication Github

CultureLLM: Incorporating Cultural Differences into Large Language Models

Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, Xing Xie

NeurIPS 2024 | February 2024

DOI Publication Publication

MEGA: Multilingual Evaluation of Generative AI

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram

EMNLP 2023 | October 2023

Publication Github Project Project Project

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?

Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, M. Choudhury, Kalika Bali, Sunayana Sitaram

2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA) | September 2023, pp. 1051-1070

DOI Publication Publication PDF

A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models

Krithika Ramesh, Arnav Chavan, Shrey Pandit, Sunayana Sitaram

ACL 2023 | July 2023

Publication Project

ACL 2023 Multilingual Models Tutorial Slides

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Barun Patra, Vishrav Chaudhary, Kabir Ahuja

July 2023

Publication Project

On Evaluating and Mitigating Gender Biases in Multilingual Settings

Aniket Vashishtha, Kabir Ahuja, Sunayana Sitaram

ACL 2023 | July 2023

Publication Project

LITMUS Predictor: An AI Assistant for Building Reliable, High-Performing and Fair Multilingual NLP Systems

Anirudh Srinivasan, Gauri Kholkar, Rahul Kejriwal, Tanuja Ganu, Sandipan Dandapat, Sunayana Sitaram, Balakrishnan Santhanam, Somak Aditya, Kalika Bali, Monojit Choudhury

Thirty-sixth AAAI Conference on Artificial Intelligence | February 2022

System Demonstration

Publication Video Github Project

A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies

A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, Almeida Jacqueline Toribio

2021 Meeting of the Association for Computational Linguistics | August 2021

DOI PDF Publication Publication Publication Publication Publication Project

Multilingual and code-switching ASR challenges for low resource Indian languages

Anuj Diwan, Rakesh Vaideeswaran, Sanket Shah, Ankita Singh, Srinivasa Raghavan, Shreya Khare, Vinit Unni, Saurabh Vyas, Akash Rajpuria, Chiranjeevi Yarra, Ashish Mittal, Prasanta Kumar Ghosh, Preethi Jyothi, Kalika Bali, Vivek Seshadri, Sunayana Sitaram, Samarth Bharadwaj, Jai Nanavati, Raoul Nanavati, Karthik Sankaranarayanan, Tejaswi Seeram, Basil Abraham

Interspeech 2021 | August 2021

Publication

A Case Study of Efficacy and Challenges in Practical Human-in-Loop Evaluation of NLP Systems Using Checklist

Shaily Bhatt, Rahul Jain, Sandipan Dandapat, Sunayana Sitaram

Workshop on Human Evaluation of NLP Systems | April 2021

Publication Project

GCM: A Toolkit for Generating Synthetic Code-mixed Text

Mohd Sanad Zaki Rizvi, Anirudh Srinivasan, Tanuja Ganu, Monojit Choudhury, Sunayana Sitaram

2021 Conference of the European Chapter of the Association for Computational Linguistics | April 2021

DOI Publication Publication Project

GLUECoS: An Evaluation Benchmark for Code-Switched NLP

Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, Monojit Choudhury

ACL 2020 | July 2020

Publication Github

GLUECoS: An Evaluation Benchmark for Code-Switched NLP

Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, Monojit Choudhury

Association for Computational Linguistics (ACL) | June 2020

DOI Publication Project

Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition

Gurunath Reddy Madhumani, Sanket Shah, Basil Abraham, Vikas Joshi, Sunayana Sitaram

arXiv preprint arXiv:2006.05257 | June 2020

PDF Publication Project

A New Dataset for Natural Language Inference from Code-mixed Conversations

Simran Khanuja, Sandipan Dandapat, Sunayana Sitaram, Monojit Choudhury

Workshop on Computational Approaches to Code Switching | May 2020

PDF Publication Project

Crowdsourcing Speech Data for Low-Resource Languages from Low-Income Workers

Basil Abraham, Danish Goel, Divya Siddarth, Kalika Bali, Manu Chopra, Monojit Choudhury, Pratik Joshi, Preethi Jyothi, Sunayana Sitaram, Vivek Seshadri

Language Resources and Evaluation Conference (LREC) | May 2020

PDF Publication Project Project

Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities.

Pratik Joshi, Christain Barnes, Sebastin Santy, Simran Khanuja, Sanket Shah, Anirudh Srinivasan, Satwik Bhattamishra, Sunayana Sitaram, Monojit Choudhury, Kalika Bali

International Conference on Natural Language Processing (ICON) | December 2019

PDF Publication Project

Deploying Language Technologies for Underserved Communities

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Sebastin Santy

UNESCO International Conference on Language Technologies for all (LT4All) | December 2019

Publication Project

ELLORA: Enabling Low Resource Languages with Technology

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Vivek Seshadri

UNESCO International Conference on Language Technologies for all (LT4All) | December 2019

PDF Publication Video Github Project

CoSSAT: Code-Switched Speech Annotation Tool

Sanket Shah, Pratik Joshi, Sunayana Sitaram (susitara), Sebastin Santy

EMNLP AnnoNLP workshop | November 2019

Publication Project

Using Monolingual Speech Recognition for Spoken Term Detection in Code-switched Hindi-English Speech

Sanket Shah, Sunayana Sitaram

International Conference on Data Mining | November 2019

DOI Publication Project

End-to-End ASR for Code-switched Hindi-English Speech

Brij Mohan Lal Srivastava, Basil Abraham, Sunayana Sitaram, Rupesh K. Mehta, Preethi Jyothi

arXiv preprint arXiv:1906.09426 | June 2019

Publication PDF Project

A Survey of Code-switched Speech and Language Processing

Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, Alan W. Black

March 2019

Publication Project

Word Embeddings for Code-Mixed Language Processing

Adithya Pratapa, Monojit Choudhury, Sunayana Sitaram

Empirical Methods in Natural Language Processing (EMNLP) | November 2018

Publication Project

Effect of TTS Generated Audio on OOV Detection and Word Error Rate in ASR for Low-resource Languages

Savitha Murthy, Dinkar Sitaram, Sunayana Sitaram

Interspeech 2018 | September 2018

Publication

Homophone Identification and Merging for Code-switched Speech Recognition

Brij Mohan Lal Srivastava, Sunayana Sitaram

Interspeech 2018 | September 2018

Publication Project

Interspeech 2018 Low Resource Automatic Speech Recognition Challenge for Indian Languages.

Brij Mohan Lal Srivastava, Sunayana Sitaram, Rupesh Kumar Mehta, Krishna Doss Mohan, Pallavi Matani, Sandeepkumar Satpal, Kalika Bali, Radhakrishnan Srikanth, Niranjan Nayak

Workshop Spoken Language Technologies for Under-resourced Languages | August 2018

DOI Publication Publication Project

Interspeech 2018 Low Resource Automatic Speech Recognition Challenge for Indian Languages

Brij Mohan Lal Srivastava, Sunayana Sitaram, Kalika Bali, Rupesh Kumar Mehta, Krishna Doss Mohan, Pallavi Matani, Sandeepkumar Satpal, Kalika Bali, Radhakrishnan Srikanth, Niranjan Nayak

SLTU | August 2018

Publication Project

Automatic Detection of Code-switching Style from Acoustics

SaiKrishna Rallabandi, Sunayana Sitaram, Alan W Black

Workshop on Computational Approaches to Linguistic Code Switching, 2018 | July 2018

Publication Project

Phone Merging For Code-switched Speech Recognition

Sunit Sivasankaran, Brij Mohan Lal Srivastava, Sunayana Sitaram, Kalika Bali, Monojit Choudhury

Workshop on Computational Approaches to Linguistic Code Switching, 2018 | July 2018

Publication Project

Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data

Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, Kalika Bali

Proceedings of ACL 2018 | July 2018

Publication Project

Discovering Canonical Indian English Accents: A Crowdsourcing-based Approach

Sunayana Sitaram, Varun Manjunath, Varun Bharadwaj, Monojit Choudhury, Kalika Bali, Michael Tjalve

LREC 2018 | May 2018

Publication

Discovering Canonical Indian English Accents: A Crowdsourcing-based Approach

Sunayana Sitaram, Varun Manjunath, Varun Bharadwaj, Monojit Choudhury, Kalika Bali, Michael Tjalve

Language Resources and Evaluation (LREC) | April 2018

PDF Publication Publication Project

Curriculum Design for Code-switching: Experiments with Language Identification and Language Modeling with Deep Neural Networks

Ashutosh Baheti, Sunayana Sitaram, Monojit Choudhury, Kalika Bali

ICON 2017 | December 2017

Publication Project

Speech Synthesis for Mixed-Language Navigation Instructions

Khythiraghavi Chandu, Sai Krishna Rallabandi, Sunayana Sitaram, Alan W Black

Interspeech 2017 | August 2017

Publication Project

Experiments with Cross-lingual Systems for Synthesis of Code-Mixed Text

Sunayana Sitaram, Sai Krishna Rallabandi, Shruti Rijhwani, Alan W Black

Speech Synthesis Workshop 8 | August 2016

Publication Project

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

Bridging the Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs

Somnath Kumar, Vaibhav Balloli, Mercy Ranjit, Kabir Ahuja, Tanuja Ganu, Sunayana Sitaram, Kalika Bali, Akshay Nambi

May 2024

Preprint

PARIKSHA: A Scalable, Democratic, Transparent Evaluation Platform for Assessing Indic Large Language Models

Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Swami Manohar, Sunayana Sitaram

May 2024

Preprint Video Project

DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures

Agrima Seth, Sanchit Ahuja, Kalika Bali, Sunayana Sitaram

February 2024

DOI Publication Publication Github

CultureLLM: Incorporating Cultural Differences into Large Language Models

Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, Xing Xie

NeurIPS 2024 | February 2024

DOI Publication Publication

MEGA: Multilingual Evaluation of Generative AI

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram

EMNLP 2023 | October 2023

Publication Github Project Project Project

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?

Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, M. Choudhury, Kalika Bali, Sunayana Sitaram

2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA) | September 2023, pp. 1051-1070

DOI Publication Publication PDF

LITMUS Predictor: An AI Assistant for Building Reliable, High-Performing and Fair Multilingual NLP Systems

Anirudh Srinivasan, Gauri Kholkar, Rahul Kejriwal, Tanuja Ganu, Sandipan Dandapat, Sunayana Sitaram, Balakrishnan Santhanam, Somak Aditya, Kalika Bali, Monojit Choudhury

Thirty-sixth AAAI Conference on Artificial Intelligence | February 2022

System Demonstration

Publication Video Github Project

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs

Nishanth Chandran, Sunayana Sitaram, Divya Gupta, Rahul Sharma, Kashish Mittal, Swami Manohar

February 2024

DOI Publication Publication Github

PARIKSHA: A Scalable, Democratic, Transparent Evaluation Platform for Assessing Indic Large Language Models

Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Swami Manohar, Sunayana Sitaram

May 2024

Preprint Video Project

DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures

Agrima Seth, Sanchit Ahuja, Kalika Bali, Sunayana Sitaram

February 2024

DOI Publication Publication Github

MEGA: Multilingual Evaluation of Generative AI

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram

EMNLP 2023 | October 2023

Publication Github Project Project Project

ACL 2023 Multilingual Models Tutorial Slides

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Barun Patra, Vishrav Chaudhary, Kabir Ahuja

July 2023

Publication Project

On Evaluating and Mitigating Gender Biases in Multilingual Settings

Aniket Vashishtha, Kabir Ahuja, Sunayana Sitaram

ACL 2023 | July 2023

Publication Project

A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models

Krithika Ramesh, Arnav Chavan, Shrey Pandit, Sunayana Sitaram

ACL 2023 | July 2023

Publication Project

LITMUS Predictor: An AI Assistant for Building Reliable, High-Performing and Fair Multilingual NLP Systems

Anirudh Srinivasan, Gauri Kholkar, Rahul Kejriwal, Tanuja Ganu, Sandipan Dandapat, Sunayana Sitaram, Balakrishnan Santhanam, Somak Aditya, Kalika Bali, Monojit Choudhury

Thirty-sixth AAAI Conference on Artificial Intelligence | February 2022

System Demonstration

Publication Video Github Project

A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies

A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, Almeida Jacqueline Toribio

2021 Meeting of the Association for Computational Linguistics | August 2021

DOI PDF Publication Publication Publication Publication Publication Project

Multilingual and code-switching ASR challenges for low resource Indian languages

Anuj Diwan, Rakesh Vaideeswaran, Sanket Shah, Ankita Singh, Srinivasa Raghavan, Shreya Khare, Vinit Unni, Saurabh Vyas, Akash Rajpuria, Chiranjeevi Yarra, Ashish Mittal, Prasanta Kumar Ghosh, Preethi Jyothi, Kalika Bali, Vivek Seshadri, Sunayana Sitaram, Samarth Bharadwaj, Jai Nanavati, Raoul Nanavati, Karthik Sankaranarayanan, Tejaswi Seeram, Basil Abraham

Interspeech 2021 | August 2021

Publication

GCM: A Toolkit for Generating Synthetic Code-mixed Text

Mohd Sanad Zaki Rizvi, Anirudh Srinivasan, Tanuja Ganu, Monojit Choudhury, Sunayana Sitaram

2021 Conference of the European Chapter of the Association for Computational Linguistics | April 2021

DOI Publication Publication Project

A Case Study of Efficacy and Challenges in Practical Human-in-Loop Evaluation of NLP Systems Using Checklist

Shaily Bhatt, Rahul Jain, Sandipan Dandapat, Sunayana Sitaram

Workshop on Human Evaluation of NLP Systems | April 2021

Publication Project

GLUECoS: An Evaluation Benchmark for Code-Switched NLP

Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, Monojit Choudhury

ACL 2020 | July 2020

Publication Github

GLUECoS: An Evaluation Benchmark for Code-Switched NLP

Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, Monojit Choudhury

Association for Computational Linguistics (ACL) | June 2020

DOI Publication Project

Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition

Gurunath Reddy Madhumani, Sanket Shah, Basil Abraham, Vikas Joshi, Sunayana Sitaram

arXiv preprint arXiv:2006.05257 | June 2020

PDF Publication Project

A New Dataset for Natural Language Inference from Code-mixed Conversations

Simran Khanuja, Sandipan Dandapat, Sunayana Sitaram, Monojit Choudhury

Workshop on Computational Approaches to Code Switching | May 2020

PDF Publication Project

Crowdsourcing Speech Data for Low-Resource Languages from Low-Income Workers

Basil Abraham, Danish Goel, Divya Siddarth, Kalika Bali, Manu Chopra, Monojit Choudhury, Pratik Joshi, Preethi Jyothi, Sunayana Sitaram, Vivek Seshadri

Language Resources and Evaluation Conference (LREC) | May 2020

PDF Publication Project Project

Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities.

Pratik Joshi, Christain Barnes, Sebastin Santy, Simran Khanuja, Sanket Shah, Anirudh Srinivasan, Satwik Bhattamishra, Sunayana Sitaram, Monojit Choudhury, Kalika Bali

International Conference on Natural Language Processing (ICON) | December 2019

PDF Publication Project

Deploying Language Technologies for Underserved Communities

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Sebastin Santy

UNESCO International Conference on Language Technologies for all (LT4All) | December 2019

Publication Project

ELLORA: Enabling Low Resource Languages with Technology

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Vivek Seshadri

UNESCO International Conference on Language Technologies for all (LT4All) | December 2019

PDF Publication Video Github Project

Using Monolingual Speech Recognition for Spoken Term Detection in Code-switched Hindi-English Speech

Sanket Shah, Sunayana Sitaram

International Conference on Data Mining | November 2019

DOI Publication Project

CoSSAT: Code-Switched Speech Annotation Tool

Sanket Shah, Pratik Joshi, Sunayana Sitaram (susitara), Sebastin Santy

EMNLP AnnoNLP workshop | November 2019

Publication Project

End-to-End ASR for Code-switched Hindi-English Speech

Brij Mohan Lal Srivastava, Basil Abraham, Sunayana Sitaram, Rupesh K. Mehta, Preethi Jyothi

arXiv preprint arXiv:1906.09426 | June 2019

Publication PDF Project

A Survey of Code-switched Speech and Language Processing

Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, Alan W. Black

March 2019

Publication Project

Word Embeddings for Code-Mixed Language Processing

Adithya Pratapa, Monojit Choudhury, Sunayana Sitaram

Empirical Methods in Natural Language Processing (EMNLP) | November 2018

Publication Project

Effect of TTS Generated Audio on OOV Detection and Word Error Rate in ASR for Low-resource Languages

Savitha Murthy, Dinkar Sitaram, Sunayana Sitaram

Interspeech 2018 | September 2018

Publication

Homophone Identification and Merging for Code-switched Speech Recognition

Brij Mohan Lal Srivastava, Sunayana Sitaram

Interspeech 2018 | September 2018

Publication Project

Interspeech 2018 Low Resource Automatic Speech Recognition Challenge for Indian Languages.

Brij Mohan Lal Srivastava, Sunayana Sitaram, Rupesh Kumar Mehta, Krishna Doss Mohan, Pallavi Matani, Sandeepkumar Satpal, Kalika Bali, Radhakrishnan Srikanth, Niranjan Nayak

Workshop Spoken Language Technologies for Under-resourced Languages | August 2018

DOI Publication Publication Project

Interspeech 2018 Low Resource Automatic Speech Recognition Challenge for Indian Languages

Brij Mohan Lal Srivastava, Sunayana Sitaram, Kalika Bali, Rupesh Kumar Mehta, Krishna Doss Mohan, Pallavi Matani, Sandeepkumar Satpal, Kalika Bali, Radhakrishnan Srikanth, Niranjan Nayak

SLTU | August 2018

Publication Project

Automatic Detection of Code-switching Style from Acoustics

SaiKrishna Rallabandi, Sunayana Sitaram, Alan W Black

Workshop on Computational Approaches to Linguistic Code Switching, 2018 | July 2018

Publication Project

Phone Merging For Code-switched Speech Recognition

Sunit Sivasankaran, Brij Mohan Lal Srivastava, Sunayana Sitaram, Kalika Bali, Monojit Choudhury

Workshop on Computational Approaches to Linguistic Code Switching, 2018 | July 2018

Publication Project

Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data

Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, Kalika Bali

Proceedings of ACL 2018 | July 2018

Publication Project

Discovering Canonical Indian English Accents: A Crowdsourcing-based Approach

Sunayana Sitaram, Varun Manjunath, Varun Bharadwaj, Monojit Choudhury, Kalika Bali, Michael Tjalve

LREC 2018 | May 2018

Publication

Discovering Canonical Indian English Accents: A Crowdsourcing-based Approach

Sunayana Sitaram, Varun Manjunath, Varun Bharadwaj, Monojit Choudhury, Kalika Bali, Michael Tjalve

Language Resources and Evaluation (LREC) | April 2018

PDF Publication Publication Project

Curriculum Design for Code-switching: Experiments with Language Identification and Language Modeling with Deep Neural Networks

Ashutosh Baheti, Sunayana Sitaram, Monojit Choudhury, Kalika Bali

ICON 2017 | December 2017

Publication Project

Speech Synthesis for Mixed-Language Navigation Instructions

Khythiraghavi Chandu, Sai Krishna Rallabandi, Sunayana Sitaram, Alan W Black

Interspeech 2017 | August 2017

Publication Project

Experiments with Cross-lingual Systems for Synthesis of Code-Mixed Text

Sunayana Sitaram, Sai Krishna Rallabandi, Shruti Rijhwani, Alan W Black

Speech Synthesis Workshop 8 | August 2016

Publication Project

Multilingual and code-switching ASR challenges for low resource Indian languages

Anuj Diwan, Rakesh Vaideeswaran, Sanket Shah, Ankita Singh, Srinivasa Raghavan, Shreya Khare, Vinit Unni, Saurabh Vyas, Akash Rajpuria, Chiranjeevi Yarra, Ashish Mittal, Prasanta Kumar Ghosh, Preethi Jyothi, Kalika Bali, Vivek Seshadri, Sunayana Sitaram, Samarth Bharadwaj, Jai Nanavati, Raoul Nanavati, Karthik Sankaranarayanan, Tejaswi Seeram, Basil Abraham

Interspeech 2021 | August 2021

Publication

Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities.

Pratik Joshi, Christain Barnes, Sebastin Santy, Simran Khanuja, Sanket Shah, Anirudh Srinivasan, Satwik Bhattamishra, Sunayana Sitaram, Monojit Choudhury, Kalika Bali

International Conference on Natural Language Processing (ICON) | December 2019

PDF Publication Project

ELLORA: Enabling Low Resource Languages with Technology

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Vivek Seshadri

UNESCO International Conference on Language Technologies for all (LT4All) | December 2019

PDF Publication Video Github Project

Microsoft New Future of Work Report 2024

Jenna Butler, Mihaela Vorvoreanu, Rebecca Janssen, Abigail Sellen, Nicole Immorlica, Adam Troy, Advait Sarkar, Alex Farach, Alex Chouldechova, Alexandra Olteanu, Alexia Cambon, Arjun Radhakrishna, Asta Roseway, Ben Zorn, Brent Hecht, Daniel G. Goldstein, Dhruv Joshi, Ed Cutrell, Emre Kiciman, Gonzalo Ramos, Gustavo Soares, Hanna Wallach, Ian Drosos, Jack Williams (johnwilliams), Jacki O'Neill, Jake Hofman, Jaime Teevan, Javier Hernandez, Jennifer Wortman Vaughan, Jina Suh, John Tang, Justin Edwards, Kalika Bali, Kori Inkpen, Krishna Madhavan, Laylah Bulman, Leon Reicherts, Lev Tankelevitch, Longqi Yang, Martez Mott, Millicent Ochieng, Mercy Muchai, Nancy Baym, Najeeb Abdulhamid, Nicolai Marquardt, Ken Hinckley, Michael Bentley, Dave Brown, Hugo Romat, Nathalie Henry Riche, Samuel Maina, Shamsi Iqbal, Siân Lindley, Stephanie Nyairo, Su Lin Blodgett, Sumit Gulwani, Sunayana Sitaram, Vu Le

MSR-TR-2024-56 | December 2024

Published by Microsoft

Publication Project Project

CultureLLM: Incorporating Cultural Differences into Large Language Models

Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, Xing Xie

NeurIPS 2024 | February 2024

DOI Publication Publication

MEGA: Multilingual Evaluation of Generative AI

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram

EMNLP 2023 | October 2023

Publication Github Project Project Project

On Evaluating and Mitigating Gender Biases in Multilingual Settings

Aniket Vashishtha, Kabir Ahuja, Sunayana Sitaram

ACL 2023 | July 2023

Publication Project

A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models

Krithika Ramesh, Arnav Chavan, Shrey Pandit, Sunayana Sitaram

ACL 2023 | July 2023

Publication Project

LITMUS Predictor: An AI Assistant for Building Reliable, High-Performing and Fair Multilingual NLP Systems

Anirudh Srinivasan, Gauri Kholkar, Rahul Kejriwal, Tanuja Ganu, Sandipan Dandapat, Sunayana Sitaram, Balakrishnan Santhanam, Somak Aditya, Kalika Bali, Monojit Choudhury

Thirty-sixth AAAI Conference on Artificial Intelligence | February 2022

System Demonstration

Publication Video Github Project

A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies

A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, Almeida Jacqueline Toribio

2021 Meeting of the Association for Computational Linguistics | August 2021

DOI PDF Publication Publication Publication Publication Publication Project

Multilingual and code-switching ASR challenges for low resource Indian languages

Anuj Diwan, Rakesh Vaideeswaran, Sanket Shah, Ankita Singh, Srinivasa Raghavan, Shreya Khare, Vinit Unni, Saurabh Vyas, Akash Rajpuria, Chiranjeevi Yarra, Ashish Mittal, Prasanta Kumar Ghosh, Preethi Jyothi, Kalika Bali, Vivek Seshadri, Sunayana Sitaram, Samarth Bharadwaj, Jai Nanavati, Raoul Nanavati, Karthik Sankaranarayanan, Tejaswi Seeram, Basil Abraham

Interspeech 2021 | August 2021

Publication

GCM: A Toolkit for Generating Synthetic Code-mixed Text

Mohd Sanad Zaki Rizvi, Anirudh Srinivasan, Tanuja Ganu, Monojit Choudhury, Sunayana Sitaram

2021 Conference of the European Chapter of the Association for Computational Linguistics | April 2021

DOI Publication Publication Project

A Case Study of Efficacy and Challenges in Practical Human-in-Loop Evaluation of NLP Systems Using Checklist

Shaily Bhatt, Rahul Jain, Sandipan Dandapat, Sunayana Sitaram

Workshop on Human Evaluation of NLP Systems | April 2021

Publication Project

GLUECoS: An Evaluation Benchmark for Code-Switched NLP

Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, Monojit Choudhury

ACL 2020 | July 2020

Publication Github

GLUECoS: An Evaluation Benchmark for Code-Switched NLP

Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, Monojit Choudhury

Association for Computational Linguistics (ACL) | June 2020

DOI Publication Project

A New Dataset for Natural Language Inference from Code-mixed Conversations

Simran Khanuja, Sandipan Dandapat, Sunayana Sitaram, Monojit Choudhury

Workshop on Computational Approaches to Code Switching | May 2020

PDF Publication Project

Crowdsourcing Speech Data for Low-Resource Languages from Low-Income Workers

Basil Abraham, Danish Goel, Divya Siddarth, Kalika Bali, Manu Chopra, Monojit Choudhury, Pratik Joshi, Preethi Jyothi, Sunayana Sitaram, Vivek Seshadri

Language Resources and Evaluation Conference (LREC) | May 2020

PDF Publication Project Project

Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities.

Pratik Joshi, Christain Barnes, Sebastin Santy, Simran Khanuja, Sanket Shah, Anirudh Srinivasan, Satwik Bhattamishra, Sunayana Sitaram, Monojit Choudhury, Kalika Bali

International Conference on Natural Language Processing (ICON) | December 2019

PDF Publication Project

Deploying Language Technologies for Underserved Communities

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Sebastin Santy

UNESCO International Conference on Language Technologies for all (LT4All) | December 2019

Publication Project

ELLORA: Enabling Low Resource Languages with Technology

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Vivek Seshadri

UNESCO International Conference on Language Technologies for all (LT4All) | December 2019

PDF Publication Video Github Project

Using Monolingual Speech Recognition for Spoken Term Detection in Code-switched Hindi-English Speech

Sanket Shah, Sunayana Sitaram

International Conference on Data Mining | November 2019

DOI Publication Project

CoSSAT: Code-Switched Speech Annotation Tool

Sanket Shah, Pratik Joshi, Sunayana Sitaram (susitara), Sebastin Santy

EMNLP AnnoNLP workshop | November 2019

Publication Project

Word Embeddings for Code-Mixed Language Processing

Adithya Pratapa, Monojit Choudhury, Sunayana Sitaram

Empirical Methods in Natural Language Processing (EMNLP) | November 2018

Publication Project

Effect of TTS Generated Audio on OOV Detection and Word Error Rate in ASR for Low-resource Languages

Savitha Murthy, Dinkar Sitaram, Sunayana Sitaram

Interspeech 2018 | September 2018

Publication

Homophone Identification and Merging for Code-switched Speech Recognition

Brij Mohan Lal Srivastava, Sunayana Sitaram

Interspeech 2018 | September 2018

Publication Project

Interspeech 2018 Low Resource Automatic Speech Recognition Challenge for Indian Languages.

Brij Mohan Lal Srivastava, Sunayana Sitaram, Rupesh Kumar Mehta, Krishna Doss Mohan, Pallavi Matani, Sandeepkumar Satpal, Kalika Bali, Radhakrishnan Srikanth, Niranjan Nayak

Workshop Spoken Language Technologies for Under-resourced Languages | August 2018

DOI Publication Publication Project

Interspeech 2018 Low Resource Automatic Speech Recognition Challenge for Indian Languages

Brij Mohan Lal Srivastava, Sunayana Sitaram, Kalika Bali, Rupesh Kumar Mehta, Krishna Doss Mohan, Pallavi Matani, Sandeepkumar Satpal, Kalika Bali, Radhakrishnan Srikanth, Niranjan Nayak

SLTU | August 2018

Publication Project

Automatic Detection of Code-switching Style from Acoustics

SaiKrishna Rallabandi, Sunayana Sitaram, Alan W Black

Workshop on Computational Approaches to Linguistic Code Switching, 2018 | July 2018

Publication Project

Phone Merging For Code-switched Speech Recognition

Sunit Sivasankaran, Brij Mohan Lal Srivastava, Sunayana Sitaram, Kalika Bali, Monojit Choudhury

Workshop on Computational Approaches to Linguistic Code Switching, 2018 | July 2018

Publication Project

Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data

Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, Kalika Bali

Proceedings of ACL 2018 | July 2018

Publication Project

Discovering Canonical Indian English Accents: A Crowdsourcing-based Approach

Sunayana Sitaram, Varun Manjunath, Varun Bharadwaj, Monojit Choudhury, Kalika Bali, Michael Tjalve

LREC 2018 | May 2018

Publication

Discovering Canonical Indian English Accents: A Crowdsourcing-based Approach

Sunayana Sitaram, Varun Manjunath, Varun Bharadwaj, Monojit Choudhury, Kalika Bali, Michael Tjalve

Language Resources and Evaluation (LREC) | April 2018

PDF Publication Publication Project

Curriculum Design for Code-switching: Experiments with Language Identification and Language Modeling with Deep Neural Networks

Ashutosh Baheti, Sunayana Sitaram, Monojit Choudhury, Kalika Bali

ICON 2017 | December 2017

Publication Project

Speech Synthesis for Mixed-Language Navigation Instructions

Khythiraghavi Chandu, Sai Krishna Rallabandi, Sunayana Sitaram, Alan W Black

Interspeech 2017 | August 2017

Publication Project

Experiments with Cross-lingual Systems for Synthesis of Code-Mixed Text

Sunayana Sitaram, Sai Krishna Rallabandi, Shruti Rijhwani, Alan W Black

Speech Synthesis Workshop 8 | August 2016

Publication Project

Bridging the Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs

Somnath Kumar, Vaibhav Balloli, Mercy Ranjit, Kabir Ahuja, Tanuja Ganu, Sunayana Sitaram, Kalika Bali, Akshay Nambi

May 2024

Preprint

PARIKSHA: A Scalable, Democratic, Transparent Evaluation Platform for Assessing Indic Large Language Models

Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Swami Manohar, Sunayana Sitaram

May 2024

Preprint Video Project

A Survey of Code-switched Speech and Language Processing

Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, Alan W. Black

March 2019

Publication Project

TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs

Nishanth Chandran, Sunayana Sitaram, Divya Gupta, Rahul Sharma, Kashish Mittal, Swami Manohar

February 2024

DOI Publication Publication Github

DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures

Agrima Seth, Sanchit Ahuja, Kalika Bali, Sunayana Sitaram

February 2024

DOI Publication Publication Github

ACL 2023 Multilingual Models Tutorial Slides

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Barun Patra, Vishrav Chaudhary, Kabir Ahuja

July 2023

Publication Project

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?

Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, M. Choudhury, Kalika Bali, Sunayana Sitaram

2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA) | September 2023, pp. 1051-1070

DOI Publication Publication PDF

Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition

Gurunath Reddy Madhumani, Sanket Shah, Basil Abraham, Vikas Joshi, Sunayana Sitaram

arXiv preprint arXiv:2006.05257 | June 2020

PDF Publication Project

End-to-End ASR for Code-switched Hindi-English Speech

Brij Mohan Lal Srivastava, Basil Abraham, Sunayana Sitaram, Rupesh K. Mehta, Preethi Jyothi

arXiv preprint arXiv:1906.09426 | June 2019

Publication PDF Project