{"id":950022,"date":"2023-06-16T16:11:13","date_gmt":"2023-06-16T23:11:13","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-event&#038;p=950022"},"modified":"2023-07-09T05:31:10","modified_gmt":"2023-07-09T12:31:10","slug":"acl-2023-multilingual-models-tutorial","status":"publish","type":"msr-event","link":"https:\/\/www.microsoft.com\/en-us\/research\/event\/acl-2023-multilingual-models-tutorial\/","title":{"rendered":"ACL 2023 Multilingual Models Tutorial"},"content":{"rendered":"\n\n\n\n\n<h2 class=\"wp-block-heading\" id=\"everything-you-need-to-know-about-multilingual-llms-towards-fair-performant-and-reliable-models-for-the-languages-of-the-world\">Everything you need to know about Multilingual LLMs: Towards fair, performant and reliable models for the languages of the world<\/h2>\n\n\n\n<p><strong>Date\/time<\/strong>: July 9, 2023 | 9:00 AM &#8211; 12:30 PM<\/p>\n\n\n\n<p><strong>Location<\/strong>: Metropolitan West, Westin Harbour Castle, Toronto, Canada<\/p>\n\n\n\n<p><em>This tutorial workshop is co-located with\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/2023.aclweb.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">ACL 2023<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/em><\/p>\n\n\n\n<p><strong><mark class=\"has-inline-color has-yellow-color\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/07\/ACL2023MultilingualModelsTutorial.pdf\">Tutorial Slides<\/a><\/mark><\/strong><\/p>\n\n\n\n<p>The technology landscape is being rapidly transformed by Large Language Models (LLMs), allowing users to address real-world applications in various domains. However, a&nbsp;<em>digital divide<\/em>&nbsp;exists that may exclude large populations from benefiting and contributing to this technological revolution due to factors such as language, income, digital awareness, and access to information. At Microsoft, we are dedicated to making Large Language Models inclusive to everyone on the planet.<\/p>\n\n\n\n<p>This tutorial will describe various aspects of scaling up language technologies to many of the world\u2019s languages by presenting the latest research in Massively Multilingual Language Models (MMLMs). We will cover topics such as data collection, training and fine-tuning of models, Responsible AI issues such as fairness, bias and toxicity, linguistic diversity and evaluation in the context of MMLMs, specifically focusing on issues in non-English and low-resource languages. Further, we will also talk about some of the real-world challenges in deploying these models in language communities in the field.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"tutorial-topics\">Tutorial topics<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Linguistic Diversity in MMLMs<\/li>\n\n\n\n<li>Data collection and training of MMLMs<\/li>\n\n\n\n<li>Prompting strategies for MMLMs<\/li>\n\n\n\n<li>Evaluation, analysis and interpretability of MMLMs<\/li>\n\n\n\n<li>Responsible AI issues in MMLMs<\/li>\n\n\n\n<li>Working with Language Communities<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"organizing-committee\">Organizing committee<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/susitara\/\">Sunayana Sitaram<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/monojitc\/\">Monojit Choudhury<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/kalikab\/\">Kalika Bali<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/vchaudhary\/\">Vishrav Chaudhary<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/bapatra\/\">Barun Patra<\/a><\/li>\n\n\n\n<li>Kabir Ahuja<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button is-style-outline is-style-outline--474b24b2315169381cba099d818bcb46\"><a data-bi-type=\"button\" class=\"wp-block-button__link wp-element-button\" href=\"mailto:sunayana.sitaram@microsoft.com\">Contact us with questions<\/a><\/div>\n<\/div>\n\n\n\n\n\n<p><strong>Data Collection and Training of\u00a0Multilingual LLMs<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Fan, Angela, et al. &#8220;Beyond English-Centric Multilingual Machine Translation. arXiv e-prints, page.&#8221;\u00a0<em>arXiv preprint arXiv:2010.11125<\/em>\u00a0(2020).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Devlin, Jacob, et al. &#8220;Bert: Pre-training of deep bidirectional transformers for language understanding.&#8221;\u00a0<em>arXiv preprint arXiv:1810.04805<\/em>\u00a0(2018).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Conneau, Alexis, et al. &#8220;Unsupervised cross-lingual representation learning at scale.&#8221;\u00a0<em>arXiv preprint arXiv:1911.02116<\/em>\u00a0(2019).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Conneau, Alexis, et al. &#8220;Unsupervised cross-lingual representation learning at scale.&#8221;\u00a0<em>arXiv preprint arXiv:1911.02116<\/em>\u00a0(2019).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Xue, Linting, et al. &#8220;mT5: A massively multilingual pre-trained text-to-text transformer.&#8221;\u00a0<em>arXiv preprint arXiv:2010.11934<\/em>\u00a0(2020).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Chi, Zewen, et al. &#8220;Xlm-e: Cross-lingual language model pre-training via electra.&#8221;\u00a0<em>arXiv preprint arXiv:2106.16138<\/em>\u00a0(2021).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Liu, Yinhan, et al. &#8220;Multilingual denoising pre-training for neural machine translation.&#8221;\u00a0<em>Transactions of the Association for Computational Linguistics<\/em>\u00a08 (2020): 726-742.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Patra, Barun, et al. &#8220;Beyond english-centric bitexts for better multilingual language representation learning.&#8221;\u00a0<em>arXiv preprint arXiv:2210.14867<\/em>\u00a0(2022).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Chung, Hyung Won, et al. &#8220;Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.&#8221;\u00a0<em>arXiv preprint arXiv:2304.09151<\/em>\u00a0(2023).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>He, Pengcheng, Jianfeng Gao, and Weizhu Chen. &#8220;Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.&#8221;\u00a0<em>arXiv preprint arXiv:2111.09543<\/em>\u00a0(2021).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Chen, Ting, et al. &#8220;A simple framework for contrastive learning of visual representations.&#8221;\u00a0<em>International conference on machine learning<\/em>. PMLR, 2020.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>He, Kaiming, et al. &#8220;Momentum contrast for unsupervised visual representation learning.&#8221;\u00a0<em>Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition<\/em>. 2020.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Chi, Zewen, et al. &#8220;InfoXLM: An information-theoretic framework for cross-lingual language model pre-training.&#8221;\u00a0<em>arXiv preprint arXiv:2007.07834<\/em>\u00a0(2020).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Xue, Linting, et al. &#8220;mT5: A massively multilingual pre-trained text-to-text transformer.&#8221;\u00a0arXiv preprint arXiv:2010.11934\u00a0(2020).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Xue, Linting, et al. &#8220;Byt5: Towards a token-free future with pre-trained byte-to-byte models.&#8221;\u00a0<em>Transactions of the Association for Computational Linguistics<\/em>\u00a010 (2022): 291-306.<\/li>\n\n\n\n<li>.Liu, Yinhan, et al. &#8220;Multilingual denoising pre-training for neural machine translation.&#8221;\u00a0<em>Transactions of the Association for Computational Linguistics<\/em>\u00a08 (2020): 726-742.<\/li>\n\n\n\n<li>Soltan, Saleh, et al. &#8220;Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model.&#8221;\u00a0<em>arXiv preprint arXiv:2208.01448<\/em>\u00a0(2022).<\/li>\n\n\n\n<li>Lin, Xi Victoria, et al. &#8220;Few-shot learning with multilingual language models.&#8221;\u00a0<em>arXiv preprint arXiv:2112.10668<\/em>\u00a0(2021).<\/li>\n\n\n\n<li>Wang, Thomas, et al. &#8220;What language model architecture and pretraining objective works best for zero-shot generalization?.&#8221;\u00a0<em>International Conference on Machine Learning<\/em>. PMLR, 2022.<\/li>\n\n\n\n<li>Tay, Yi, et al. &#8220;Transcending scaling laws with 0.1% extra compute.&#8221;\u00a0<em>arXiv preprint arXiv:2210.11399<\/em>\u00a0(2022).<\/li>\n\n\n\n<li>Chung, Hyung Won, et al. &#8220;Scaling instruction-finetuned language models.&#8221;\u00a0<em>arXiv preprint arXiv:2210.11416<\/em>\u00a0(2022).<\/li>\n\n\n\n<li>Muennighoff, Niklas, et al. &#8220;Crosslingual generalization through multitask finetuning.&#8221;\u00a0<em>arXiv preprint arXiv:2211.01786<\/em>\u00a0(2022).<\/li>\n\n\n\n<li>OpenAI. 2023.\u00a0GPT-4 Technical Report.<\/li>\n\n\n\n<li>Google. 2023.\u00a0PaLM 2 Technical Report.<\/li>\n\n\n\n<li>Hyung Won Chung\u2217, Le Hou\u2217, Shayne Longpre\u2217 et al. 2022. Scaling Instruction-Finetuned Language Models.<\/li>\n\n\n\n<li>Aakanksha Chowdhery\u2217, Sharan Narang\u2217, Jacob Devlin\u2217 et al. 2022.\u00a0PaLM: Scaling Language Modeling with Pathways.<\/li>\n\n\n\n<li>Weijia Shi et al. 2023. REPLUG: Retrieval-Augmented Black-Box Language Models\u00a0\u00a0<\/li>\n\n\n\n<li>BigScienceWorkshop. 2022. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model.<\/li>\n\n\n\n<li>Yi Tay et al. 2023.\u00a0Transcending Scaling Laws with 0.1% Extra Compute.<\/li>\n\n\n\n<li>Hugo Touvron\u2217 , Thibaut Lavril\u2217 , Gautier Izacard\u2217 et al. 2023.\u00a0LLaMA: Open and Efficient Foundation Language Models.<\/li>\n\n\n\n<li>Jack W. Rae et al. 2022. Scaling Language Models: Methods, Analysis & Insights from Training Gopher.<\/li>\n\n\n\n<li>Gautier Izacard\u2217, Patrick Lewis\u2217. 2023.\u00a0Atlas: Few-shot Learning with Retrieval Augmented Language Models.<\/li>\n\n\n\n<li>Aohan Zeng\u2217, Xiao Liu* et al. 2022.\u00a0GLM-130B: AN OPEN BILINGUAL PRE-TRAINED MODEL.<\/li>\n\n\n\n<li>Saleh Soltan et al. 2022. AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2seq Model.<\/li>\n\n\n\n<li>Xi Victoria Lin et al. 2022.\u00a0Few-shot Learning with Multilingual Generative Language Models.<\/li>\n\n\n\n<li>Zhihong Chen et al. 2023.\u00a0Phoenix: Democratizing ChatGPT across Languages.<\/li>\n<\/ul>\n\n\n\n<p><strong>Prompting Strategies for Multilingual LLMs<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3560815\">Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won<br>Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought<br>reasoners. arXiv preprint arXiv:2210.03057, 2022.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Lifu Tu, Caiming Xiong, and Yingbo Zhou. Prompt-tuning can be much better than fine-tuning on cross-lingual understanding with multilingual language models, 2022.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Mengjie Zhao and Hinrich Sch\u00fctze. Discrete and soft prompting for multilingual models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547\u20138555, 2021<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei, and Houfeng Wang. Zero-shot cross-lingual transfer of prompt-based tuning with a unified multilingual prompt, 2022.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting, 2023.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Yuxuan Chen, David Harbecke, and Leonhard Hennig. Multilingual relation classification via efficient and<br>effective prompting. arXiv preprint arXiv:2210.13838, 2022.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Breaking Language Barriers with a LEAP: Learning Strategies for Polyglot LLMs, Akshay Nambi, Vaibhav Balloli, Mercy Ranjit, Tanuja Ganu, Kabir Ahuja, Sunayana Sitaram, Kalika Bali<\/li>\n<\/ul>\n\n\n\n<p><strong>Evaluation, Interpretability and Analysis of Multilingual LLMs<\/strong><\/p>\n\n\n\n<p>Datasets<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/D18-1269\">XNLI: Evaluating Cross-lingual Sentence Representations<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 2475\u20132485, Brussels, Belgium. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/D19-1382\">PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)<\/em>, pages 3687\u20133692, Hong Kong, China. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Nivre, Joakim and Abrams, Mitchell and Agi{\\&#8217;c}, {\\v{Z}}eljko and Ahrenberg, Lars and Antonsen, Lene and Aranzabe, Maria Jesus and Arutie, Gashaw and Asahara, Masayuki and Ateyah, Luma and Attia, Mohammed and others. Universal Dependencies 2.2. 2018.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/P17-1178\">Cross-lingual Name Tagging and Linking for 282 Languages<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)<\/em>, pages 1946\u20131958, Vancouver, Canada. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2020.acl-main.421\">On the Cross-lingual Transferability of Monolingual Representations<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics<\/em>, pages 4623\u20134637, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2020.acl-main.653\">MLQA: Evaluating Cross-lingual Extractive Question Answering<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics<\/em>, pages 7315\u20137330, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2020.tacl-1.30\">TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.\u00a0<em>Transactions of the Association for Computational Linguistics<\/em>, 8:454\u2013470.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2021.findings-acl.413\">XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021<\/em>, pages 4693\u20134703, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, Pratyush Kumar. Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages. 2020.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra, and Pratyush Kumar. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.emnlp-main.360\">IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 5363\u20135394, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, et al.. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.emnlp-main.298\">MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 4488\u20134508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cheikh M. Bamba Dione, David Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue and others. MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages. 2023.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Gim\u00e9nez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.acl-long.435\">AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)<\/em>, pages 6279\u20136299, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Arnav Mhaske, Harshit Kedia, Sumanth Doddapaneni, Mitesh M. Khapra, Pratyush Kumar, Rudra Murthy V, Anoop Kunchukuttan. 2023. Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages<\/li>\n<\/ul>\n\n\n\n<p>Benchmarking Exercises<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation. Proceedings of the 37th International Conference on Machine Learning, PMLR 119:4411-4421, 2020.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2021.emnlp-main.802\">XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 10215\u201310245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, Sunayana Sitaram. 2023. MEGA: Multilingual Evaluation of Generative AI.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, Hannaneh Hajishirzi. 2023. BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, Yulia Tsvetkov. 2023. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models<\/li>\n<\/ul>\n\n\n\n<p>Evaluation Beyond Task Performance<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kabir Ahuja, Sunayana Sitaram, Sandipan Dandapat, and Monojit Choudhury. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.emnlp-main.290\">On the Calibration of Massively Multilingual Language Models<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 4310\u20134323, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Zhengping Jiang, Anqi Liu, and Benjamin Van Durme. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.emnlp-main.170\">Calibrating Zero-shot Cross-lingual (Un-)structured Predictions<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 2648\u20132674, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, and Naoaki Okazaki. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.naacl-main.197\">Gender Bias in Masked Language Models for Multiple Languages<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies<\/em>, pages 2740\u20132750, Seattle, United States. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Aniket Vashishtha, Kabir Ahuja, and Sunayana Sitaram. 2023. On Evaluating and Mitigating Gender Biases in Multilingual Settings.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902\u20134912, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Karthikeyan K, Shaily Bhatt, Pankaj Singh, Somak Aditya, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.findings-aacl.27\">Multilingual CheckList: Generation and Evaluation<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022<\/em>, pages 282\u2013295, Online only. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<p>Challenges in Multilingual Evaluation<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kabir Ahuja, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2022. Beyond Static models and test sets: Benchmarking the potential of pre-trained models across tasks and languages. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP, pages 64\u201374, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kabir Ahuja, Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.acl-long.374\">Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)<\/em>, pages 5454\u20135467, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2020.acl-main.764\">Predicting Performance for Natural Language Processing Tasks<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics<\/em>, pages 8625\u20138646, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kabir Ahuja, Antonios Anastasopoulos, Barun Patra, Graham Neubig, Monojit Choudhury, Sandipan Dandapat, Sunayana Sitaram, and Vishrav Chaudhary. 2022.\u00a0<em><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.sumeval-1.0\">Proceedings of the First Workshop on Scaling Up Multilingual Evaluation<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/em>. Association for Computational Linguistics, Online, edition.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2021.emnlp-main.818\">Visually Grounded Reasoning across Languages and Cultures<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 10467\u201310485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, Kristina Toutanova. 2021. Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer<\/li>\n<\/ul>\n\n\n\n<p>Analysis and Interpretability<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Anne Lauscher, Vinit Ravishankar, Ivan Vuli\u0107, and Goran Glava\u0161. 2020.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2020.emnlp-main.363\">From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)<\/em>, pages 4483\u20134499, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Karthikeyan K and Zihan Wang and Stephen Mayhew and Dan Roth,\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/cogcomp.seas.upenn.edu\/papers\/KWMR20.pdf\">Cross-Lingual Ability of Multilingual BERT: An Empirical Study<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>\u00a0<em>ICLR\u00a0<\/em>(2020)<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2020.acl-main.493\">Finding Universal Grammatical Relations in Multilingual BERT<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics<\/em>, pages 5564\u20135577, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Karolina Stanczak, Edoardo Ponti, Lucas Torroba Hennigen, Ryan Cotterell, and Isabelle Augenstein. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.naacl-main.114\">Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies<\/em>, pages 1589\u20131598, Seattle, United States. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Aaron Mueller, Yu Xia, and Tal Linzen. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.conll-1.8\">Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)<\/em>, pages 95\u2013109, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<p>What multilingual evaluation tells us about the current state of NLP<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022.\u00a0<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/2022.acl-long.376\">Systematic Inequalities in Language Technology Performance across the World\u2019s Languages<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. In\u00a0<em>Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)<\/em>, pages 5486\u20135505, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<p><strong>Responsible AI for Multilingual LLMs<\/strong><\/p>\n\n\n\n<p>Socio-Cultural Aspects<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ma\u0142gorzata Suszczy\u00b4nska. 1999. Apologizing in english, polish and hungarian: Different languages, different strategies. Journal of Pragmatics, 31(8):1053\u20131065.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, and David R. Mortensen. 2021. Crosscultural similarity features for cross-lingual transfer learning of pragmatically motivated tasks. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2403\u20132414, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Dong Nguyen, A. Seza Do\u02d8gru\u00f6z, Carolyn P. Ros\u00e9, and Franciska de Jong. 2016. Computational Sociolinguistics: A Survey. Computational Linguistics, 42(3):537\u2013593.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Daniel Hershcovich, et al. 2022. Challenges and strategies in crosscultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997\u20137013, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>N\u00e9v\u00e9ol et al. French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English. ACL 2022<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Blodgett et al., Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. ACL-IJCNLP 2021<\/li>\n<\/ul>\n\n\n\n<p>Gender Bias and Grammatical Gender<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical gender. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5276\u20135284<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender bias in multilingual embeddings and cross-lingual transfer. In Proceedings of the 58<sup>th<\/sup> Annual Meeting of the Association for Computational Linguistics, pages 2896\u20132907, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Sheng Liang, Philipp Dufter, and Hinrich Sch\u00fctze. 2020. Monolingual and multilingual reduction of gender bias in contextualized representations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5082\u20135093, Barcelona, Spain (Online). International Committee on Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, and Naoaki Okazaki. 2022. Gender bias in masked language models for multiple languages. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2740\u20132750, Seattle, United States. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>[MITIGATION] Xiaolei Huang. 2022. Easy adaptation to mitigate gender bias in multilingual text classification. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 717\u2013723, Seattle, United States. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>[Model Compression] Jaimeen Ahn, Hwaran Lee, Jinhwa Kim, and Alice Oh. 2022. Why knowledge distillation amplifies gender bias and how to mitigate from the perspective of DistilBERT. In Proceedings of the 4th Workshopon Gender Bias in Natural Language Processing (GeBNLP), pages 266\u2013272, Seattle, Washington. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Zmigrod et al., <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/aclanthology.org\/P19-1161.pdf\">Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, ACL 2019<\/li>\n<\/ul>\n\n\n\n<p>Bias and Fairness<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Jialu Wang, Yang Liu, and Xin Wang. 2022. Assessing multilingual fairness in pre-trained multimodal representations. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2681\u20132695, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Zeerak Talat, et al. 2022. You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Proceedings of BigScience Episode #5 \u2013 Workshop on Challenges & Perspectives in Creating Large Language Models, pages 26\u201341, virtual+Dublin. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Aristides Milios and Parishad BehnamGhader. 2022. An analysis of social biases present in bert variants across multiple languages. ArXiv, abs\/2211.14402.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cristina Espa\u00f1a-Bonet and Alberto Barr\u00f3n-Cede\u00f1o. 2022. The (undesired) attenuation of human biases by multilinguality. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages \u2013, Online and Abu Dhabi, UAE. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>[Dataset] Ilias Chalkidis, et al., 2022. FairLex: A multilingual benchmark for evaluating fairness in legal text processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4389\u20134406, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<p>Hate Speech, Toxicity and Sentiment<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and Michael J. Paul. 2020. Multilingual Twitter corpus and baselines for evaluating demographic bias in hate speech recognition. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1440\u20131448, Marseille, France. European Language Resources Association.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ant\u00f3nio C\u00e2mara, Nina Taneja, Tamjeed Azad, Emily Allaway, and Richard Zemel. 2022. Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic.In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, pages 90\u2013106, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<p>Discursive Aspects (Decolonizing RAI)<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021. Re-imagining algorithmic fairness in india and beyond. CoRR, abs\/2101.09995.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. 2022. Recontextualizing fairness in NLP: The case of India. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 727\u2013740, Online only.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Krithika Ramesh, Sunayana Sitaram, Monojit Choudhury. 2023. Fairness in Language Models Beyond English: Gaps and Challenges. Findings of EACL 2023<\/li>\n<\/ul>\n\n\n\n<p>Distributive Aspects<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282\u20136293, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Monojit Choudhury and Amit Deshpande. 2021. How linguistically fair are multilingual pre-trained language models? Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12710\u201312718.<\/li>\n<\/ul>\n\n\n\n<p>General<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT \u201921, page 610\u2013623, New York, NY, USA. Association for Computing Machinery.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Su Lin Blodgett, Solon Barocas, Hal Daum\u00e9 III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of \u201cbias\u201d in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454\u20135476, Online. Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<p><strong>Working with Multilingual Language Communities<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Abraham, Basil, et al. &#8220;Crowdsourcing Speech Data for Low-Resource Languages from Low-Income Workers.&#8221; Proceedings of the 12th Language Resources and Evaluation Conference. 2020.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Almaliki, Malik, et al. &#8220;ABMM: Arabic BERT-Mini Model for Hate-Speech Detection on Social Media.&#8221; Electronics 12.4 (2023): 1048.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Bird, Steven. &#8220;Decolonising Speech and Language Technology.&#8221; Proceedings of the 28th International Conference on Computational Linguistics. 2020.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Bird, Steven. &#8220;Local Languages, Third Spaces, and Other High-Resource Scenarios.&#8221; Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Chopra, Manu, et al. &#8220;Exploring Crowdsourced Work in Low-Resource Settings.&#8221; Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 2019.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Currin, Christopher Brian, et al. &#8220;A Framework for Grassroots Research Collaboration in Machine Learning and Global Health.&#8221; 2023 ICLR First Workshop on &#8220;Machine Learning & Global Health&#8221;. 2023.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Diddee, Harshita, Bali, Kalika, Choudhury, Monojit, Ganu, Tanuja, and Dandapat, Sandipan. 2022. Too Brittle to Touch: Comparing the Stability of Quantization and Distillation towards Developing Low-Resource MT Models. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 870\u2013885, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Diddee, Harshita, Bali, Kalika, Choudhury, Monojit, and Mukhija, Namrata. 2022. The Six Conundrums of Building and Deploying Language Technologies for Social Good. In ACM SIGCAS\/SIGCHI Conference on Computing and Sustainable Societies (COMPASS) (COMPASS &#8217;22). Association for Computing Machinery, New York, NY, USA, 12\u201319.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Goswami, Dipam, et al. &#8220;Analysis of Word-level Embeddings for Indic Languages on AI4Bharat-IndicNLP Corpora.&#8221; 2021 IEEE 8th Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON). IEEE, 2021.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Gitau, Catherine, et al. &#8220;Masakhane Web: A Machine Translation Platform for African Languages.&#8221; (2023)<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Harrigan, Atticus, et al. &#8220;Proceedings of the Sixth Workshop on the Use of Computational Methods in the Study of Endangered Languages.&#8221; Proceedings of the Sixth Workshop on the Use of Computational Methods in the Study of Endangered Languages. 2023.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Joshi, Pratik, et al. &#8220;Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities.&#8221; arXiv preprint arXiv:1912.03457 (2019).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kann, Katharina, et al. &#8220;AmericasNLI: Machine Translation and Natural Language Inference Systems for Indigenous Languages of the Americas.&#8221; Frontiers in Artificial Intelligence 5 (2022): 266.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Koto, Fajri, et al. &#8220;IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP.&#8221; arXiv preprint arXiv:2011.00677 (2020).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kumar, Ritesh, et al. &#8220;Annotated Speech Corpus for Low Resource Indian Languages: Awadhi, Bhojpuri, Braj and Magahi.&#8221; arXiv preprint arXiv:2206.12931 (2022).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kunchukuttan, Anoop, et al. &#8220;Ai4bharat-indicnlp Corpus: Monolingual Corpora and Word Embeddings for Indic Languages.&#8221; arXiv preprint arXiv:2005.00085 (2020).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Mehta, Devansh, et al. &#8220;Learnings from Technological Interventions in a Low Resource Language: Enhancing Information Access in Gondi.&#8221; arXiv preprint arXiv:2211.16172 (2022).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Moeller, Sarah, et al. &#8220;Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages.&#8221; Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages. 2022.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Moradshahi, Mehrad, et al. &#8220;X-RiSAWOZ: High-Quality End-to-End Multilingual Dialogue Datasets and Few-shot Agents.&#8221; arXiv preprint arXiv:2306.17674 (2023).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Muhammad, Shamsuddeen Hassan, et al. &#8220;Naijasenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis.&#8221; arXiv preprint arXiv:2201.08277 (2022).<\/li>\n\n\n\n<li>AS Do\u011fru\u00f6z and S Sitaram, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/scholar.google.co.in\/citations?view_op=view_citation&hl=en&user=PUxwYrkAAAAJ&sortby=pubdate&citation_for_view=PUxwYrkAAAAJ:GnPB-g6toBAC\">Language technologies for low resource languages: sociolinguistic and multilingual insights<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, LREC 2022.<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n","protected":false},"excerpt":{"rendered":"<p>Date\/time: July 9, 2023 | 9:00 AM &#8211; 12:30 PM Location: Metropolitan West, Westin Harbour Castle, Toronto, Canada This tutorial workshop is co-located with\u00a0ACL 2023 (opens in new tab). Tutorial Slides The technology landscape is being rapidly transformed by Large Language Models (LLMs), allowing users to address real-world applications in various domains. However, a&nbsp;digital divide&nbsp;exists [&hellip;]<\/p>\n","protected":false},"featured_media":947454,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr_startdate":"2023-07-09","msr_enddate":"2023-07-09","msr_location":"Toronto, Canada","msr_expirationdate":"","msr_event_recording_link":"","msr_event_link":"","msr_event_link_redirect":false,"msr_event_time":"Eastern Daylight Time (UTC -4)","msr_hide_region":false,"msr_private_event":false,"footnotes":""},"research-area":[13556],"msr-region":[197900],"msr-event-type":[197941,210063],"msr-video-type":[],"msr-locale":[268875],"msr-program-audience":[],"msr-post-option":[],"msr-impact-theme":[264846],"class_list":["post-950022","msr-event","type-msr-event","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-region-north-america","msr-event-type-conferences","msr-event-type-workshop","msr-locale-en_us"],"msr_about":"<!-- wp:msr\/event-details {\"title\":\"ACL 2023 Multilingual Models Tutorial\",\"image\":{\"id\":947454,\"url\":\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/06\/ACL_Conference_Banner_1920x720_C.jpg\",\"alt\":\"abstract shapes\"}} \/-->\n\n<!-- wp:msr\/content-tabs -->\n<!-- wp:msr\/content-tab -->\n<!-- wp:heading -->\n<h2 class=\"wp-block-heading\" id=\"everything-you-need-to-know-about-multilingual-llms-towards-fair-performant-and-reliable-models-for-the-languages-of-the-world\">Everything you need to know about Multilingual LLMs: Towards fair, performant and reliable models for the languages of the world<\/h2>\n<!-- \/wp:heading -->\n\n<!-- wp:paragraph -->\n<p><strong>Date\/time<\/strong>: July 9, 2023 | 9:00 AM - 12:30 PM<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p><strong>Location<\/strong>: Metropolitan West, Westin Harbour Castle, Toronto, Canada<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p><em>This tutorial workshop is co-located with\u00a0<a href=\"https:\/\/2023.aclweb.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">ACL 2023<\/a>.<\/em><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p><strong><mark class=\"has-inline-color has-yellow-color\"><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/07\/ACL2023MultilingualModelsTutorial.pdf\">Tutorial Slides<\/a><\/mark><\/strong><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>The technology landscape is being rapidly transformed by Large Language Models (LLMs), allowing users to address real-world applications in various domains. However, a&nbsp;<em>digital divide<\/em>&nbsp;exists that may exclude large populations from benefiting and contributing to this technological revolution due to factors such as language, income, digital awareness, and access to information. At Microsoft, we are dedicated to making Large Language Models inclusive to everyone on the planet.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>This tutorial will describe various aspects of scaling up language technologies to many of the world\u2019s languages by presenting the latest research in Massively Multilingual Language Models (MMLMs). We will cover topics such as data collection, training and fine-tuning of models, Responsible AI issues such as fairness, bias and toxicity, linguistic diversity and evaluation in the context of MMLMs, specifically focusing on issues in non-English and low-resource languages. Further, we will also talk about some of the real-world challenges in deploying these models in language communities in the field.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3 class=\"wp-block-heading\" id=\"tutorial-topics\">Tutorial topics<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Linguistic Diversity in MMLMs<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Data collection and training of MMLMs<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Prompting strategies for MMLMs<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Evaluation, analysis and interpretability of MMLMs<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Responsible AI issues in MMLMs<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Working with Language Communities<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:heading {\"level\":3} -->\n<h3 class=\"wp-block-heading\" id=\"organizing-committee\">Organizing committee<\/h3>\n<!-- \/wp:heading -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/susitara\/\">Sunayana Sitaram<\/a><\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/monojitc\/\">Monojit Choudhury<\/a><\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/kalikab\/\">Kalika Bali<\/a><\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/vchaudhary\/\">Vishrav Chaudhary<\/a><\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/bapatra\/\">Barun Patra<\/a><\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Kabir Ahuja<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:buttons -->\n<div class=\"wp-block-buttons\"><!-- wp:button {\"className\":\"is-style-outline\"} -->\n<div class=\"wp-block-button is-style-outline\"><a class=\"wp-block-button__link wp-element-button\" href=\"mailto:sunayana.sitaram@microsoft.com\">Contact us with questions<\/a><\/div>\n<!-- \/wp:button --><\/div>\n<!-- \/wp:buttons -->\n<!-- \/wp:msr\/content-tab -->\n\n<!-- wp:msr\/content-tab {\"title\":\"References\"} -->\n<!-- wp:paragraph {\"placeholder\":\"Write content\u2026\"} -->\n<p><strong>Data Collection and Training of\u00a0Multilingual LLMs<\/strong><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Fan, Angela, et al. \"Beyond English-Centric Multilingual Machine Translation. arXiv e-prints, page.\"\u00a0<em>arXiv preprint arXiv:2010.11125<\/em>\u00a0(2020).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Devlin, Jacob, et al. \"Bert: Pre-training of deep bidirectional transformers for language understanding.\"\u00a0<em>arXiv preprint arXiv:1810.04805<\/em>\u00a0(2018).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Conneau, Alexis, et al. \"Unsupervised cross-lingual representation learning at scale.\"\u00a0<em>arXiv preprint arXiv:1911.02116<\/em>\u00a0(2019).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Conneau, Alexis, et al. \"Unsupervised cross-lingual representation learning at scale.\"\u00a0<em>arXiv preprint arXiv:1911.02116<\/em>\u00a0(2019).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Xue, Linting, et al. \"mT5: A massively multilingual pre-trained text-to-text transformer.\"\u00a0<em>arXiv preprint arXiv:2010.11934<\/em>\u00a0(2020).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Chi, Zewen, et al. \"Xlm-e: Cross-lingual language model pre-training via electra.\"\u00a0<em>arXiv preprint arXiv:2106.16138<\/em>\u00a0(2021).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Liu, Yinhan, et al. \"Multilingual denoising pre-training for neural machine translation.\"\u00a0<em>Transactions of the Association for Computational Linguistics<\/em>\u00a08 (2020): 726-742.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Patra, Barun, et al. \"Beyond english-centric bitexts for better multilingual language representation learning.\"\u00a0<em>arXiv preprint arXiv:2210.14867<\/em>\u00a0(2022).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Chung, Hyung Won, et al. \"Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.\"\u00a0<em>arXiv preprint arXiv:2304.09151<\/em>\u00a0(2023).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>He, Pengcheng, Jianfeng Gao, and Weizhu Chen. \"Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.\"\u00a0<em>arXiv preprint arXiv:2111.09543<\/em>\u00a0(2021).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Chen, Ting, et al. \"A simple framework for contrastive learning of visual representations.\"\u00a0<em>International conference on machine learning<\/em>. PMLR, 2020.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>He, Kaiming, et al. \"Momentum contrast for unsupervised visual representation learning.\"\u00a0<em>Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition<\/em>. 2020.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Chi, Zewen, et al. \"InfoXLM: An information-theoretic framework for cross-lingual language model pre-training.\"\u00a0<em>arXiv preprint arXiv:2007.07834<\/em>\u00a0(2020).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Xue, Linting, et al. \"mT5: A massively multilingual pre-trained text-to-text transformer.\"\u00a0arXiv preprint arXiv:2010.11934\u00a0(2020).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Xue, Linting, et al. \"Byt5: Towards a token-free future with pre-trained byte-to-byte models.\"\u00a0<em>Transactions of the Association for Computational Linguistics<\/em>\u00a010 (2022): 291-306.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>.Liu, Yinhan, et al. \"Multilingual denoising pre-training for neural machine translation.\"\u00a0<em>Transactions of the Association for Computational Linguistics<\/em>\u00a08 (2020): 726-742.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Soltan, Saleh, et al. \"Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model.\"\u00a0<em>arXiv preprint arXiv:2208.01448<\/em>\u00a0(2022).<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Lin, Xi Victoria, et al. \"Few-shot learning with multilingual language models.\"\u00a0<em>arXiv preprint arXiv:2112.10668<\/em>\u00a0(2021).<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Wang, Thomas, et al. \"What language model architecture and pretraining objective works best for zero-shot generalization?.\"\u00a0<em>International Conference on Machine Learning<\/em>. PMLR, 2022.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Tay, Yi, et al. \"Transcending scaling laws with 0.1% extra compute.\"\u00a0<em>arXiv preprint arXiv:2210.11399<\/em>\u00a0(2022).<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Chung, Hyung Won, et al. \"Scaling instruction-finetuned language models.\"\u00a0<em>arXiv preprint arXiv:2210.11416<\/em>\u00a0(2022).<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Muennighoff, Niklas, et al. \"Crosslingual generalization through multitask finetuning.\"\u00a0<em>arXiv preprint arXiv:2211.01786<\/em>\u00a0(2022).<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>OpenAI. 2023.\u00a0GPT-4 Technical Report.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Google. 2023.\u00a0PaLM 2 Technical Report.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Hyung Won Chung\u2217, Le Hou\u2217, Shayne Longpre\u2217 et al. 2022. Scaling Instruction-Finetuned Language Models.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Aakanksha Chowdhery\u2217, Sharan Narang\u2217, Jacob Devlin\u2217 et al. 2022.\u00a0PaLM: Scaling Language Modeling with Pathways.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Weijia Shi et al. 2023. REPLUG: Retrieval-Augmented Black-Box Language Models\u00a0\u00a0<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>BigScienceWorkshop. 2022. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Yi Tay et al. 2023.\u00a0Transcending Scaling Laws with 0.1% Extra Compute.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Hugo Touvron\u2217 , Thibaut Lavril\u2217 , Gautier Izacard\u2217 et al. 2023.\u00a0LLaMA: Open and Efficient Foundation Language Models.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Jack W. Rae et al. 2022. Scaling Language Models: Methods, Analysis &amp; Insights from Training Gopher.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Gautier Izacard\u2217, Patrick Lewis\u2217. 2023.\u00a0Atlas: Few-shot Learning with Retrieval Augmented Language Models.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Aohan Zeng\u2217, Xiao Liu* et al. 2022.\u00a0GLM-130B: AN OPEN BILINGUAL PRE-TRAINED MODEL.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Saleh Soltan et al. 2022. AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2seq Model.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Xi Victoria Lin et al. 2022.\u00a0Few-shot Learning with Multilingual Generative Language Models.<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>Zhihong Chen et al. 2023.\u00a0Phoenix: Democratizing ChatGPT across Languages.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p><strong>Prompting Strategies for Multilingual LLMs<\/strong><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li><a href=\"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3560815\">Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing<\/a><\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won<br>Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought<br>reasoners. arXiv preprint arXiv:2210.03057, 2022.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Lifu Tu, Caiming Xiong, and Yingbo Zhou. Prompt-tuning can be much better than fine-tuning on cross-lingual understanding with multilingual language models, 2022.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Mengjie Zhao and Hinrich Sch\u00fctze. Discrete and soft prompting for multilingual models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547\u20138555, 2021<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei, and Houfeng Wang. Zero-shot cross-lingual transfer of prompt-based tuning with a unified multilingual prompt, 2022.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting, 2023.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Yuxuan Chen, David Harbecke, and Leonhard Hennig. Multilingual relation classification via efficient and<br>effective prompting. arXiv preprint arXiv:2210.13838, 2022.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Breaking Language Barriers with a LEAP: Learning Strategies for Polyglot LLMs, Akshay Nambi, Vaibhav Balloli, Mercy Ranjit, Tanuja Ganu, Kabir Ahuja, Sunayana Sitaram, Kalika Bali<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p><strong>Evaluation, Interpretability and Analysis of Multilingual LLMs<\/strong><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Datasets<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.\u00a0<a href=\"https:\/\/aclanthology.org\/D18-1269\">XNLI: Evaluating Cross-lingual Sentence Representations<\/a>. In\u00a0<em>Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 2475\u20132485, Brussels, Belgium. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.\u00a0<a href=\"https:\/\/aclanthology.org\/D19-1382\">PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification<\/a>. In\u00a0<em>Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)<\/em>, pages 3687\u20133692, Hong Kong, China. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Nivre, Joakim and Abrams, Mitchell and Agi{\\'c}, {\\v{Z}}eljko and Ahrenberg, Lars and Antonsen, Lene and Aranzabe, Maria Jesus and Arutie, Gashaw and Asahara, Masayuki and Ateyah, Luma and Attia, Mohammed and others. Universal Dependencies 2.2. 2018.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.\u00a0<a href=\"https:\/\/aclanthology.org\/P17-1178\">Cross-lingual Name Tagging and Linking for 282 Languages<\/a>. In\u00a0<em>Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)<\/em>, pages 1946\u20131958, Vancouver, Canada. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.\u00a0<a href=\"https:\/\/aclanthology.org\/2020.acl-main.421\">On the Cross-lingual Transferability of Monolingual Representations<\/a>. In\u00a0<em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics<\/em>, pages 4623\u20134637, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020.\u00a0<a href=\"https:\/\/aclanthology.org\/2020.acl-main.653\">MLQA: Evaluating Cross-lingual Extractive Question Answering<\/a>. In\u00a0<em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics<\/em>, pages 7315\u20137330, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.\u00a0<a href=\"https:\/\/aclanthology.org\/2020.tacl-1.30\">TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages<\/a>.\u00a0<em>Transactions of the Association for Computational Linguistics<\/em>, 8:454\u2013470.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021.\u00a0<a href=\"https:\/\/aclanthology.org\/2021.findings-acl.413\">XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages<\/a>. In\u00a0<em>Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021<\/em>, pages 4693\u20134703, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, Pratyush Kumar. Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages. 2020.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Aman Kumar, Himani Shrotriya, Prachi Sahu, Amogh Mishra, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra, and Pratyush Kumar. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.emnlp-main.360\">IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages<\/a>. In\u00a0<em>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 5363\u20135394, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, et al.. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.emnlp-main.298\">MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition<\/a>. In\u00a0<em>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 4488\u20134508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Cheikh M. Bamba Dione, David Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue and others. MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages. 2023.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Gim\u00e9nez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.acl-long.435\">AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages<\/a>. In\u00a0<em>Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)<\/em>, pages 6279\u20136299, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Arnav Mhaske, Harshit Kedia, Sumanth Doddapaneni, Mitesh M. Khapra, Pratyush Kumar, Rudra Murthy V, Anoop Kunchukuttan. 2023. Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>Benchmarking Exercises<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation. Proceedings of the 37th International Conference on Machine Learning, PMLR 119:4411-4421, 2020.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021.\u00a0<a href=\"https:\/\/aclanthology.org\/2021.emnlp-main.802\">XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation<\/a>. In\u00a0<em>Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 10215\u201310245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, Sunayana Sitaram. 2023. MEGA: Multilingual Evaluation of Generative AI.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, Hannaneh Hajishirzi. 2023. BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, Yulia Tsvetkov. 2023. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>Evaluation Beyond Task Performance<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Kabir Ahuja, Sunayana Sitaram, Sandipan Dandapat, and Monojit Choudhury. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.emnlp-main.290\">On the Calibration of Massively Multilingual Language Models<\/a>. In\u00a0<em>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 4310\u20134323, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Zhengping Jiang, Anqi Liu, and Benjamin Van Durme. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.emnlp-main.170\">Calibrating Zero-shot Cross-lingual (Un-)structured Predictions<\/a>. In\u00a0<em>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 2648\u20132674, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, and Naoaki Okazaki. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.naacl-main.197\">Gender Bias in Masked Language Models for Multiple Languages<\/a>. In\u00a0<em>Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies<\/em>, pages 2740\u20132750, Seattle, United States. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Aniket Vashishtha, Kabir Ahuja, and Sunayana Sitaram. 2023. On Evaluating and Mitigating Gender Biases in Multilingual Settings.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902\u20134912, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Karthikeyan K, Shaily Bhatt, Pankaj Singh, Somak Aditya, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.findings-aacl.27\">Multilingual CheckList: Generation and Evaluation<\/a>. In\u00a0<em>Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022<\/em>, pages 282\u2013295, Online only. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>Challenges in Multilingual Evaluation<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Kabir Ahuja, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2022. Beyond Static models and test sets: Benchmarking the potential of pre-trained models across tasks and languages. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP, pages 64\u201374, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Kabir Ahuja, Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.acl-long.374\">Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models<\/a>. In\u00a0<em>Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)<\/em>, pages 5454\u20135467, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020.\u00a0<a href=\"https:\/\/aclanthology.org\/2020.acl-main.764\">Predicting Performance for Natural Language Processing Tasks<\/a>. In\u00a0<em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics<\/em>, pages 8625\u20138646, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Kabir Ahuja, Antonios Anastasopoulos, Barun Patra, Graham Neubig, Monojit Choudhury, Sandipan Dandapat, Sunayana Sitaram, and Vishrav Chaudhary. 2022.\u00a0<em><a href=\"https:\/\/aclanthology.org\/2022.sumeval-1.0\">Proceedings of the First Workshop on Scaling Up Multilingual Evaluation<\/a><\/em>. Association for Computational Linguistics, Online, edition.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021.\u00a0<a href=\"https:\/\/aclanthology.org\/2021.emnlp-main.818\">Visually Grounded Reasoning across Languages and Cultures<\/a>. In\u00a0<em>Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing<\/em>, pages 10467\u201310485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, Kristina Toutanova. 2021. Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>Analysis and Interpretability<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Anne Lauscher, Vinit Ravishankar, Ivan Vuli\u0107, and Goran Glava\u0161. 2020.\u00a0<a href=\"https:\/\/aclanthology.org\/2020.emnlp-main.363\">From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers<\/a>. In\u00a0<em>Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)<\/em>, pages 4483\u20134499, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Karthikeyan K and Zihan Wang and Stephen Mayhew and Dan Roth,\u00a0<a href=\"https:\/\/cogcomp.seas.upenn.edu\/papers\/KWMR20.pdf\">Cross-Lingual Ability of Multilingual BERT: An Empirical Study<\/a>\u00a0<em>ICLR\u00a0<\/em>(2020)<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020.\u00a0<a href=\"https:\/\/aclanthology.org\/2020.acl-main.493\">Finding Universal Grammatical Relations in Multilingual BERT<\/a>. In\u00a0<em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics<\/em>, pages 5564\u20135577, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Karolina Stanczak, Edoardo Ponti, Lucas Torroba Hennigen, Ryan Cotterell, and Isabelle Augenstein. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.naacl-main.114\">Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models<\/a>. In\u00a0<em>Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies<\/em>, pages 1589\u20131598, Seattle, United States. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Aaron Mueller, Yu Xia, and Tal Linzen. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.conll-1.8\">Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models<\/a>. In\u00a0<em>Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)<\/em>, pages 95\u2013109, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>What multilingual evaluation tells us about the current state of NLP<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022.\u00a0<a href=\"https:\/\/aclanthology.org\/2022.acl-long.376\">Systematic Inequalities in Language Technology Performance across the World\u2019s Languages<\/a>. In\u00a0<em>Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)<\/em>, pages 5486\u20135505, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p><strong>Responsible AI for Multilingual LLMs<\/strong><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Socio-Cultural Aspects<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Ma\u0142gorzata Suszczy\u00b4nska. 1999. Apologizing in english, polish and hungarian: Different languages, different strategies. Journal of Pragmatics, 31(8):1053\u20131065.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, and David R. Mortensen. 2021. Crosscultural similarity features for cross-lingual transfer learning of pragmatically motivated tasks. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2403\u20132414, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Dong Nguyen, A. Seza Do\u02d8gru\u00f6z, Carolyn P. Ros\u00e9, and Franciska de Jong. 2016. Computational Sociolinguistics: A Survey. Computational Linguistics, 42(3):537\u2013593.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Daniel Hershcovich, et al. 2022. Challenges and strategies in crosscultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997\u20137013, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>N\u00e9v\u00e9ol et al. French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English. ACL 2022<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Blodgett et al., Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. ACL-IJCNLP 2021<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>Gender Bias and Grammatical Gender<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical gender. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5276\u20135284<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender bias in multilingual embeddings and cross-lingual transfer. In Proceedings of the 58<sup>th<\/sup> Annual Meeting of the Association for Computational Linguistics, pages 2896\u20132907, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Sheng Liang, Philipp Dufter, and Hinrich Sch\u00fctze. 2020. Monolingual and multilingual reduction of gender bias in contextualized representations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5082\u20135093, Barcelona, Spain (Online). International Committee on Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, and Naoaki Okazaki. 2022. Gender bias in masked language models for multiple languages. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2740\u20132750, Seattle, United States. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>[MITIGATION] Xiaolei Huang. 2022. Easy adaptation to mitigate gender bias in multilingual text classification. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 717\u2013723, Seattle, United States. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>[Model Compression] Jaimeen Ahn, Hwaran Lee, Jinhwa Kim, and Alice Oh. 2022. Why knowledge distillation amplifies gender bias and how to mitigate from the perspective of DistilBERT. In Proceedings of the 4th Workshopon Gender Bias in Natural Language Processing (GeBNLP), pages 266\u2013272, Seattle, Washington. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Zmigrod et al., <a href=\"https:\/\/aclanthology.org\/P19-1161.pdf\">Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology<\/a>, ACL 2019<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>Bias and Fairness<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Jialu Wang, Yang Liu, and Xin Wang. 2022. Assessing multilingual fairness in pre-trained multimodal representations. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2681\u20132695, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Zeerak Talat, et al. 2022. You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Proceedings of BigScience Episode #5 \u2013 Workshop on Challenges &amp; Perspectives in Creating Large Language Models, pages 26\u201341, virtual+Dublin. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Aristides Milios and Parishad BehnamGhader. 2022. An analysis of social biases present in bert variants across multiple languages. ArXiv, abs\/2211.14402.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Cristina Espa\u00f1a-Bonet and Alberto Barr\u00f3n-Cede\u00f1o. 2022. The (undesired) attenuation of human biases by multilinguality. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages \u2013, Online and Abu Dhabi, UAE. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>[Dataset] Ilias Chalkidis, et al., 2022. FairLex: A multilingual benchmark for evaluating fairness in legal text processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4389\u20134406, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>Hate Speech, Toxicity and Sentiment<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and Michael J. Paul. 2020. Multilingual Twitter corpus and baselines for evaluating demographic bias in hate speech recognition. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1440\u20131448, Marseille, France. European Language Resources Association.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Ant\u00f3nio C\u00e2mara, Nina Taneja, Tamjeed Azad, Emily Allaway, and Richard Zemel. 2022. Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic.In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, pages 90\u2013106, Dublin, Ireland. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>Discursive Aspects (Decolonizing RAI)<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021. Re-imagining algorithmic fairness in india and beyond. CoRR, abs\/2101.09995.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. 2022. Recontextualizing fairness in NLP: The case of India. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 727\u2013740, Online only.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Krithika Ramesh, Sunayana Sitaram, Monojit Choudhury. 2023. Fairness in Language Models Beyond English: Gaps and Challenges. Findings of EACL 2023<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>Distributive Aspects<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282\u20136293, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Monojit Choudhury and Amit Deshpande. 2021. How linguistically fair are multilingual pre-trained language models? Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12710\u201312718.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p>General<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT \u201921, page 610\u2013623, New York, NY, USA. Association for Computing Machinery.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Su Lin Blodgett, Solon Barocas, Hal Daum\u00e9 III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of \u201cbias\u201d in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454\u20135476, Online. Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p><strong>Working with Multilingual Language Communities<\/strong><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Abraham, Basil, et al. \"Crowdsourcing Speech Data for Low-Resource Languages from Low-Income Workers.\" Proceedings of the 12th Language Resources and Evaluation Conference. 2020.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Almaliki, Malik, et al. \"ABMM: Arabic BERT-Mini Model for Hate-Speech Detection on Social Media.\" Electronics 12.4 (2023): 1048.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Bird, Steven. \"Decolonising Speech and Language Technology.\" Proceedings of the 28th International Conference on Computational Linguistics. 2020.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Bird, Steven. \"Local Languages, Third Spaces, and Other High-Resource Scenarios.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Chopra, Manu, et al. \"Exploring Crowdsourced Work in Low-Resource Settings.\" Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 2019.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Currin, Christopher Brian, et al. \"A Framework for Grassroots Research Collaboration in Machine Learning and Global Health.\" 2023 ICLR First Workshop on \"Machine Learning &amp; Global Health\". 2023.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Diddee, Harshita, Bali, Kalika, Choudhury, Monojit, Ganu, Tanuja, and Dandapat, Sandipan. 2022. Too Brittle to Touch: Comparing the Stability of Quantization and Distillation towards Developing Low-Resource MT Models. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 870\u2013885, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Diddee, Harshita, Bali, Kalika, Choudhury, Monojit, and Mukhija, Namrata. 2022. The Six Conundrums of Building and Deploying Language Technologies for Social Good. In ACM SIGCAS\/SIGCHI Conference on Computing and Sustainable Societies (COMPASS) (COMPASS '22). Association for Computing Machinery, New York, NY, USA, 12\u201319.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Goswami, Dipam, et al. \"Analysis of Word-level Embeddings for Indic Languages on AI4Bharat-IndicNLP Corpora.\" 2021 IEEE 8th Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON). IEEE, 2021.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Gitau, Catherine, et al. \"Masakhane Web: A Machine Translation Platform for African Languages.\" (2023)<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Harrigan, Atticus, et al. \"Proceedings of the Sixth Workshop on the Use of Computational Methods in the Study of Endangered Languages.\" Proceedings of the Sixth Workshop on the Use of Computational Methods in the Study of Endangered Languages. 2023.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Joshi, Pratik, et al. \"Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities.\" arXiv preprint arXiv:1912.03457 (2019).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Kann, Katharina, et al. \"AmericasNLI: Machine Translation and Natural Language Inference Systems for Indigenous Languages of the Americas.\" Frontiers in Artificial Intelligence 5 (2022): 266.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Koto, Fajri, et al. \"IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP.\" arXiv preprint arXiv:2011.00677 (2020).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Kumar, Ritesh, et al. \"Annotated Speech Corpus for Low Resource Indian Languages: Awadhi, Bhojpuri, Braj and Magahi.\" arXiv preprint arXiv:2206.12931 (2022).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Kunchukuttan, Anoop, et al. \"Ai4bharat-indicnlp Corpus: Monolingual Corpora and Word Embeddings for Indic Languages.\" arXiv preprint arXiv:2005.00085 (2020).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Mehta, Devansh, et al. \"Learnings from Technological Interventions in a Low Resource Language: Enhancing Information Access in Gondi.\" arXiv preprint arXiv:2211.16172 (2022).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Moeller, Sarah, et al. \"Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages.\" Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages. 2022.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Moradshahi, Mehrad, et al. \"X-RiSAWOZ: High-Quality End-to-End Multilingual Dialogue Datasets and Few-shot Agents.\" arXiv preprint arXiv:2306.17674 (2023).<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:list -->\n<ul><!-- wp:list-item -->\n<li>Muhammad, Shamsuddeen Hassan, et al. \"Naijasenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis.\" arXiv preprint arXiv:2201.08277 (2022).<\/li>\n<!-- \/wp:list-item -->\n\n<!-- wp:list-item -->\n<li>AS Do\u011fru\u00f6z and S Sitaram, <a href=\"https:\/\/scholar.google.co.in\/citations?view_op=view_citation&amp;hl=en&amp;user=PUxwYrkAAAAJ&amp;sortby=pubdate&amp;citation_for_view=PUxwYrkAAAAJ:GnPB-g6toBAC\">Language technologies for low resource languages: sociolinguistic and multilingual insights<\/a>, LREC 2022.<\/li>\n<!-- \/wp:list-item --><\/ul>\n<!-- \/wp:list -->\n\n<!-- wp:paragraph -->\n<p><\/p>\n<!-- \/wp:paragraph -->\n<!-- \/wp:msr\/content-tab -->\n<!-- \/wp:msr\/content-tabs -->","tab-content":[],"msr_startdate":"2023-07-09","msr_enddate":"2023-07-09","msr_event_time":"Eastern Daylight Time (UTC -4)","msr_location":"Toronto, Canada","msr_event_link":"","msr_event_recording_link":"","msr_startdate_formatted":"July 9, 2023","msr_register_text":"Watch now","msr_cta_link":"","msr_cta_text":"","msr_cta_bi_name":"","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/06\/ACL_Conference_Banner_1920x720_C-960x540.jpg\" class=\"img-object-cover\" alt=\"abstract shapes\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/06\/ACL_Conference_Banner_1920x720_C-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/06\/ACL_Conference_Banner_1920x720_C-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/06\/ACL_Conference_Banner_1920x720_C-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/06\/ACL_Conference_Banner_1920x720_C-343x193.jpg 343w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/06\/ACL_Conference_Banner_1920x720_C-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2023\/06\/ACL_Conference_Banner_1920x720_C-1280x720.jpg 1280w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","event_excerpt":"Date\/time: July 9, 2023 | 9:00 AM - 12:30 PM Location: Metropolitan West, Westin Harbour Castle, Toronto, Canada This tutorial workshop is co-located with\u00a0ACL 2023 (opens in new tab). Tutorial Slides The technology landscape is being rapidly transformed by Large Language Models (LLMs), allowing users to address real-world applications in various domains. However, a&nbsp;digital divide&nbsp;exists that may exclude large populations from benefiting and contributing to this technological revolution due to factors such as language, income,&hellip;","msr_research_lab":[199562],"related-researchers":[{"type":"user_nicename","display_name":"Sunayana Sitaram","user_id":37287,"people_section":"Organizing committee","alias":"susitara"},{"type":"user_nicename","display_name":"Kalika Bali","user_id":32477,"people_section":"Organizing committee","alias":"kalikab"},{"type":"user_nicename","display_name":"Barun Patra","user_id":39099,"people_section":"Organizing committee","alias":"bapatra"},{"type":"guest","display_name":"Kabir Ahuja","user_id":949593,"people_section":"Organizing committee","alias":""}],"msr_impact_theme":["Computing foundations"],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-opportunities":[],"related-publications":[],"related-videos":[],"related-posts":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event\/950022"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-event"}],"version-history":[{"count":8,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event\/950022\/revisions"}],"predecessor-version":[{"id":954438,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event\/950022\/revisions\/954438"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/947454"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=950022"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=950022"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=950022"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=950022"},{"taxonomy":"msr-video-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-video-type?post=950022"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=950022"},{"taxonomy":"msr-program-audience","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-program-audience?post=950022"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=950022"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=950022"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}