MEGA: Multi-lingual Evaluation of Generative AI

Generative AI models have impressive performance on many Natural Language Processing tasks such as language understanding, reasoning and language generation. One of the most important questions that is being asked by the AI community today is about the capabilities and limits of these models, and it is clear that evaluating generative AI is very challenging. Most studies on generative Large Language Models (LLMs) are restricted to English and it is unclear how capable these models are at understanding and generating other languages. We present the first comprehensive benchmarking of generative LLMs – MEGA, which evaluates models on standard NLP benchmarks, covering 8 diverse tasks and 33 typologically diverse languages. We also compare the performance of generative LLMs to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs. We present a thorough analysis of the performance of models across languages and discuss some of the reasons why generative LLMs are currently not optimal for all languages. We create a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.

Presentation slides

Speaker Details

Kabir Ahuja (opens in new tab)

I am Kabir Ahuja. I am currently a Research Fellow at Microsoft Research India (MSRI), where I am supervised by Dr. Sunayana Sitaram and Dr. Monojit Choudhury . I primarily work on understanding, improving and effectively evaluating Cross Lingual Transfer in Pretrained Multilingual Language Models. My research vision is about creating multilingual models that can effectively serve all of the world’s languages while requiring only a little explicit supervision on low resource languages. I also collaborate with Dr. Navin Goyal at MSRI on studying the properties of self-attention in Transformers and the types of functions that can be expressed and learned by these networks.

Millicent Ochieng

Millicent Ochieng is a Data & Applied Scientist at the Microsoft Africa Research Institute (MARI), specializing in Natural Language Processing (NLP), Machine Learning (ML), and Computer Vision. With a passion for leveraging these technologies to address pressing real-world challenges, she explores their practical applications in the domains of healthcare, the future of work, society, and sustainability. Currently, Millicent is actively working on multilingual evaluations of Large Language Models (LLMs) and enhancing Automatic Speech Recognition (ASR) systems for the Swahili language.

Date:
Speakers:
Kabir Ahuja, Millicent Ochieng
Affiliation:
Microsoft Research India, Microsoft Africa Research Institute

Watch Next