Diffusion probabilistic models have the capacity to generate high-fidelity samples for generative time series forecasting. However, they also present issues of instability due to their stochastic nature. In order to tackle this challenge, researchers from Microsoft Research Asia introduce a novel approach called “MG-TSD”. The paper “MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process (opens in new tab)”, presented at ICLR 2024, capitalizes on the intrinsic granularity levels present in the data, utilizing them as predefined targets at various stages of the diffusion process. The aforementioned targets are employed to guide the learning trajectory of the diffusion models, thereby ensuring a more stable and accurate forecast.

It is noteworthy that  the MG-TSD method yields remarkable outcomes without the necessity of additional data. In the field of long-term forecasting, researchers have established a new state-of-the-art methodology that demonstrates a notable relative improvement across six benchmarks with improvement ranging from 4.7% to 35.8%.

Guiding diffusion processes through intrinsic data granularities features in time series data

It can be observed that the forward process of the diffusion model, which sequentially corrupts the data distribution to a standard normal distribution, intuitively aligns with the process of smoothing fine-grained data into a coarser-grained representation. Both of these processes result in a gradual loss of finer distribution features. This suggests that intrinsic features within data granularities may also serve as a source of guidance.

The MG-TSD model employs multiple granularity levels within data to guide the learning process of diffusion models. The coarse-grained data at different granularity levels are utilized as targets to guide the learning of the denoising process. These targets serve as constraints for the intermediate latent states, ensuring a regularized sampling path that preserves the trends and patterns within the coarse-grained data. The introduction of inductive bias facilitates the generation of coarser features during intermediate steps and facilitates the recovery of finer features in subsequent diffusion steps. Each granularity level can guide the diffusion process through different steps. When implementing this, both the coarse-grained data and the finest-grained data share different percentages of the variance schedule (a hyperparameter of the diffusion model), referred to as the “share ratio.” Consequently, this design reduces variability and results in high-quality predictions.

MG-TSD achieves stable and outstanding prediction results

A comprehensive evaluation was conducted across six benchmarks and three performance metrics, in which nine baseline models were compared. The results demonstrate that the MG-TSD model achieves state-of-the-art (SOTA) status, with a substantial improvement ranging from 4.7% to 35.8% on the CRPS_sum metric across the six benchmarks. CRPS_sum indicates the similarity between two distributions; the smaller the value, the more similar they are.

Diffusion process aligns with data smoothing

The four subplots from Figure3(a) to Figure3(d) illustrate a gradual smoothing transformation of the distribution of increasingly coarser targets. The blue curve represents CRPS_sum values of coarse-grained targets and intermediate samples of the single-granularity diffusion model (1h) at each denoising step. As granularity transitions to coarser from left to right panels (4h→6h→12h→24h), targets progressively align with intermediate sample distributions at smaller denoising steps (approximately at steps 80→60→40→40). This comparison underscores the similarity between the diffusion process and the smoothing process, which transition from the finest-grained data to coarse-grained data. Both processes entail a gradual loss of finer characteristics from the finest-grained data through a smooth transformation.

Moreover, this observation is consistent with our selection of guiding steps for MG-TSD. The orange lines, which depict the performance of MG-TSD with different share ratios ranging from [0.2, 0.4, 0.6, 0.8, 1.0], and the blue lines, which represent the similarity of distributions, show a consistent trend. In other words, setting the guiding steps at a coarse granularity to match the steps where the diffusion intermediate samples have the closest distribution often achieves the best performance, as indicated by the grey region in the figure.

Coarse-grained samples demonstrate superior robustness in trend-capturing capabilities

Researchers visualize the ground truth and the predicted mean for both 1-hour and 4-hour granularity time series across four dimensions in the Solar dataset, as illustrated in Figure 4. In the MG-TSD model, the coarse-grained samples display a more robust capacity to capture the trends, subsequently guiding the generation of more precise fine-grained data.

For further information regarding MG-TSD, please refer to the project page at: https://github.com/Hundredl/MG-TSD (opens in new tab)

The post MG-TSD: Advancing time series analysis with multi-granularity guided diffusion model appeared first on Microsoft Research.

“Agents are a very, very powerful abstraction over things like task decomposition, specialization, tool use, etc. Really, you think about which roles you need on your team, and you put together your team of agents, and you get them to talk to one another, and then you start making progress on your task.”

– Adam Fourney, Principal Researcher, Microsoft Research AI Frontiers

Related resources

The post AutoGen Update: Complex Tasks and Agents appeared first on Microsoft Research.

“Materials design is the cornerstone of modern technology. Many of the challenges our society is facing today are bottlenecked by finding a good material. … If we can find a novel material that conducts lithium very well, it will be a key component for our next-generation battery technology. The same applies to many other domains.”

– Tian Xie, Principal Research Manager, Microsoft Research AI for Science

Related resources

The post MatterGen: A Generative Model for Materials Design appeared first on Microsoft Research.

There is “a substantial demand for advanced generative AI tailored to enhance core business operations. However, in our dialogues with strategic partners, we have identified crucial gaps in current generative AI capabilities versus the specific needs of industry applications. … Our research is crucial in addressing these limitations and amplifying the underappreciated potential of generative AI.”

– Jiang Bian, Senior Principal Research Manager, Microsoft Research Asia

“Foundation models, also known as large language models, possess immense potential across a variety of industries. Yet, some companies and organizations limit their use of these expansive AI models to niche areas, including intelligent customer service, chatbots, or text and image generation. In reality, these foundation models demonstrate robust abilities in reasoning, content creation, and generalization, making them exceptionally fit for high-stakes business tasks. These tasks range from taking accurate prediction and forecasting, optimizing industrial control and complex decision-making, and conducting intelligent and interactive industrial simulations.”

— Jiang Bian, Senior Principal Research Manager, Microsoft Research Asia

---

Maximizing the business value of foundation models in industry applications

By Jiang Bian

As the development of large AI models, also known as foundation models, progresses, companies and organizations are becoming increasingly excited about their potential for enhancing productivity. However, a significant trend has been observed: many industry practitioners focus heavily on the human-like qualities of AI, such as conversational abilities, writing skills, creativity, and perceptual capabilities. In deploying these large AI models, there is a tendency to prioritize applications in intelligent customer service, chatbots, and other so-called ”human-like” functions. Unfortunately, this emphasis may restrict our comprehension and use of these potent models, hindering our ability to fully unleash their capabilities within various industries.

This limitation is not without reason. Incorporating foundation models into practical, production-oriented scenarios is still in its infancy, with few mature and widespread examples to follow. Viewing AI as a “production tool” is akin to possessing a tool before fully understanding its potential applications. Furthermore, humanity has rarely, if ever, encountered such a versatile yet uncertain tool that is not designed for specific tasks.

Additionally, the complexity and variety inherent in different industries require foundation models that move beyond traditional perceptions. This necessitates synchronized innovation in models at the industry level, enabling them to fully exploit the capabilities of foundation models across diverse industrial landscapes and to better align with AI applications. Instead of limiting AI to a “chat robot” role, we should broaden our perspective. Transforming industries in the AI era involves rethinking current business processes and frameworks, leading to collaborative models that can effortlessly integrate humans and foundation models.

Unlocking the boundless potential of foundation models in industry

Foundation models are endowed with broad capabilities in data representation, knowledge comprehension, and reasoning, allowing them to adjust seamlessly across various domains and scenarios, and swiftly adapt to new environments. Concurrently, digital platforms across industries have evolved, amassing substantial amounts of industry-specific data. This rich repository of knowledge and information positions foundation models to integrate effortlessly into industrial settings.

In practical terms, the advanced reasoning abilities of foundation models provide users with a deeper understanding of data. By extracting valuable insights from large datasets and identifying patterns and correlations, these models deliver more effective recommendations and deeper insights. This benefit is especially vital in industrial contexts, where prediction, decision-making, and simulation play crucial roles.

One of the standout features of foundation models is their exceptional ability to generalize. Before their advent, each industry scenario required specific data to train bespoke AI models, limiting scalability and hindering the full commercial exploitation of AI. Foundation models, with their access to a global pool of knowledge, markedly improve generalization. As a result, industries are freed from the necessity of developing unique models for every situation, overcoming a major limitation of traditional AI solutions.

Moreover, foundation models can work in tandem with generative AI to increase the accuracy, realism, and interactivity of industrial simulations and intelligent modeling, facilitating the creation of digital twins. These simulations and models aim to mimic and test real-world scenarios, which often involve complex roles and intricate environments. Traditional AI models may simplify real-world complexities or miss crucial extreme events, compromising the fidelity and authenticity of simulations. In contrast, generative large AI models, steeped in domain-specific knowledge, establish accurate mappings between specific data dimensions and real-world occurrences. This method allows for simulations that closely mirror reality, significantly aiding industrial forecasting and decision-making processes while maintaining adherence to industry standards.

Navigating four key challenges in implementing foundation models for industry

In the industrial sector, tasks of paramount importance and commercial value include precise forecasting and control, efficient optimization of decisions, and complex duties associated with intelligent and interactive industrial simulations. These areas should be the primary focus for traditional industrial enterprises. Yet, when assessing existing foundation models like GPT and the actual needs within industrial domains, we uncover significant mismatches between the capabilities of these models and the real demands of industry. To bridge this gap and fully leverage their potential, several challenges must be addressed.

First, there is a notable absence of a universal framework capable of effectively extracting complex domain knowledge from diverse field data and using this knowledge to construct intelligent agents. Various domains contain rich and complex data, such as logistics companies dealing with customs information and cross-national policies, pharmaceutical industries with FDA drug review documents, and the legal industry with numerous regulations. Developing intelligent agents that are deeply rooted in domain knowledge calls for a more generalized framework. This framework should be proficient in extracting crucial domain knowledge, identifying hidden connections between data and knowledge, and managing this information efficiently.

Second, while foundation models are adept at generating textual content, their proficiency in processing and understanding structured data, like numerical or tabular information, is lacking. Industrial scenarios often involve structured data, such as health monitoring indicators, battery charge-discharge cycles, and financial transactions. Current large models are not specifically designed or optimized for processing such data, which complicates accurate prediction and classification tasks based on structured inputs.

Third, in practical applications, foundation models currently fall short in stability and reliability for decision-making. Critical industries like energy, logistics, finance, and healthcare require dependable decision-making for tasks such as optimizing logistics routes, controlling energy equipment, formulating investment strategies, and allocating medical resources. These tasks often involve numerous variables and constraints, especially under dynamic environmental changes. Foundation models have yet to fully adapt to these complex industrial tasks, making direct application challenging.

Lastly, there is a lack of insight into domain-specific foundational data, as well as methodologies and experience for developing domain-specific foundation models. Essential information in many specialized fields extends beyond mere text, incorporating unique data structures and semantic relationships. For example, transaction order information in the financial investment field or molecular structure details in the biopharmaceutical industry contain critical knowledge often embedded in such foundational data. A deeper, more nuanced analysis is required. Creating domain-specific foundation models grounded in this detailed understanding is crucial for effectively leveraging and unlocking the potential of data in these fields.

Constructing industry foundation models: harmonizing general knowledge and domain expertise

To expedite the adoption and application of foundation models in industry, we can concentrate on several pivotal areas.

First, we can harness rich and complex industrial domain data to construct a more versatile, efficient, and practical retrieval-augmented generation (RAG) framework. This framework is designed to adapt seamlessly to various vertical domains, extracting essential domain knowledge, uncovering hidden associations between data and knowledge, and effectively organizing and managing this wealth of information.

Second, by carefully considering critical numerical data and the corresponding structured dependencies prevalent in industrial scenarios, we can design foundation models specifically optimized for industrial applications. These models effectively integrate general knowledge with domain-specific expertise derived from temporal or tabular data, thereby enabling more effective solutions for tasks such as prediction and classification within the industry.

Another avenue we are actively exploring involves harnessing the potent generation, generalization, and transfer capabilities inherent in foundation models to elevate the quality and efficiency of industrial decision-making. We are pursuing two distinct paths: first, treating foundation models as intelligent agents, and; second, leveraging foundation models to assist reinforcement-learning agents.

Treating foundation models as intelligent agents: By leveraging the pre-existing knowledge encoded in foundation models and integrating offline reinforcement learning, we can continuously acquire new domain-specific insights and fine-tune the models. This evolutionary process enhances the optimization and decision-making capabilities of foundation models, enabling them to prioritize industry-specific tasks.

Foundation models optimized for specific tasks can play a pivotal role across various industrial contexts. In formula racing, for example, these foundation models can optimize tire-maintenance strategies. By considering tire wear and repair costs, they determine the optimal pit stop timing, thereby shortening race duration and improving car rankings. In chemical manufacturing, leveraging these foundation models can significantly enhance efficiency in product storage and pipeline coordination during production processes, ultimately boosting overall production-execution efficiency. Furthermore, due to their generalization capabilities and robustness, foundation models can be swiftly adapted to optimize air conditioning control, ensuring comfortable temperatures while minimizing energy consumption.

Assisting reinforcement learning agents with foundation models: We can empower models to acquire universal representations that rapidly adapt to diverse environments and tasks, thereby enhancing their generalization capabilities. In this approach, we introduce a pre-trained world model that emulates human learning and decision-making processes, ultimately bolstering industrial decision-making. By harnessing a pre-trained world model with extensive knowledge and adopting a two-stage pre-training framework, developers can comprehensively and flexibly train foundation models for industrial decision-making, extending their applicability to any specific decision scenario.

We partnered with the Microsoft Xbox team to rigorously validate the effectiveness of our framework in game-testing scenarios. By harnessing this framework, we pre-trained a specialized world model tailored for game maps. This model directly tackles the challenge of long-term spatial reasoning and navigation, leveraging landmark observations within novel game environments. The results were remarkable: our pre-trained model significantly outperformed counterparts that lacked a world model or relied on traditional learning methods. As a result, game exploration efficiency was greatly enhanced.

Moreover, we can harness domain-specific foundational data and the precise semantic information it encapsulates to develop foundation models within the domain, thereby unlocking novel opportunities for intelligent, interactive decision-making, and simulation. For example, by analyzing transactional data from financial markets, we can construct robust investment models. These foundational datasets extend beyond mere textual characters; they embody intricate semantic structures and valuable information. Leveraging this financial foundation model, we can generate customized order flows for various market styles, simulate large-scale order transactions across diverse market environments, and conduct controlled experiments in the financial investment landscape. This approach empowers us to gain deeper insights into market fluctuations and devise strategies for extreme scenarios.

Foundation models propel the next industrial digital transformation

Microsoft Research Asia has long recognized that the widespread adoption of AI in industry necessitates continuous technological exploration, experimentation, and breakthroughs. Through collaborative efforts with partners across various industries, we have developed open-source models, including the Qlib AI quantitative investment platform, the MARO multi-agent resource optimization platform, the FOST spatial-temporal prediction tool, and the BatteryML battery performance analysis and prediction platform. These industry-oriented AI platforms, tools, and models not only play a pivotal role in industry but also serve as critical data and foundational components for implementing cutting-edge foundation models.

Building upon successful experiences in industrializing AI, we have embarked on the exploration of domain-specific foundation models tailored for industry, drawing from the dimensions previously discussed. Our findings reveal that these foundation models possess significant potential to diverge from conventional large-scale model paradigms and profoundly impact industrial transformation.

Envision a future where foundation models empower knowledge management, extraction, and iterative processes across industries. Furthermore, we are actively investigating how foundation models can support companies in achieving automated research and development (R&D). This encompasses tasks such as automatically identifying R&D directions, generating algorithmic research proposals, automating R&D processes and scientific experiments, and iteratively refining research approaches. In essence, AI will autonomously propel data-centric industrial R&D, fundamentally revolutionizing industry operations.

Foundation models are poised to become the driving force behind industrial digital transformation, mirroring the transformative impact of the internet and cloud computing. These models are set to unleash a new wave of industrial innovation. We eagerly anticipate collaborating with additional industry partners, immersing ourselves in real-world scenarios, and exploring diverse applications for foundation models within the industrial landscape, thereby fully unlocking their commercial potential.

---

Author

Dr. Jiang Bian currently serves as a senior principal research manager at Microsoft Research Asia. He leads the Machine Learning Group and the Industry Innovation Center at Microsoft Research Asia.

His team’s research spans deep learning, reinforcement learning, and privacy computing, with a focus on cutting-edge applications of AI in vertical domains such as finance, energy, logistics, manufacturing, healthcare, and sustainable development.

Dr. Jiang Bian has authored over a hundred research papers published in top-tier international conferences and journals. Also, he holds several U.S. patents. Dr. Jiang actively contributes to the academic community by serving on program committees for various prestigious international conferences and acting as a reviewer for leading international journals. In recent years, Dr. Jiang’s team has made significant strides in applying AI-based prediction and optimization techniques to critical scenarios across diverse fields, such as finance, logistics, and healthcare. Furthermore, they have generously shared relevant technologies and frameworks with the open-source community.

Dr. Jiang Bian completed his undergraduate studies at Peking University, earning a bachelor’s degree in computer science. He then pursued further studies at the Georgia Institute of Technology in the United States, where he obtained his Ph.D. in computer science.

Related resources

The post Driving Industry Evolution: Exploring the Impact of Generative AI on Sector Transformation appeared first on Microsoft Research.

“Today’s AI models hold incredible potential for assisting the blind community—from text recognition to object identification to question answering. Apps like Seeing AI are already deploying some of these AI features. But there is potential for much more.”

– Daniela Massiceti, Senior Researcher, Microsoft Research Cambridge

Related resources

The post Insights into the Challenges and Opportunities of Large Multi-Modal Models for Blind and Low Vision Users: A Case Study on CLIP appeared first on Microsoft Research.

“One of the solutions that we’ve been using is to actually design with ‘human in the loop’ in mind because we know that these technologies are not perfect. And so, we really want to figure out ways in which humans and AI systems can work together in order to create the most effective outcome.”

– Sunayana Sitaram, Principal Researcher, Microsoft Research India

Related resources

The post Panel Discussion: Generative AI for Global Impact: Challenges and Opportunities appeared first on Microsoft Research.

“It’s only by working together can we solve the challenges we face with generative AI at scale and, in doing so, capitalize on the opportunities these new technologies offer. … Now is the time to change the dialogue and change the direction of these new powerful technologies to ensure that they are globally equitable by design.”

– Jacki O’Neill, Lab Director, Microsoft Research Africa, Nairobi

Related resources

The post Keynote: Building Globally Equitable AI appeared first on Microsoft Research.

[Music]

Sunayana Sitaram: Our ultimate goal is to build evaluation systems and also other kinds of systems in general where humans and LLMs can work together. We’re really trying to get humans to do the evaluation, get LLM’s to do the evaluation, use the human data in order to improve the LLM. And then just this continues in a cycle. And the ultimate goal is, send the things to the LLM that it’s good at doing and send the rest of the things that the LLM can’t do to humans who are like the ultimate authority on the evaluation.

Sridhar Vedantham: Welcome to the Microsoft Research India podcast, where we explore cutting-edge research that’s impacting technology and society. I’m your host, Sridhar Vedantham.

[Music]

Sridhar Vedantham: LLM’s are perhaps the hottest topic of discussion in the tech world today. And they’re being deployed across domains, geographies, industries and applications. I have an extremely interesting conversation with Sunayana Sitaram, principal researcher at Microsoft Research about LLMs, where they work really well and also challenges that arise when trying to build models with languages that may be under resourced. We also talk about the critical work she and her team are doing in creating state-of-the-art methods to evaluate the performance of LLMs, including those LLMs that are based on Indic languages.

Related

• Microsoft Research India Podcast: More podcasts from MSR India
• iTunes (opens in new tab): Subscribe and listen to new podcasts on iTunes
• Android (opens in new tab)
• RSS Feed (opens in new tab)
• Spotify (opens in new tab)
• Google Podcasts (opens in new tab)
• Email (opens in new tab)

[Music]

Sridhar Vedantham: Sunayana, welcome to the podcast.

Sunayana Sitaram: Thank you.

Sridhar Vedantham: And I’m very excited to have you here because we get to talk about a subject that seems to be top of mind for everybody right now. Which is obviously LLMs.  And what excites me even more is I think, we’re going to be talking about LLMs in a way that’s slightly different from what the common discourse is today, right?

Sunayana Sitaram: That’s right.

Sridhar Vedantham: OK. So before we jump into it, why don’t you give us a little bit of background about yourself and how you came to be at MSR?

Sunayana Sitaram: Sure. So it’s been eight years now since I came to MSR. I came here as a postdoc after finishing my PhD at Carnegie Mellon. And so yeah, it’s been around 15 years now for me in the field, and it’s been super exciting, especially the last few years.

Sridhar Vedantham: So, I’m guessing that these eight years have been interesting, otherwise we won’t be having this conversation. What areas of research, I mean, have you changed course over the years and how is that progressed?

Sunayana Sitaram: Yeah, actually, I’ve been working pretty much on the same thing for the last 15 years or so. So I’ll describe how I got started. When I was an undergrad, I actually met the principal of a blind children’s school who himself was visually impaired. And he was talking about some of the technologies that he uses in order to be independent. And one of those was using optical character recognition and text to speech in order to take documents or letters that people sent him and have them read out without having to depend on somebody. And he was in Ahmedabad, which is where I grew up. And his native language was Gujarati.  And he was not able to do this for that language. Whereas for English, the tools that he required to be independent were available. And so, he told me like it would be really great if somebody could actually build this kind of system in Gujarati. And that is when it sort of it was like a, you know, aha moment for me. And I decided to take that up as my undergrad project. And ever since then, I’ve been trying to work on technologies trying to bridge that gap between English and other languages- under resourced languages. And so, since then, I’ve worked on very related areas. So, my PhD thesis was on text to speech systems for low resource languages. And after I came to MSR I started working on what is called code switching, which is a very common thing that multilinguals all over the world do. So they use multiple languages in the same conversation or sometimes even in the same sentence. And so you know, this was a project called Project Melange that was started here and that really pioneered the code switching work in the research community in NLP. And after that it’s been about LLMs and evaluation but again from a multilingual under resource languages standpoint.

Sridhar Vedantham: Right. So I have been here for quite a while at MSR myself and one thing that I always heard is that there is this in general, a wide gulf in terms of the resources available for a certain set of languages to do say NLP type work. And the other languages is just the tail, it’s a long tail, but the tail just falls off dramatically. So, I wanted you to answer me in a couple of ways. One is, what is the impact that this generally has in the field of NLP itself and in the field of research into language technologies, and what’s the resultant impact on LLMs?

Sunayana Sitaram: Yeah, that’s a great question. So, you know the paradigm has shifted a little bit after LLM’s have come into existence. Before this, so this was around say a few years ago, the paradigm would be that you would need what is called unlabeled data. So, that is raw text that you can find on the web, say Wikipedia or something like that, as well as labeled data. So, this is something that a human being has actually sat and labeled for some characteristic of that text, right? So these are the two different kinds of texts that you need if you want to build a text based language model for a particular language. And so there were languages where, you know, you would find quite a lot of data on the web because it was available in the form of documents or social media, etc. for certain languages. But nobody had actually created the labeled resources for those languages, right? So that was the situation a few years ago. And you know the paradigm at that time was to use both these kinds of data in order to build these models, and our lab actually wrote quite a well-regarded paper called, ‘The State and Fate of Linguistic Diversity and Inclusion’, where they grouped different languages into different classes based on how much data they had labeled, as well as unlabeled.

Sridhar Vedantham: Right.

Sunayana Sitaram: And it was very clear from that work that, you know only around 7 or 8 languages of the world actually can be considered to be high resource languages which have this kind of data. And most of the languages of the world spoken by millions and millions of speakers don’t have these resources. Now with LLMs, the paradigm changed slightly, so there was much less reliance on this labeled data and much more on the vast amount of unlabeled data that exists, say, on the web. And so, you know, we were wondering what would happen with the advent of LLMs now to all of the languages of the world, which ones would be well represented, which ones wouldn’t etc. And so that led us to do, you know, the work that we’ve been doing over the last couple of years. But the story is similar, that even on the web some of these languages dominate and so many of these models have, you know, quite a lot of data from only a small number of languages, while the other languages don’t have much representation.

Sridhar Vedantham: OK. So, in real terms, in this world of LLMs that we live in today, what kind of impact are we looking at? I mean, when you’re talking about inequities and LLMs and in this particular field, what’s the kind of impact that we’re seeing across society?

Sunayana Sitaram: Sure. So when it comes to LLMs and language coverage, what we found from our research is that there are a few languages that LLMs perform really well on. Those languages tend to be high resource languages for which there is a lot of data on the web and they also tend to be languages that are written in the Latin script because of the way the LLMs are designed currently with the tokenization. And the other languages, unfortunately there is a large gap between the performance in English and other languages, and we also see that a lot of capabilities that we see in LLMs in English don’t always hold in other languages. So a lot of capabilities, like really good reasoning skills, etc, may only be present in English and a few other languages, and they may not be seen in other languages. And this is also true when you go to smaller models that you see that their language capabilities fall off quite drastically compared to the really large models that we have, like the GPT 4 kind of models. So when it comes to real world impact of this, you know, if you’re trying to actually integrate one of these language models into an application and you’re trying to use it in a particular language, chances are that you may not get as good performance in many languages compared to English. And this is especially true if you’re already used to using these systems in English and you want to use them in a second language. You expect them to have certain capabilities which you’ve seen in English, and then when you use them in another language, you may not find the same capabilities. So in that sense, I think there’s a lot of catching up to do for many languages. And the other issue also is that we don’t even know how well these systems perform for most languages of the world because we’ve only been able to evaluate them on around 50 to 60 or maybe 100 languages. So for the rest of the 6000ish languages of the world, many of which don’t even have a written form, most of which are not there on the web. We don’t even know whether these language models are, you know, able to do anything in them at all. So I think that is another, you know, big problem that is there currently.

Sridhar Vedantham: So, if you want to change the situation where we say that you know even if you’re a speaker of a language that might be small, maybe say only two million speakers as opposed to a larger language that might have 100 million or 200 million speakers. How do we even go about addressing inequities like that because at some level it just seems unfair, that for no fault of their own, you know, large sections of population could be excluded from the benefits of LLM’s, right? Because there could be any number of languages in which the number of speakers might be, say, 1,000,000 or 100,000.

Sunayana Sitaram: Right. I think that’s a very hard question. How to actually involve language communities into our efforts, but do that at scale, so that we can actually involve all language communities, all cultures, etc. into the whole building process. So we’ve had some success with doing this with some language communities. So there is a project called ELLORA in MSR India that you know Kalika leads where you know they work with specific language communities, try to understand what the language communities actually need and then try to co-build those resources with them. And so you know in that sense, you know, working directly with these language communities, especially those that have a desire to build this technology and can contribute to some of the data aspects, etc. That’s definitely one way of doing things. We’ve also done some work recently where we’ve engaged many people in India in trying to contribute resources in terms of cultural artifacts and also evaluation. And so you know, we’re trying to do that with the community itself, with the language community that is underrepresented in these LLMs, but doing that at scale is the challenge to try and really bring everyone together. Another way of course, is just raising awareness about the fact that this issue exists, and so I think our work over the last couple of years has really, you know, moved the needle on that. So we’ve done the most comprehensive multilingual evaluation effort that exists both within the large models as well as across different sized models which we call Mega and Megaverse respectively.

Sridhar Vedantham: So if I can just interrupt here, what I’d like is if you could, you know, spend a couple of minutes maybe talking about what evaluating an LLM actually means and how do you go about that?

Sunayana Sitaram: Sure. So when we talk about evaluating LLM’s right, there are multiple capabilities that we expect LLMs to possess. And so, our evaluation should ideally try to test for all of those different kinds of capabilities. So, this could be the ability to reason, this could be the ability to produce output that actually sounds natural to a native speaker of the language. It could be completing some particular task, it could be not hallucinating or not making up things. And also of course, responsible AI, you know, metrics. So things like, you know, being safe and fair, no bias, etc. Only if all of those things work in a particular language, can you say that, that LLM actually works for that language. And so there are several dimensions that we need to consider when we are evaluating these LLMs.

[Music]

Sridhar Vedantham: Before I interrupted you, you were talking about certain projects that you were working on, which are to do with evaluating LLMs, right? I think there’s something called Mega there’s something called Megaverse. Could you tell us a little bit about those and what exactly they do?

Sunayana Sitaram: Sure. So Mega project we started when ChatGPT came out basically. And the question that we were trying to answer was how well these kinds of LLMs perform on languages of the world. So with Mega what we did was, we took already existing open source benchmarks that tested for different kinds of capabilities. So some of them were question-answering benchmarks. Some of them were testing for whether it can summarize text properly or not. Some of them were testing for other capabilities like reasoning etc. And we tested a bunch of models across all of these benchmarks and we covered something like 80 different languages across all these benchmarks. And our aim with Mega was to figure out what the gap was between English and other languages for all of these different tasks, but also what the gap was between the older models, so the models pre LLM and LLMs. Whether we’ve become better or worse in terms of linguistic diversity and performance on different languages in the new era of LLMs or not. And that was the aim with Mega.

Sridhar Vedantham: Sorry, but what was the result of it? Have we become better or not?

Sunayana Sitaram: Yeah. So we have mixed results. So for certain languages, we are doing quite well, but for some languages, unfortunately the larger models don’t do as well as some of the older models used to do. And the older models used to be specialized and trained on labeled data as I said in the beginning, right? And that would help them also be better at all the languages under consideration, whereas with the LLMs we were not really using labeled data in a particular language to train them. And so, we found that, you know, in some cases, the performance of English had shot up drastically and so the gap between English and other languages had also increased in the LLM’s case.

Sridhar Vedantham: Ok. So, the performance of English, I mean the LLMs, that’s much better than what there was earlier, but the other languages didn’t manage to show the same performance increase.

Sunayana Sitaram: That’s right. They didn’t always show. Some of them did, some of the higher resource languages written in the Latin script, for example, did perform quite well, but some of the others did not.

Sridhar Vedantham: OK. And after Mega, then what happened?

Sunayana Sitaram: Yes. So with Mega, we were primarily trying to evaluate the GPT family of models with the older generation of models as I mentioned. But then we realized by the time we finished the work on Mega, there was a plethora of models that came out. So there’s Llama and, you know other models by competitors as well as smaller models, the SLMs, you know, like the Llama sized models, Mistral etc, right. So, there were all of these different models. And then we wanted to see across different models, especially when you’re trying to compare larger models with smaller models, how do these trends look? And that is what we call Megaverse, where we do all of these different evaluations, but not just for the GPT family, but across different models. And what we found in Megaverse were the trends were similar that there were some languages that were doing well, some of the other lower resource languages, especially the ones written in other scripts, were not doing so well. So, for example, the Indian languages were not doing very well across the board. But we also found that the larger frontier models, like the GPT model, they were doing much better than the smaller models for multilingual. And this is again something that you know was shown for the first time in our work that there is this additional gap when you have this large model and small model and there are important practical implications of this. So, say you’re trying to integrate the small model into your workflow as a startup or something like that in a particular language then because it is cheaper it is much more cost efficient, etc, you may not get the same performance in non-English languages as you would get with the larger model right? So that has an impact in the real world.

Sridhar Vedantham: Interesting. And how do you draw this line between what constitutes a large language model and what constitutes a small language model? And I’m also increasingly hearing of this thing called a tiny language model.

Sunayana Sitaram: That’s right. Yeah. So the large language models are the GPTs, the Geminis, you know, those kinds of models. Everything else, we just club as a smaller language model. We don’t really draw a line there. I haven’t actually seen any tiny models that do very well on multilingual. They’re tiny, because they are, you know, trained on a smaller set of data, they have fewer parameters and typically we haven’t seen too many multilingual tiny models so we haven’t really evaluated those. Although there is a new class of models that have started coming up, which are language specific models. So, for example a lot of the Indic model developers have started building specialized model for one language or a small family of languages.

Sridhar Vedantham: OK, so going back to something you said earlier, how do these you know kind of models that people are building for specific Indian languages actually work or perform, given that, I think we established quite early in this podcast that, these are languages that are highly under resourced in terms of data to build models.

Sunayana Sitaram: That’s right. So I think it’s, it’s not just a problem of them being under resourced, it’s also that the proportion of data in the model for a particular language that is not English, say Hindi or Malayalam or Kannada, is very tiny compared to English. And so there are ways to actually change this by doing things with the model after it has been trained. So this is called fine tuning. So what you could do is you could take, say, an open source model which is like a medium sized or a small model and then you could fine tune it or specialize it with data in a particular language, and that actually makes it work much better for that particular language because the distributions shift towards the language that you’re targeting. And so, it’s not just, you know, about the amount of data, but also the proportion of data and how the model has been trained in these giant models that cover hundreds of languages in a single model versus, you know, having a model that is specialized to just one language which makes it do much better. So these Indic models we have found actually do better than the open source models that they were built on top of, because now they have been specialized to a particular language.

Sridhar Vedantham: Ok. I know that your work focuses primarily on the evaluation of LLM’s, right? There must be a lot of other people who are also doing similar work in terms of evaluating performance of LLM on different parameters. How do you differentiate your work from what others are doing?

Sunayana Sitaram: Yeah, that’s a great question. So we’ve been doing evaluation work pre LLM actually. We started this a few years ago. And so we’ve actually done several evaluation projects. The previous one was called LITMUS where we were trying to evaluate without even having a benchmark in a particular language, right? And so we’ve built up a lot of expertise in how to do evaluation, and this has actually become a very hard problem in the LLM world because it’s becoming increasingly difficult to figure out what the strengths and weaknesses are of these LLMs because of how they’re built and how they behave, right. And so I think we bring in so much rich evaluation expertise that we’ve been able to do these kinds of, you know, Mega evaluations in a very systematic way where we’ve taken care of all of the we’ve taken care of all of the hanging loose threads that otherwise others don’t take care of. And that is why we managed to do these comprehensive giant exercises of Mega and Megaverse and also got these clear trends from them. So in that sense I would say that our evaluation research is very mature and we’ve been spending a lot of time thinking about how to evaluate which is unique in our group.

Sridhar Vedantham: OK, so one thing I’ve been curious about for a while is there seems to be a lot of cultural and social bias that creeps into these models, right? How does one even try to address these issues?

Sunayana Sitaram: That’s right. So, I think over the last few months, building culture, specific language models or even evaluating whether language models are appropriate for a particular culture, etc, that has become a really hot topic. Because people have started seeing that, you know,   most of these language models are a little tilted towards western protestant and, rich, industrialized kind of worldviews and the values that they encode may not be appropriate for all cultures. And so there have been some techniques that we’ve been working on in order to again shift the balance back into other target cultures that we want to fine tune the model for, so again, you know, you could take data that has characteristics of a particular culture, values of a particular culture, and then do some sort of fine tuning on a model in order to shift  its distribution more towards a target culture. There are techniques that are coming to be for these kinds of culture specific language models. However, I still think that we are far away from a comprehensive solution, because even defining what culture is and what constitutes, you know, say an Indic culture LLM, I think that’s a really hard problem.  Because culture is complex and there are so many factors that go into determining what culture is, and also it’s deeply personal. So, each individual has their own mix of factors that determine their own culture, right? So, generalizing that to an entire population is also quite hard, I think to do. So, I think we’re still in the very initial stages in terms of actually figuring out how well aligned these models are to different cultures and also trying to sort of align them to any specific target cultures. But it is a hot topic that a lot of people are currently working on.

Sridhar Vedantham: Yeah. You know, while you’re talking and giving me this answer, I was thinking that if you’re going to go culture by culture, first of all, you know, what is the culture, what are you doing about subcultures and how many cultures are there in the world, so I was just wondering how it’s going to even work in the long term? But I guess you answered the question by saying it’s just starting. Now let’s see how it goes.

Sunayana Sitaram: Absolutely.

Sridhar Vedantham:  It’s a very, very open canvas right now, I guess.

Sunayana Sitaram: Yeah.

Sridhar Vedantham: Sunayana, you know you’ve been speaking a lot about evaluation and so on and so forth and especially in the context of local languages and smaller languages and Indic languages and so on. Are these methods of evaluation that you talk about, are they applicable to different language groups and languages spoken in different geographies too?

Sunayana Sitaram: Absolutely. So in the Mega and Megaverse work, we covered 80 languages and many of them were not Indic languages. In fact, in the Megaverse work, we included a whole bunch of African languages as well. So the techniques, you know, would be applicable to all languages for which we have data for which data exists on the web. Where it is challenging is the languages that are only spoken, that are not written, and languages for which there is absolutely no data or representation available, on the web, for example. So, unfortunately, there aren’t benchmarks available for those languages, and so we would need to look at other techniques. But other than that, our evaluation techniques are for, you know,  all languages, all non-English languages.

[Music]

Sridhar Vedantham: There is something that I heard recently from you which again I found extremely interesting. It’s a project called Pariksha, which I know, in Hindi and derived from Sanskrit, basically means test or exam. And I remember this project because I’m very scared of tests and exams, and I’ve always been from school. But what is this?

Sunayana Sitaram: Yes, Pariksha is actually quite a new project. It’s under the project VeLLM that is on universal empowerment with Large Language Models and Pariksha is something that we are super excited about because it’s a collaboration with Karya, which is an ethical data company that was spun off from MSR India. So what we realized a few months ago is that you know there is just so much happening in the Indic LLM space and there are so many people building specialized models either for a single language or for a group of languages like Dravidian languages, for example. And of course, there are also the GPTs of the world which do support Indian languages as well, right. So now at last count, there are something like 30 different Indic LLMs available today. And if you’re a model builder, how do you know whether your Indic LLM is good or better than all of the other LLMs? If you’re somebody who wants to use these models, how do you know which ones to pick for your application? And if you’re a researcher, you know, how do you know what the big challenges are that still remain, right?

Sridhar Vedantham: Right

Sunayana Sitaram: And so to address this, of course, you know, one way to do this is to do evaluation, right. And try to figure out, you know, compare all these models on some standard benchmarks and then try to figure out which ones are the best. However, what we found from our work with Mega and Megaverse is that the Indian language benchmarks unfortunately are usually translations of already existing English benchmarks and also many of them are already present in training data of these large models, which means that we can’t use the already existing benchmarks to get a very good idea about whether these Indic LLMs are culturally appropriate, whether they capture linguistic nuances in the Indian languages or not, right. So we decided to sort of reinvent evaluation for these Indic LLMs and that’s where Pariksha came in. But then how do we scale if we want to actually, you know, get this kind of evaluation done and we were looking at human evaluation to do this, right. And so, we thought of partnering with Karya on this. Because Karya has reached in all the states in India and they have, you know, all of these workers who can actually do this kind of evaluation for different Indian languages. And so, what Pariksha is, it’s a combination of human evaluation as well as automated evaluation. And with this combination we can scale and we can do thousands and thousands of evaluations, which we have already done actually on all of these different models. And so this is the first time actually that all of the different Indic LLMs that are available are being compared to each other in a fair way. And we are able to come up with a leaderboard now of all of the Indic models for each Indic language that we are looking at. So that’s what Pariksha is. It’s quite a new project and we’ve already done thousands of evaluations and we are continuing to scale this up even further.

Sridhar Vedantham: So how does someone, you know, if I have a LLM of my own in any given Indic language, how do I sign up for Pariksha, or how do I get myself to be evaluated against the others?

Sunayana Sitaram: Yeah. So you can contact any of us for that, the Pariksha team. And we will basically include this model, the new model into the next round of evaluation. So what we do with Pariksha is we do several rounds. So we’ve already finished a pilot round and we’re currently doing the first round of evaluations. So we would include the new model in the next round of evaluations. And you know, as long as it’s an open source model or there is an API access available for that model, we can evaluate the model for you. We are also planning to release all the artifacts from Pariksha, including all the evaluation prompts. So even if it is a closed model, you can use these to do your own evaluation as well later to figure out how you compare with the other models on the leaderboard.

Sridhar Vedantham: Right. Quick question. When you say that you’re working with Karya, and you also say that you’re looking at human evaluation along with the regular methods of evaluation. Why do you need human evaluation at all in these situations? It’s just simpler to throw everything into a machine and let it do the work?

Sunayana Sitaram: Yeah, that’s a great question. So we did some work on, you know, making machines evaluators. So basically asking GPT itself to be the evaluator and it does a very good job at that. However, it has some blind spots. So we found that GPT is not a very good evaluator in languages other than English. Basically, it’s not a good evaluator in the languages that it doesn’t do well in otherwise and so using only automated techniques to do evaluation may actually give you the wrong picture. It may give you the wrong sort of trends, right? And so we need to be very careful. And so our ultimate goal is to build evaluation systems and also other kinds of systems in general where humans and LLMs can work together. And so the human evaluation part is to have checks and balances on the LLM evaluation part. Initially, what we are doing is we’re getting the same things evaluated by the human, and the LLM is doing the exact same evaluation. So we have a point by point comparison of what the humans are saying and what the LLM is saying so that we can really see where the LLM goes wrong, right. Where it doesn’t agree with humans. And then we use all of this information to improve the LLM evaluator itself. So we’re really trying to get humans to, you know, do the evaluation, get LLM’s to do the evaluation, use the human data in order to improve the LLM. And then just this continues in a cycle. And the ultimate goal is, send the things to the LLM that it’s good at doing and send the rest of the things that the LLM can’t do to humans who are like the ultimate authority on the evaluation. So it’s like this hybrid system that we are designing with Pariksha.

Sridhar Vedantham: Interesting. OK, so I know we are kind of running out of time. My last question to you would be, where do you see evaluation of LLM’s and where do you see your work going or progressing in the near future?

Sunayana Sitaram: So evaluation for me is a path to understanding what these systems can do and cannot do and then improving them, right. So our evaluations are always actionable. So we try to figure out why something is not working well. So even in the Mega paper, we had lots of analysis about what factors may lead to, you know, lower performance in certain languages, etc. So I see all of this as providing a lot of rich information to model developers in order to figure out what the next steps should be, how they should be designing the next generation of models and I think that has already happened. It’s, you know, systems have already improved from the time we started working on Mega and a lot of the issues that we pointed out in Mega, like tokenization etcetera, now they are well known in the field and people are actually taking steps in order to make those better in these language specific models etc. So I see the work as being, you know, first of all raising awareness about the problems that exist, but also providing actionable insights on how we could improve things. And with Pariksha also the idea is to release all the artifacts from our evaluation so that Indic model developers can use those in order to improve their systems. And so I see that you know better evaluation will lead to better quality models. That’s the aim of the work.

Sridhar Vedantham: Sunaya, thank you so much for your time. I really had a lot of fun during this conversation.

Sunayana Sitaram: Same here. Thank you so much.

Sridhar Vedantham: Thank you.

[Music ends]

The post Podcast: Evaluating LLMs using novel approaches. With Dr. Sunayana Sitaram appeared first on Microsoft Research.

Over recent years, Microsoft Research Asia has deepened its collaboration with medical institutions and academic experts, attracting professionals in healthcare to foster AI’s profound application in medical health, thereby contributing to a healthier global community.

Early detection, early treatment: AI in disease detection and rehabilitation

The early diagnosis of diseases is vital for enhancing treatment outcomes and patient quality of life. Rehabilitation training, a critical component of many treatment regimens, plays a significant role in restoring patient functions. Traditional methods face limitations due to resource distribution, geographic constraints, and a shortage of medical professionals, affecting the accessibility and efficiency of healthcare services. AI can support medical staff by providing automated, intelligent early disease detection, enabling timely intervention and treatment.

AI-powered voice recognition for speech rehabilitation in cleft palate patients

Cleft palate and cleft lip, prevalent congenital deformities in the oral and maxillofacial region, often result in hyper nasal speech due to velopharyngeal insufficiency. Microsoft Research Asia, in collaboration with medical institutions, recognizes hypernasality detection as crucial for treating cleft palate patients.

Traditionally, speech-language pathologists assess hypernasality, but their limited availability and concentration in certain hospitals necessitate extensive, costly cross-regional patient travel. An automated hypernasality assessment method would not only aid pathologists in making accurate evaluations but also facilitate remote patient diagnosis and treatment, significantly reducing costs.

Leveraging transfer learning technology, Microsoft Research Asia has developed an innovative approach using an automatic speech recognition (ASR) model to enhance hypernasality assessment. This innovative model excels in extracting acoustic features and demonstrates robust generalization capabilities. Comparative studies on two cleft palate datasets reveal that it surpasses existing methods, significantly enhancing the precision of pathologists’ diagnostic processes.

Following hypernasality evaluations, physicians devise tailored speech therapy regimens for patients. Microsoft Research Asia has advanced this process by developing the Masked Pre-training Pronunciation Assessment (MPA) model. This model supports end-to-end training and adapts to both unsupervised and supervised learning environment, enabling user-friendly remote deployment. Utilizing reference texts and integrating masking with prediction tactics, the MPA model adeptly circumvents issues of misalignment or misrecognition in pronunciation assessments, offering more precise speech correction support for individuals with cleft palate.

Microsoft Research Asia is actively collaborating with healthcare providers to assess the feasibility of deploying this cutting-edge speech assessment technology. The goal is to enhance the efficiency of medical diagnoses and treatments, lower the financial burden on patients, and extend the benefits of this technology to numerous cleft palate sufferers in isolated regions.

Related papers:

• Paper on hypernasality detection: Improving Hypernasality Estimation with Automatic Speech Recognition in Cleft Palate Speech (opens in new tab). INTERSPEECH 2022.
• Paper on speech assessment: End-to-End Word-Level Pronunciation Assessment with MASK Pre-training (opens in new tab). INTERSPEECH 2023.

Voice analysis model enhances Alzheimer’s disease screening

Alzheimer’s disease, a prevalent neurodegenerative condition primarily affecting the elderly, leads to progressive cognitive decline, including memory loss, language difficulties, and impaired reasoning. While there’s no cure for Alzheimer’s, early detection and intervention are key to decelerating its progression.

Traditional diagnostic methods, such as brain scans, blood tests, and cognitive assessments, are extensive and expensive. However, research indicates that early Alzheimer’s can be detected through speech analysis, identifying symptoms like fluent aphasia and word retrieval challenges.

Capitalizing on this insight, Microsoft Research Asia has pioneered speech and language analysis technologies to detect Alzheimer’s indicators from sophisticated acoustic and linguistic data. A novel task-oriented model has also been introduced, correlating language descriptions with cognitive tasks.

In the ADReSS dataset’s subtask involving “Cookie Theft” picture descriptions (Figure 1), these methods attained a 91.4% accuracy rate. This innovative approach, merging speech and semantic analysis, significantly increases disease detection accuracy. The model’s high efficiency and performance on new test sets offer promising prospects for Alzheimer’s screening at scale.

Related paper:

• Leveraging Pretrained Representations with Task-related Keywords for Alzheimer’s Disease Detection (opens in new tab). ICASSP 2023.

Advancing autism diagnosis: Unsupervised detection of stereotypical behaviors

Autism Spectrum Disorder (ASD) typically manifests in early childhood, presenting challenges in social interaction and communication, accompanied by repetitive behaviors. These behaviors, which may include actions like persistent hand-flapping or head-banging, serve as vital indicators for ASD diagnosis. Early detection and intervention are crucial for improving outcomes, yet traditional methods relying on prolonged observation by specialists are not always efficient. Hence, the development of a swift, automated detection system is invaluable.

Traditional approaches have utilized computer vision and supervised learning to analyze video data of individuals with ASD. However, these methods face limitations due to the diverse range of stereotypical behaviors and privacy concerns associated with video data collection.

Addressing these challenges, Microsoft Research Asia, in collaboration with medical institutions, has innovated an unsupervised approach using video anomaly recognition. The new Dual-Stream Stereotypical Behavior Detector (DS-SBD) model leverages the temporal dynamics of human posture and repetitive motion patterns. Remarkably, DS-SBD requires only non-anomalous behavior for training and can identify previously unseen stereotypical behaviors during inference, such as identifying circling behaviors in the training data.

Extensive studies validate that DS-SBD’s unsupervised technique has increased the micro-average AUROC from 60.43% to 71.04% and the macro-average AUROC from 56.45% to 73.39%, signifying a substantial improvement in both accuracy and the scope of detectable behaviors. This breakthrough outperforms current state-of-the-art methods and is poised to set a new standard in the field. While DS-SBD marks a significant advancement in recognizing stereotypical behaviors, it represents only one facet of the broader ASD diagnostic process. Comprehensive early diagnosis and intervention strategies will benefit from continued interdisciplinary collaboration and societal engagement.

Related paper:

• Unsupervised Video Anomaly Detection for Stereotypical Behaviors in Autism (opens in new tab). ICASSP 2023.

Advancing neonatal seizure detection through brainwave analysis

Epilepsy in children is a multifaceted, often recurring neurological disorder that predominantly occurs in the formative years (0-18 years). The prompt identification of epilepsy in newborns is vital to safeguard their developmental trajectory.

The genesis of epileptic seizures lies in the abnormal discharges of neurons within the brain, rendering brainwave analysis a pivotal tool for epilepsy diagnosis. Nonetheless, the nascent state of neonatal brain development, coupled with the pronounced noise in brainwave data and the marked variability among infants, renders the detection of neonatal epilepsy a formidable medical challenge.

Microsoft Research Asia and its collaborators have unveiled a deep learning paradigm, harnessing artificial intelligence and electroencephalogram (EEG) signals – dubbed the Spatial-Temporal EEG Network (STATENet). This model adeptly processes neural signals, nimbly adjusts to neonatal EEG channel variations, and addresses the challenges outlined above. Additionally, the team has introduced a model-level integration technique that synergistically combines outcomes from various spatial-temporal deep models, thereby bolstering the STATENet model’s generalization ability across diverse neonatal subjects.

Extensive studies utilizing a comprehensive dataset of real-world neonatal EEG data have demonstrated the STATENet model’s substantial enhancement in detection precision. The model’s area under the precision-recall curve (AUPRC) witnessed an improvement exceeding 30% relative to prevailing top-tier methods, equipping physicians with a novel diagnostic instrument for pediatric epilepsy.

Moreover, Microsoft Research Asia has pioneered the inaugural cross-dataset EEG model capable of deciphering any EEG data, thus achieving a ‘one-to-many’ brainwave comprehension. This breakthrough underpins the AI Neurologist system, designed to augment brainwave signal analysis in both clinical and research settings, elevating diagnostic accuracy from 75% to 90% in a case study. The associated models are now open source on GitHub, inviting global research participation to extend this technology’s impact across the medical spectrum and catalyze new diagnostic and therapeutic innovations.

Related papers:

• Paper on neonatal seizure detection: Protecting the Future: Neonatal Seizure Detection with Spatial-Temporal Modeling (opens in new tab). IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2023.
• Paper on brainwave pre-training model: Learning Topology-Agnostic EEG Representations with Geometry-Aware Modeling. (opens in new tab)

Enhancing disease progression prediction and personalized care: The role of AI in precision medicine

Precision medicine represents a transformative approach to healthcare, tailoring treatments to individual patient profiles. Despite the promise, the complexity and unique nature of diseases present significant hurdles. AI emerges as a powerful ally, leveraging data analysis, pattern detection, and predictive modeling to forecast disease progression and risks. This capability is especially crucial in chronic disease management, aiding clinicians and patients in mitigating illness severity and preventing complications.

Graph neural networks: A novel approach to Parkinson’s disease progression

Parkinson’s disease, a prevalent neurodegenerative condition in seniors, progresses gradually. Patient conditions may remain stable or even improve over time with proper medication and therapy, maintaining optimal physical functions. Yet, Parkinson’s presents a spectrum of symptoms, from sleep disturbances to motor challenges, making disease progression prediction complex.

Researchers at Microsoft Research Asia advocate for the analysis of multimodal data to extract similar symptoms, thus enhancing prediction accuracy. Graph neural networks (GNNs) excel in mapping patient interconnections, forming networks where nodes represent patients linked by shared attributes. Selecting these attributes, however, largely demands expert knowledge and experience.

To overcome this, Microsoft Research Asia collaborated closely with medical institutions.  Based on recommendations from professional medical personnel, a new algorithm called AdaMedGraph was proposed. AdaMedGraph autonomously identifies key features to construct patient similarity graphs, harmonizing with existing knowledge and integrating expert-designed graphs into a comprehensive model. Unifying individual and collective data, this innovation simplifies the graph construction process.

Evaluated on two public datasets, the Parkinson’s Progression Markers Initiative (PPMI) and the Parkinson’s Disease Biomarkers Program (PDBP), AdaMedGraph outperformed benchmarks in predicting Parkinson’s progression over 24 months, setting the stage for personalized treatment strategies.

Moreover, AdaMedGraph’s robust generalization ability shines in metabolic syndrome prediction, achieving an AUROC of 0.675 on test datasets. This underscores the model’s efficacy in integrating intra- and inter-patient data for individual disease progression forecasting, inspiring new avenues in medical research.

Related paper:

• AdaMedGraph: Adaboosting Graph Neural Networks for Personalized Medicine. (opens in new tab)

Enhancing interdisciplinary collaboration to maximize AI’s potential

Microsoft Research Asia endeavors extend beyond mere disease detection and progression prediction. In collaboration with the medical sector, Microsoft Research Asia is probing the vast capabilities of AI in advancing drug development and medical research. This includes leveraging state-of-the-art technology in constructing artificial retinas, analyzing drug dependency, advancing cancer therapies, and exploring human metabolism, among other areas.

As AI technology matures and progresses, its practical application potential becomes increasingly evident. Yet, unlocking AI’s full value across diverse sectors necessitates essential interdisciplinary and cross-domain collaboration. “The synergistic collaboration with medical professionals from healthcare and research institutions have enabled Microsoft Research Asia to conduct extensive research projects within the medical and health domain. Our continuous exploration into AI’s application in critical healthcare aspects—ranging from disease detection to rehabilitation and disease progression forecasting—is a testament to our collective dedication. We invite more exceptional individuals passionate about interdisciplinary research to join us in our quest to safeguard human health and foster medical advancements,” expressed Lili Qiu, assistant managing director of Microsoft Research Asia.

Note: The medical health research conducted by Microsoft Research Asia, as discussed in this article, is purely exploratory and guided by professional medical entities and research institutions. Our aim is to further scientific advancement and offer theoretical and technical support for the future medical applications benefiting humanity. All research is in strict adherence to Microsoft’s responsible AI principles, upholding fairness, inclusiveness, reliability and safety, transparency, privacy and security, and accountability. The technologies and methodologies referenced herein are in the R&D phase and are not yet commercialized products or services, nor do they represent medical advice or treatment plans. For health-related concerns, we advise consulting with certified medical practitioners.

The post Guarding human health: AI empowers innovative applications in healthcare appeared first on Microsoft Research.

After a comprehensive evaluation, Dongqi Han selected Microsoft Research Asia (MSR Asia) for his initial foray into fulfilling his aspirations. Prior to completing his doctorate, he undertook an internship at MSR Asia – Shanghai. During his six-month internship, Han gained firsthand experience of the lab’s commitment to pioneering basic research and its active engagement in fostering industrial collaborations, thereby facilitating the practical application of innovative findings. This experience sowed the seeds for his eventual formal engagement with the lab.

“MSR Asia has established a dynamic platform that seamlessly integrates fundamental research with practical industrial applications. Within this environment, I have the opportunity to work alongside eminent researchers, delving into the underlying principles and methodologies of intelligence. Moreover, I am able to harness the power of AI in domains such as healthcare. This synergy of theory and practice was a pivotal factor in my decision to join the lab after graduation. Undoubtedly, it represents an ideal launchpad for my career in scientific research,” Dongqi Han articulated.

Unveiling the core of intelligence: At the crossroads of AI and neuroscience

In recent years, the synergy between computer technologies like AI and various other fields has grown remarkably. MSR Asia is at the forefront, spearheading pivotal research and progressively amplifying its investments. Shanghai, a metropolis celebrated for its diversity and home to esteemed academic and leading medical institutions, offers fertile ground for the interdisciplinary fusion of AI and other fields. Consequently, the confluence of AI with neuroscience and other healthcare domains has emerged as a key research focus for MSR Asia – Shanghai.

Dr. Dongqi Han, a graduate of the Okinawa Institute of Science and Technology Graduate University (OIST) in Japan, majored in cognitive neuro-robotics, which encompasses the study of robotics integrated with neuroscience. As a neuroscientist, Dr. Han’s expertise significantly enhances the professional capabilities of MSR Asia’s interdisciplinary team, focusing on AI and brain science research.

Dongqi Han’s research primarily explores two areas: the convergence of AI with neuroscience, and AI’s applications in healthcare. He believes that this synergy is not only theoretically profound but also immensely practical. Dr. Han asserts, “To create more intuitive and effective interfaces, whether they be brain-computer or human-computer, a more intricate comprehension of the human cognitive and perceptual processes is essential.” In healthcare, neurological disorders impact approximately one billion individuals globally. Studies at the nexus of AI and brain science have the potential to enrich the knowledge base for both clinicians and patients, leading to improved diagnosis, prevention, and management of these conditions.

“AI and brain science are deeply intertwined, both delving into the core and mechanisms of intelligence. They encounter similar issues and can mutually benefit from shared insights.”

Indeed, AI technologies often draw from the brain’s neural networks, with structures like multilayer perceptron (MLP) and long short-term memory (LSTM) networks mirroring our own cognitive architecture. By examining human and animal cognition—learning, memory, decision-making—we can augment AI’s capabilities. A critical hurdle for AI is “catastrophic forgetting”, where new learning can erase old knowledge, a flaw not seen in the human brain. Dr. Han and his team colleagues are dedicated to resolving such AI challenges by gleaning lessons from our neurological processes.

Conversely, the robust data processing and modeling prowess of AI holds the potential to advance neuroscience research and its applications significantly. The human brain is composed of approximately 10^11 neurons interconnected by roughly 10^15 synapses. Harnessing AI to model the brain’s operational principles and computational strategies is essential to manage this immense data complexity and to substantiate the veracity of theoretical frameworks in neuroscience.

Presently, Dr. Han and his team colleagues, along with collaborators, have garnered preliminary findings from their research endeavors. Their study primarily examines two distinct human behavioral types: habitual and goal-directed behaviors [1]. For instance, habitual behavior, akin to the automatic act of selecting a familiar route home post-work, requires no conscious deliberation. In contrast, goal-directed behavior involves intentional consideration of both purpose and outcome, exemplified by plotting a course to the Civil Affairs Bureau to obtain a marriage certificate. While these behavioral models elucidate numerous aspects of biological conduct, the mechanisms by which the brain decides between these patterns and their mutual interactions remain an enigma.

Dongqi Han said, “Our team employs deep learning and machine learning methodologies to model and investigate the characteristics and underlying neural mechanisms of two distinct behavioral types. This endeavor not only contributes to the advancement of cognitive science and psychology but also serves as a source of inspiration for the development of innovative AI algorithms.”

Dr. Han’s recent research, conducted alongside his colleagues, revolves around emulating the brain’s neural circuitry. This has culminated in the development of a novel neural network model named CircuitNet [2]. Characterized by densely interconnected neurons within neural clusters and sparse connections across different brain regions, this model mirrors the human brain’s unique wiring. The team at MSR Asia is delving into the intricacies and benefits of such a neural architecture. Dr. Han, who has been involved in this project since his internship, has seen CircuitNet come to fruition through collaborative efforts, culminating in its selection for presentation at ICML 2023.

CircuitNet represents an advancement in neural network architectures, offering enhanced performance with a reduced parameter count, thus leading to greater energy efficiency. Remarkably, the human brain operates on less than 20 watts of power on average—a stark contrast to the substantial energy demands of large-scale AI models such as GPT-4, which may require hundreds to thousands of watts. Moving forward, Dongqi Han and his team colleagues are dedicated to unraveling the human brain’s mechanisms for energy conservation, drawing inspiration from CircuitNet’s design.

Dr. Han’s research extends to deep reinforcement learning and embodied AI, with the goal of refining AI to improve learning, decision-making, and real-world interaction capabilities of intelligent robots. He observes that while current large AI models predominantly generate content like text and images, embodied AI outputs dynamic actions, introducing a myriad of real-world uncertainties. For instance, a robot engaged in painting might encounter various challenges such as errors, equipment failure, or interference, all influencing the final outcome. Navigating these complexities requires sophisticated action selection processes. Dr. Han believes that by drawing parallels to human cognitive decision-making, we can expedite the advancement of embodied intelligence.

Fostering innovation: The power of interdisciplinary learning

Dongqi Han’s insatiable curiosity about the world fuels his wide-ranging and intense passion for scientific inquiry. His academic journey began with an undergraduate major in theoretical physics—a field he regards as exceptionally demanding. It necessitates a learner to possess stringent logical reasoning, robust mathematical prowess, and the capacity for experimental design and data analysis. These skills are instrumental in enhancing an individual’s intellectual caliber. Furthermore, physical science serves as the bedrock for numerous contemporary technologies, offering expansive applications.

During his undergraduate study in physics, Dongqi Han was deeply engrossed in the detection of physical parameters within tokamaks—devices pivotal for controlled nuclear fusion [4]. He considers nuclear fusion as a boundless, efficient, and clean energy source, believing its mastery to herald unparalleled benefits for humanity. Initially, Dr. Han viewed achieving controlled nuclear fusion as his scientific beacon. Yet, as his studies progressed, he discerned that the real hurdles to implementation lay not in the realm of theoretical physics, but within the ambit of engineering challenges. This revelation steered him to realize that his knowledge in theoretical physics might not directly contribute to the fruition of his aspirations. It was during this phase, amidst his tokamak investigations, that Dr. Han’s encounter with machine learning technology sparked a transformative shift in his academic pursuit, leading him to specialize in cognitive neuro-robotics for his doctoral research.

Embracing interdisciplinary learning, Dongqi Han embarked on a journey of discovery, building his foundation from the ground up. His initial year in the doctoral program was marked by an intensive curriculum that spanned basic neuroscience, machine learning, robotics control automation, and the integrative domains of cognitive science and psychology. Despite the rigorous challenges of navigating multiple disciplines, this cross-pollination of knowledge ignited innovative ideas. These insights have proven to be invaluable, enriching his research in AI and brain science.

In exploring the domains of reinforcement learning and embodied intelligence, Dongqi Han aims to integrate the methodical thinking approaches of physics. This will involve transferring the discipline’s rigorous thought processes (for example, commonly used measurement statistical methods) and logical reasoning, honed through physical experimentation, into the realm of AI study.

“The benefits of interdisciplinary learning are not only reflected in the cross-domain application of knowledge but also in the borrowing and inspiration of logical thinking. For example, when solving problems in neural network machine learning, traditional machine learning thinking often first considers data volume and model scale. Training in neuroscience can allow us to start from the perspective of the human or animal brain, thinking about problems in a more expansive and flexible way,” said Dongqi Han.

Collaborative synergy: Harnessing advanced technology for real-world solutions

Dongqi Han and his team colleagues are not only dedicated to interdisciplinary research but also actively engage in cross-disciplinary collaborations. They partner with universities and medical institutions around the globe, harnessing advanced technologies to tackle real-world challenges and forge superior solutions.

Dongqi Han and his team colleagues, in partnership with Fudan University, have pioneered an AI model for machine vision that replicates human visual perception [3], with a focus on boosting the energy efficiency of computer vision systems. Dr. Han notes, “Through collaborative research, we have discovered notable distinctions between human and computer vision. Human vision exhibits a significantly higher resolution at the fovea—the central point—compared to the peripheral vision. Additionally, the human brain processes information by transmitting spike-based signals, a characteristic that is mirrored in the architecture of spiking neural networks.” To harness the superior aspects of human vision, the team employed a spiking neural network to emulate a visual system with variable resolution and neuronal spike-based communication. This innovative model is poised to revolutionize energy efficiency, potentially achieving up to a hundredfold improvement in performing visual tasks with significantly reduced energy demands.

In a groundbreaking robotics initiative, researchers from MSR Asia and Korea Advanced Institute of Science and Technology (KAIST) have been pioneering the use of wearable, non-invasive devices to decipher human brainwave signals. This technology is poised to significantly enhance a robot’s ability to interpret human intentions with greater precision. “Consider an elderly individual reaching towards the kitchen—does this gesture indicate hunger or the need to retrieve an item? Or when they gesture towards a table adorned with both a water bottle and a tissue box, which is their intended choice?” Dongqi Han said, “KAIST’s robust expertise in neuroscience, combined with our advanced AI algorithms, creates a powerful alliance. Together, we’re pushing the boundaries of what’s possible in robotics, enabling more nuanced task execution through a fusion of brain science and AI. This interdisciplinary collaboration is sparking innovative research avenues for both teams.

From delving into games to persevering in scientific research

Dongqi Han’s diverse interests extend beyond his professional pursuits. He likes hiking and badminton, as well as various types of video games. “My passion for video games began in elementary school, evolving into a dedication to a particularly challenging game that demanded patience and perseverance. The process of advancing through persistent practice instilled in me a profound sense of accomplishment and shaped my approach to life, fostering patience and resilience against adversity” , Dr. Han shares.

This mindset has also permeated his research philosophy, which is characterized by deep, sustained inquiry. The supportive and diverse research environment at MSR Asia encourages vibrant collaboration and communication with colleagues from diverse disciplines and walks of life, bolstering his commitment to long-term research and connecting him with fellow researchers who share his vision.

Related links:

[1] Synergizing Habits and Goals with Variational Bayes, Nature Communications, 2024
(preprint at https://osf.io/preprints/psyarxiv/v63yj)

[2] CircuitNet：A Generic Neural Network to Realize Universal Circuit Motif Modeling
https://proceedings.mlr.press/v202/wang23k/wang23k.pdf

[3] Energy-Efficient Visual Search by Eye Movement and Low-Latency Spiking Neural Network
https://arxiv.org/abs/2310.06578

[4] In situ relative self-dependent calibration of electron cyclotron emission imaging via shape matching
https://doi.org/10.1063/1.5038866

The post Dongqi Han: An interdisciplinary odyssey with AI and other fields appeared first on Microsoft Research.