{"id":1122837,"date":"2025-02-19T08:05:46","date_gmt":"2025-02-19T16:05:46","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1122837"},"modified":"2025-03-05T16:35:15","modified_gmt":"2025-03-06T00:35:15","slug":"introducing-muse-our-first-generative-ai-model-designed-for-gameplay-ideation","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/introducing-muse-our-first-generative-ai-model-designed-for-gameplay-ideation\/","title":{"rendered":"Introducing Muse: Our first generative AI model designed for gameplay ideation"},"content":{"rendered":"\n
\"Three<\/figure>\n\n\n\n

Today, the journal Nature (opens in new tab)<\/span><\/a> is publishing our latest research, which introduces the first<\/strong> World and Human Action Model (WHAM). <\/strong>The WHAM, which we\u2019ve named \u201cMuse,\u201d is a generative AI model of a video game that can generate game visuals, controller actions, or both.

The paper in Nature offers a detailed look at Muse, which was developed by the Microsoft Research Game Intelligence (opens in new tab)<\/span><\/a> and Teachable AI Experiences (opens in new tab)<\/span><\/a>\u202f(Tai X) teams in collaboration with Xbox Games Studios\u2019 Ninja Theory (opens in new tab)<\/span><\/a>. Simultaneously, to help other researchers explore these models and build on our work, we are open sourcing the weights and sample data and making the executable available for the WHAM Demonstrator\u2014a concept prototype that provides a visual interface for interacting with WHAM models and multiple ways of prompting the models. Developers can learn and experiment with the weights, sample data, and WHAM Demonstrator on Azure AI Foundry (opens in new tab)<\/span><\/a>.\u00a0<\/p>\n\n\n\n

In our research, we focus on exploring the capabilities that models like Muse need to effectively support human creatives. I\u2019m incredibly proud of our teams and the milestone we have achieved, not only by showing the rich structure of the game world that a model like Muse can learn, as you see in the video demo below, but also, and even more importantly, by demonstrating how to develop research insights to support creative uses of generative AI models.<\/p>\n\n\n\n

Generated gameplay examples<\/h2>\n\n\n\n
\n
\"10<\/figure>\n\n\n\n
\"10<\/figure>\n\n\n\n
\"10<\/figure>\n\n\n\n
\"10<\/figure>\n\n\n\n
\"10<\/figure>\n\n\n\n
\"10<\/figure>\n\n\n\n
\"10<\/figure>\n\n\n\n
\"10<\/figure>\n\n\n\n
\"10<\/figure>\n<\/div>\n\n\n\n
Example gameplay sequences generated by Muse (based on WHAM-1.6B) demonstrate that our model can generate complex gameplay sequences that are consistent over several minutes. All examples shown here were generated by prompting the model with 10 initial frames (1 second) of human gameplay and the controller actions of the whole play sequence. Muse is used in \u201cworld model mode\u201d meaning that it is used to predict how the game will evolve from the initial prompt sequence. The more closely the generated gameplay sequence resembles the actual game, the more accurately Muse has captured the dynamics of that game.<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

What motivated this research?<\/h2>\n\n\n\n

As we release our research insights and model today, I keep thinking back to how this all started.  There was a key moment back in December 2022 that I remember clearly. I had recently returned from maternity leave, and while I was away the machine learning world had changed in fundamental ways. ChatGPT had been publicly released, and those who had tried it were in awe of OpenAI\u2019s technical achievements and the model\u2019s capabilities. It was a powerful demonstration of what transformer-based generative models could do when trained on large amounts of (text) data. Coming back from leave at that moment, the key question on my mind was, \u201cWhat are the implications of this achievement for our team\u2019s work at the intersection of artificial intelligence and video games?\u201d<\/p>\n\n\n\n

A new research opportunity enabled by data<\/h2>\n\n\n\n

In our team, we had access to a very different source of data. For years, we had collaborated with Xbox Game Studios\u2019 Ninja Theory (based in Cambridge, UK, just like our research team) to collect gameplay data from Bleeding Edge, their 2020 Xbox game. Bleeding Edge is a 4-versus-4 game where all games are played online, and matches are recorded if the player agrees to the End User License Agreement (EULA). We worked closely with our colleagues at Ninja Theory and with Microsoft compliance teams to ensure that the data was collected ethically and used responsibly for research purposes.<\/p>\n\n\n\n

“It’s been amazing to see the variety of ways Microsoft Research has used the Bleeding Edge environment and data to explore novel techniques in a rapidly moving AI industry,” said Gavin Costello, technical director at Ninja Theory. “From the hackathon that started it all, where we first integrated AI into Bleeding Edge, to building AI agents that could behave more like human players, to the World and Human Action Model being able to dream up entirely new sequences of Bleeding Edge gameplay under human guidance, it’s been eye-opening to see the potential this type of technology has.” <\/p>\n\n\n\n

Muse Training Data<\/h3>\n\n\n\n
\n
\n