Research at Microsoft 2020: Addressing the present while looking to the future

Microsoft researchers pursue the big questions about what the world will be like in the future and the role technology will play. Not only do they take on the responsibility of exploring the long-term vision of their research, but they must also be ready to react to the immediate needs of the present. This year in particular, they were asked to use their roles as futurists to address pressing societal challenges.

In early 2020, as countries began responding to COVID-19 with stay-at-home orders and business operations moved from offices into homes, researchers sprang into action to identify ways their skills and projects could help while also making personal and professional adjustments of their own. In some cases, they pivoted to directly address the pandemic. A team from Microsoft Research Asia developed the COVID Insights website to promote scientific analysis and understanding of the disease, while the Socially Intelligent Meetings program expanded its work in telepresence technologies to include the Meetings During COVID-19 project. From responses provided by employee volunteers, these researchers are piecing together the effects of taking meetings almost entirely via screens.

Researchers also turned to the wider research community in their pursuit of solutions that would allow people to persevere in these challenging times and prosper beyond them with academic collaboration around topics related to pandemic preparedness (opens in new tab) and—in August—the New Future of Work symposium (opens in new tab). A series of reports conducted by Microsoft considers a variety of information, including research from throughout the company, in studying worker productivity and well-being (opens in new tab). The insights are leading to enhancements in Microsoft productivity tools that are available now and in the near future, such as Together mode (opens in new tab), virtual commutes, and meditation experiences (opens in new tab) in Teams (the latter two features roll out next year).

“When a major crisis strikes the world, science and technology research is almost always of paramount importance in response, rehabilitation, and—ultimately—creating resilience for the future. Today, the people of Microsoft Research are proving to be critical in dealing with climate calamities, such as famines and major wildfires; global health threats, such as that posed by the COVID-19 pandemic; the weakening of democratic institutions posed by misinformation and insecure voting infrastructure; wildlife extinction caused by pollution and illegal poaching; and more. While our main mission continues to be grounded in fundamental, long-term research, contributing to societal resilience is also a growing element in how we ensure a good future for all.”

Peter Lee, Corporate Vice President, Microsoft Research & Incubations

Meanwhile, research started before the pandemic feels increasingly significant. In April, two papers with implications on workplace well-being and hybrid scenarios, respectively, were presented on the conference circuit. Researchers from the productivity group developed models that leverage digital activities and other data to suggest appropriate times for workers to switch tasks and take breaks (opens in new tab), while researchers in the United Kingdom and Canada built a two-way telepresence system (opens in new tab) to enhance collaboration among remote and local individuals. In December, Eyal Ofek (opens in new tab) shared how advances in virtual reality could be used to maximize our workspaces—wherever they may be—during a Microsoft Research webinar (opens in new tab).

And while a lot has been said about work in these unprecedented times, research into the dynamics of epidemics themselves moved forward. In September, it was announced that Microsoft Premonition, a system leveraging robotics and genomics to track pathogens responsible for widespread disease, is being made available to additional partners.

(opens in new tab)

While research relevant to the pandemic has been of the highest importance this year, Microsoft researchers took their research in a broad range of directions—progress in AI, healthcare technology, and security advanced rapidly. Below is a selection of highlights that came out of Microsoft Research in 2020.

Scaling AI for better performance and real-world applications

This year saw significant breakthroughs for creating AI that is substantially more powerful, scalable, and readily integrated into Microsoft products. The AI at Scale initiative, born from a cornucopia of work in the area in 2020, combines large-scale models, AI supercomputing, and teams of researchers and product engineers working together to implement AI in a variety of Microsoft products and infrastructure.

The Microsoft Turing team’s natural language generation and natural language representation models went from being announced in February to implementation into products like Microsoft Word and Bing, ultimately with one model setting a record on the Xtreme benchmark for cross-lingual transfer learning in October.

Helping to power large AI models, the DeepSpeed library with Zero Redundancy Optimizer (ZeRO) also underwent major transformations since its introduction in February. The library initially included support for training models up to 100 billion parameters in size, but by the end of the year, the library was capable of training models up to 1 trillion parameters while also introducing new methods to train models with lower resource costs. On the testing side of natural language generation and understanding, researchers released XGLUE, a benchmark dataset for gauging models’ zero-shot cross-lingual transfer capabilities across 19 different languages.

In the realm of vision and language pretraining (VLP), Microsoft researchers released OSCAR (Object-Semantics Aligned Pretraining) in May, leading to a novel framework with state-of-the-art performance on six different vision-and-language tasks. In October, researchers collaborating with Azure Cognitive Services created VIVO (Visual Vocabulary Pretraining), which resulted in a framework for novel object captioning that achieved state of the art and even surpassed human performance on the novel object captioning (nocaps) benchmark.

Technologies like Microsoft Floating Point, used in the Project Brainwave architecture, are helping to lower the cost of deep neural network (DNN) inference. Improvements like this allow Microsoft to power large AI models on the scale needed to empower Microsoft users around the world. Head over to the AI at Scale page to learn about some of the many other projects undertaken at Microsoft Research to advance large-scale AI this year.

Building AI responsibly by pursuing safety, fairness, interpretability, and accessibility

As AI techniques have made leaps and bounds, researchers are undertaking the crucial task of examining responsible practices in AI, which include accessibility, fairness, and interpretability. Methods for understanding and explaining what AI does, as well as assessing fairness at all stages of development, are big trends in this area.

“Building and fielding AI responsibly is a challenging, cross-disciplinary research area. Our progress over the last year builds on insights from previous years with an emphasis on applying our research and learnings to developing usable methods and tools that can help engineers design and develop trustworthy AI systems. We’ve made strong progress, but our journey is far from over.”

Eric Horvitz, Technical Fellow and Chief Scientific Officer

In January, researchers shared their insights into how societal bias in historical data used to train algorithmic decision-making systems could be reinforced by future data (opens in new tab) and explored two interventions that could help to correct this bias. Meanwhile, another team of researchers developed a framework and open-source library for generating explanations (opens in new tab) that individuals adversely impacted by system decisions—such as those who’ve been denied a loan or insurance—can use to work toward a positive determination. These counterfactual explanations can also be used for system evaluation, becoming a tool for practitioners. Builders of AI tech are at the center of two papers recognized at CHI 2020. Hanna Wallach (opens in new tab), Jennifer Wortman Vaughan (opens in new tab), and their co-authors sought to empower the group with an actionable checklist—co-designed with practitioners—for discussing and addressing fairness throughout the AI life cycle and an examination into the effectiveness of available interpretability tools (opens in new tab) based on an interview study and survey with practitioners. Wortman Vaughan explored understanding AI systems in a January webinar (opens in new tab).

Collaboration with and inclusion of those closest to the tech was also happening on the accessibility front, where advances in computer vision and natural language processing (NLP) are allowing researchers to improve AI for alt text generation and object identification. Researchers worked with people who are blind or have low vision to understand how to improve the alt text generated by automated systems and to develop a dataset for personalized object recognition. In a March webinar, Dr. Danna Gurari and Dr. Ed Cutrell discussed developing impactful vision systems and the role dataset challenges play. Meanwhile, a partnership with people living with and affected by amyotrophic lateral sclerosis (ALS) laid the groundwork for Expressive Pixels, a platform for creating LED-display animations that simultaneously offers opportunities for creating, learning, and communicating in new ways.

Responsible AI requires that the talent, knowledge, and experiences of those developing it are as diverse as the people using it. Microsoft is committed to clearing a path into the industry for underrepresented groups, including sponsoring and participating in events like the Black in AI, Queer in AI, and Women in Machine Learning NeurIPS workshops and creating professional and academic opportunities through Microsoft Research. To have your work supported or to join the Microsoft Research team, see our Academic Programs—such as the Dissertation Grant for PhD students from underrepresented groups (proposals will start to be accepted in February)—and open research positions and internships.

Practical and theoretical advances in cryptography and security

As technology becomes more embedded in and essential to people’s lives, creating new technologies for 2020 and beyond demands deeper consideration of people’s privacy, security of the web and internet-connected devices, and safeguards on human rights. This year, researchers introduced ElectionGuard (opens in new tab), a system that applies homomorphic encryption to both secure people’s votes—so that no one else can see how they voted—and allow voters to verify their votes are properly counted. The system was piloted in Fulton, Wisconsin, in February, where election officials tested machines running ElectionGuard; the final tally was gathered through traditional paper ballot methods. The code for ElectionGuard was made open source, and Josh Benaloh (opens in new tab) presented on the technology in a webinar in April (opens in new tab).

Cryptography was also explored from a post-quantum angle. Future quantum computers could decipher even the most secure current cryptographic techniques, so researchers in this space have begun to investigate new methods for making the cryptography of the future equal to the task of protecting people’s privacy and information in a world where quantum computers far exceed the power of supercomputers now. These methods can also keep information more secure in the present computing landscape. To learn more about the world of post-quantum cryptography, check out webinars from Craig Costello and Christian Paquin below.

Researchers and engineers released resources and technologies to uncover security vulnerabilities and identify potential attacks. In March, Patrice Godefroid made a case for developers adding fuzzing to their toolkit to detect vulnerabilities in software through automated testing, and in November, a team introduced RESTler, “the first stateful REST API fuzzing tool for automatically testing and finding security and reliability bugs in cloud/web services through their REST APIs.” Researchers also released Project Freta, a service for Linux systems that detects evidence of OS and sensor sabotage by analyzing a memory snapshot to find rootkits and other malware.

Improving healthcare through technology

The importance of healthcare technology came into especially strong focus as this year progressed. In many instances, projects already underway were particularly timely, as was the case with investigations into making online mental health interventions more effective through data analysis, the potential for personalizing those mental health interventions via subtyping, improving mental health helpline technology, and using chat apps to help facilitate patient care in hospitals.

In late August, researchers announced a method for biomedical NLP pretraining that could enable researchers to stay up to date with the continually increasing amount of new scientific knowledge in the field by using NLP to quickly identify and cross-reference important findings. Their model, PubMedBERT, obtained state-of-the-art results in several biomedical applications.

As medical and mental health professionals adapt how they provide care for people in the 21st century, researchers intend to continue to create technology that complements experts’ skills.

Bringing reinforcement learning into the real world

Reinforcement learning—a framework in which ML systems learn via interactions with their environment—has long been an active area at Microsoft Research, and the drive to advance RL has increased thanks to the approach’s success in Microsoft products and services. Researchers are tackling RL both empirically and theoretically, and their enthusiasm and efforts were on full display in 17 NeurIPS-accepted papers that pursued a variety of promising avenues.

Two papers designed methods for leveraging existing logged datasets in an area of RL that uses past experience to give agents a leg up prior to deployment, while separate work brought strategic exploration to the popular gradient decent–based approaches for RL. Other work highlighted a trend in learning good representations for agents’ observations. In their respective papers, Akshay Krishnamurthy, Devon Hjelm, and their coauthors incorporated auxiliary prediction problems to discover representations that simplify downstream learning tasks. Earlier in the year, researchers deployed Transformers in several ways to develop Working Memory Graph, an RL agent capable of more efficient learning when advanced reasoning is involved, such as future planning in the game Sokoban.

Games make great arenas in which to train agents for use in gaming or in more general applications. In August, Sam Devlin and Katja Hofmann shared work done as part of a new collaboration with game developer Ninja Theory around enhancing gaming with RL agents capable of teamwork with human counterparts. Also during the summer, researchers kicked off the second iteration of MineRL, a sample-efficient RL challenge based on the platform Project Malmo, which uses Minecraft as a playground for AI experimentation.

For an overview of RL, check out Hofmann’s webinar, and to learn about one RL framework in particular, multi-armed bandits, read this introductory text on the subject. And if you can hardly wait to learn more, fret not—you can start the new year off strong with Reinforcement Learning Day 2021 in January. Until then, check out content, including videos, from last year’s event.

Optimizing AI

On the winding road of this year in technology research, it’s only fitting that we loop back to AI, which researchers sought to optimize from multiple perspectives. One perspective considered how AI has evolved alongside the game of chess. In the last few decades, AI has advanced to a point where it can spar with and succeed against the best players in the world. This led researchers to shift their focus from how to make AI better at chess to how chess-playing AI can be refined to better match human playing styles and skill levels. As a result, researchers created Maia, a Leela Chess Zero–based engine that matches human play more closely than previously achieved.

To ramp up neural architecture search (NAS) research, ARCHAI was introduced to make work in this area more usable, reproducible, and unified. The framework allows standard NAS algorithms to be executed with a single command line, making it easy to experiment with and add new algorithms and datasets. Researchers also proposed an autoML approach to compare any two classification datasets, even if their labels aren’t directly comparable, called Optimal Transport Dataset Distance. If you’re interested in learning more about autoML, check out the Directions in ML Speaker Series, which kicked off in July.

The Semantic Machines research team introduced a new framework for conversational AI (opens in new tab) in which dialogues represented as dataflow graphs make AI more flexible in its ability to adapt to the natural flow of conversation. Along with this, they released the largest, most complex task-oriented dialogue dataset (opens in new tab) to help advance conversational AI research more broadly.

Computer vision moved forward on many fronts in 2020. Researchers created a visual question answering (VQA) evaluation score in their work to understand the connection between visual understanding and neuro-symbolic reasoning. The score betters prior evaluation methods by isolating reasoning from perception in VQA models with a differentiable first-order logic framework. Researchers also investigated how two concepts integral to human reasoning, locality and compositionality, can help to enhance zero-shot representation learning.

Researchers out of Microsoft Research Asia developed methods to improve visual recognition with HRNet and boost photo enhancement with two AI techniques—one that transfers high-resolution texture information to low-resolution images and another that uses variational autoencoders (VAEs) to restore old photos. Other advances in deep generative networks by included Optimus, FQ-GAN, and Prevalent, while researchers also found ways to extend adversarial robustness and training, concepts closely associated with GANs, to transfer learning and causal inference.

Finally, researchers are looking at a future AI landscape that is increasingly multimodal and interactive. However, engineering AI that uses multimodal streaming data in real time is time consuming because programming infrastructures in this area are lacking. Researchers built Platform for Situated Intelligence to provide an open-source framework for experimentation, development, and research in this area.

Continuing a tradition of research with real-world impact

The above research represents a small portion of the great work that was enthusiastically pursued by dedicated researchers at Microsoft Research in 2020, and even if we were to consider all the work produced this year, it would still tell only part of a bigger research story at Microsoft.

2020 Awards

Over the years, the high quality and forward-thinking nature of Microsoft researchers’ work has been recognized by the scientific community. Explore the links below to see how researchers were acknowledged in 2020:   Kori Inkpen receives 2020 Lasting impact Award at CSCW  Susan Dumais receives 2020 SIGCHI Lifetime Research Award  Ant Rowstron Named Fellow of the Royal Academy of Engineering  Explore the complete index of awards recognizing Microsoft researchers’ contributions in 2020 on our News and Awards page.

In a new series of posts this year, we began connecting the dots to provide an overview of how the individual contributions of researchers and their collaborators are coming together to have a profound impact on customers and society at large. Our collection on responsible AI shows how researchers are upholding and advancing the Microsoft commitment to AI grounded in principles that put people first and benefit society, while our collection on reinforcement learning recounts the history of work in the field and its application to Microsoft products and services. Visit our collections archive for more.

2020 has been a year like no other, underscoring the importance of the relationship between research and resilience. As we look to the future, researchers are an important part of answering the big questions that will shape the direction of society. We have been inspired by the research community’s continued commitment to technological advancements—those in direct response to these unprecedented times and those keeping research on all fronts moving forward. We wish you and yours a safe and healthy new year.

To stay up to date on all things research at Microsoft, follow our blog (opens in new tab) and subscribe to our newsletter (opens in new tab) and the Microsoft Research Podcast (opens in new tab). You can also follow us on Facebook,  (opens in new tab)Twitter,  (opens in new tab)YouTube, (opens in new tab) and Instagram (opens in new tab).

---

Opens in a new tab