Introduction

Vision-Language-Action (VLA) models enable broad manipulation capabilities by leveraging large-scale pretraining and robot demonstrations. However, imitation learning can cause small execution errors to accumulate over time, pushing the robot into states that demonstrations did not cover well.

Therefore, we present an object-centric residual reinforcement learning framework that enhances a frozen base VLA without real-world RL. We train a lightweight residual policy entirely in simulation and condition it on object-centric state, proprioception, and the base VLA action rather than raw images. As a result, this observation interface lets the residual transfer zero-shot to the real robot while avoiding the visual sim-to-real gap that affects image-based residuals.

Across five real-robot manipulation tasks, the simulation-trained residual improves average real-robot success from 42% to 76%. In addition, we show that successful residual-corrected rollouts can retrain the base VLA, enabling a self-improvement loop without additional teleoperation.

Method

Stage 1 : Paired Sim/Real VLA via Teleoperation Replay. We start from a VLA trained on real-robot demonstrations, then replay the same teleoperation actions in simulation to train a simulation counterpart. Because identical action trajectories supervise both the real and simulated VLAs, their action distributions remain aligned despite different visual domains.

Stage 2 : Object-Centric Residual RL. Next, we train the residual policy in simulation on a compact observation consisting of 6-DoF task-relevant object poses, proprioception, and the current base VLA action. Pose noise injection and pose dropout make the residual robust to real-world pose-estimation errors.

Stage 3 : Zero-Shot Real Deployment. Finally, at deployment, the real VLA replaces the simulated VLA, and we add the same residual to the base action at every timestep. Since the residual does not observe images, the simulation need not look visually realistic; moreover, deployment requires no real-world RL, distillation, or residual-policy fine-tuning.

Real-Robot comparison

The following videos demonstrate our real-world robot experiments. The baseline VLA fails in the video on the left, whereas the residual framework successfully completes the task on the right.

Cube Lift

Pick and Place

Stack Cube

Stand Cup Up

Main results

Success rates in simulation and real-robot evaluation. We report simulation results as mean ± standard deviation over three seeds. Meanwhile, real-robot results report successes out of 20 trials.

Analysis

To better understand the advantages of our residual framework over the baseline, we conducted a series of quantitative analyses. The charts below highlight the specific improvements in efficiency and generalization.

Action correction

The arrows show the base VLA action, the residual correction, and the combined action. When the base action becomes misaligned, the residual steers it back toward the goal.

Conclusion and future work

Object-centric residual RL combines the generalization ability of VLAs with the precise corrective capability of reinforcement learning. By choosing an observation interface that works in both simulation and reality, we can train the residual entirely in simulation and deploy it zero-shot on the real robot. As a result, the method improves manipulation performance without requiring real-world RL, distillation, or residual-policy fine-tuning.

Beyond direct deployment, the residual-corrected policy also produces better real-robot rollouts. These rollouts can retrain the base VLA and convert task-specific residual corrections into standalone policy improvements. In this way, residual RL can serve not only as an inference-time correction module, but also as a mechanism for generating higher-quality supervision.

Finally, future work includes extending the approach to more cluttered scenes and broader task families. It will also be important to develop more autonomous mechanisms for identifying which task-relevant objects should condition the residual policy. Ultimately, we view object-centric residual RL as one step toward robot learning systems that can use simulation to improve real-world behavior with less human intervention.

The post Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement appeared first on Microsoft Research.

For a robot to truly “understand human speech,” “perceive the world,” and translate that comprehension into precise movements, the key lies in whether it possesses a “brain” capable of linking vision, language, and action. This is the Vision-Language-Action (VLA) model—an intelligent framework capable of interpreting arbitrary human language instructions and executing complex tasks in the real world.

To enable VLA models to efficiently learn human behavioral patterns, Microsoft Research Asia introduced the VITRA pre-training method. This innovative approach automatically transforms unstructured, real-world human videos into structured VLA formats consistent with existing robotic data. Thanks to this automated and scalable methodology, the resulting VLA models demonstrate enhanced zero-shot predictive capabilities in unseen environments. Furthermore, they can be efficiently fine-tuned with minimal robotic data for real-world tasks, exhibiting superior generalization across novel objects and environments.

Self-Sufficient Robotic Data Collection Methods Face Multiple Bottlenecks

The success of Large Language Models (LLMs) is inseparable from the trillions of text tokens available on the internet. Similarly, if VLA models are to possess robust generalization capabilities, they require massive and diverse training datasets. However, the traditional “self-sufficient” approach to robotic data collection is almost equivalent to a high-cost “data gold-mining” endeavor.

These traditional methods typically rely on two paths: first, manual teleoperation of robots in laboratory settings to execute tasks and record data; and second, the generation of synthetic data through simulation environments. The former is constrained by hardware deployment and labor intensity, while the latter struggles to faithfully replicate the physical interactions and environmental complexity of the real world.

Furthermore, these approaches are plagued by the following three primary challenges:

• Prohibitive Costs: The expenses associated with deploying and maintaining robots, as well as employing a large workforce of operators for teleoperation training, are staggering.
• Low Efficiency: Data acquisition is painfully slow, making it nearly impossible to achieve the exponential scaling characteristic of internet-scale data.
• Insufficient Diversity: Constrained by limited resources, the variety of objects, manipulation skills, and environmental scenarios within these datasets is extremely restricted. This failure to cover the complexities of the real world severely bottlenecks the model’s generalization capabilities.

“We discovered that a massive treasure trove of data is already all around us—namely, the vast ocean of human activity videos on the internet,” notes Yu Deng, Senior Researcher at Microsoft Research Asia. “From cooking tutorials and home repairs to handicraft making and daily chores, these videos document human actions and experiences across a diverse range of real-world environments.”

The researchers contend that since the current goal of humanoid robot development is to approximate human capabilities, human data serves as the ultimate ‘textbook’ for robot training.

“Why not allow VLA models to learn directly from human videos?” Yu Deng proposes. “We can view the humans in these videos as robots and their hands as the robot’s end-effectors. By automatically converting these unstructured, unlabeled human videos into trajectory data that is perfectly consistent with robotic data formats, we can elevate the scale, quality, and diversity of robot training data to unprecedented levels.”

Three Steps to Transform Unstructured Human Videos into Structured Robot-Ready Data

Standard VLA robotic data typically consists of atomic tasks, such as “pick and place an object,” “open or close a container,” or “push and pull” movements. Each VLA data entry comprises three essential components: the video of the robot’s physical manipulation, human-authored linguistic instructions, and the 3D motion data of each robotic joint during execution—including translation, rotation, and the state changes of the dexterous hand.

How, then, can massive, cluttered, and unlabeled human videos be transformed into structured robotic training data? The researchers decomposed the workflow into three key steps:

Step 1: Perceiving the Action — 3D Trajectory Reconstruction to Restore Real-World Spatial Motion

Most human videos are captured by a single camera, providing only 2D images of hands that fail to directly reflect their true positions in three-dimensional space. To address this, researchers employed cutting-edge 3D vision technologies—such as depth estimation, camera pose tracking, and 3D hand reconstruction. >

The process begins by determining whether the camera is stationary or in motion, followed by the automatic calibration of camera parameters. Ultimately, it reconstructs the 3D hand pose for every frame. This reconstruction captures not only the spatial position and rotation of the wrist but also the flexion of each finger joint and the camera’s own motion trajectory. Consequently, the VLA model no longer simply “watches a video”; it gains a genuine understanding of hand movements within a 3D spatial context.

Step 2: Atomic Action Segmentation — Automatically Decomposing Long-Horizon Sequences Based on Kinematic Laws

Human manipulations are typically continuous sequences, such as “taking vegetables from the fridge → washing them → placing them on the counter → chopping.” However, robot training requires atomic-level task snippets, such as a standalone “placing vegetables on the counter.” During their research, the team discovered that human motion patterns in real-world scenarios exhibit a rhythmic cadence akin to “breathing”: during the transition between different actions, the hand’s motion pattern undergoes significant changes, where the movement velocity hits a momentary local minimum. For instance, the hand decelerates to align before grasping a cup and pauses briefly after releasing it before initiating the next movement.

Leveraging this kinematic regularity, researchers utilized the velocity minima within 3D trajectories as “cut points” to automatically segment long videos into multiple short snippets. This method does not rely on predefined action categories; instead, it ensures that each snippet contains exactly one atomic action, perfectly matching the “short-task” granularity required for robot training.

Step 3: Understanding the Action — Integrating 3D Trajectories to Generate Precise Linguistic Instructions

Once segmented, video snippets must be paired with accurate linguistic instructions before they can be used for training. The researchers aimed to leverage Vision-Language Models (VLMs) for this task; however, the complexity of real-world scenarios often interferes with a model’s judgment. Feeding raw video snippets directly into a model can lead to misinterpretations of hand movements or the objects involved—for instance, misidentifying “picking up a spoon” as “picking up a fork.”

To mitigate this, the researchers overlaid the reconstructed 3D hand trajectories onto the video frames, effectively drawing an “action roadmap” on the canvas. This allows the model to clearly “see” the hand’s motion path. By providing this explicit spatial context, the model can generate linguistic instructions that better align with the requirements of robot training, significantly enhancing annotation accuracy.

Through this fully automated data transformation pipeline, the researchers constructed a large-scale VLA dataset. Trained on multiple public first-person perspective video datasets (such as Ego4D and EPIC-KITCHENS), the processed dataset contains over 1 million action snippets and 30 million frames, covering diverse scenarios including kitchen cooking, home cleaning, handicraft making, and construction maintenance. >

“This encompasses nearly all types of manipulations in both daily human life and professional work, far exceeding the diversity and scale of existing robotic data. It provides a scalable and sustainable new paradigm for training robots with general-purpose dexterous manipulation capabilities,” explains Yu Deng.

Hidden “Gems”: Two Clever Nuances in the Human-Data-Aligned VLA Model

With a high-quality human behavior dataset in hand, a VLA model capable of fully comprehending this data is essential. Building upon mainstream VLA architectures, researchers implemented two critical optimizations tailored to the unique characteristics of human video data. These refinements enable the model to more accurately learn authentic human manipulation behaviors, effectively paving the way for subsequent transfer to physical robotic platforms.

First, incorporating additional parameters to make the model “lens-aware.” Human videos are captured using a wide variety of devices—smartphones, DSLRs, action cameras—each with distinct focal lengths and fields of view (FOV). For the same movement, a telephoto lens makes the subject appear closer, while a wide-angle lens makes it seem further away. If the model fails to account for these optics, it may misjudge the scale and distance of actions in real 3D space.

To mitigate this, researchers fed camera intrinsic parameters, such as focal length and field of view, as auxiliary inputs into the model. This enables the model to automatically calibrate its spatial perception of the frame, eliminating cognitive biases caused by hardware variations and allowing for more rational 3D action predictions.

Second, introducing a Causal Attention mechanism to address incomplete action snippets. After segmenting real-world videos, a specific issue arises: some snippets may not contain a fully completed action. For instance, a “pick up cup” sequence might be cut right at the moment the “hand touches the cup,” with the subsequent lifting phase missing. Conventional methods align sequence lengths by appending “zero-actions” (non-motion frames) at the end, which inadvertently signals to the model that “motion should cease upon touching the cup,” leading to flawed behavioral patterns. Furthermore, real-life movements often contain “motion noise”—superfluous or meaningless actions at the end of a task—which can interfere with action prediction.

To solve this, researchers transitioned the action generation module to a Causal Attention mechanism. This ensures that when predicting the current action, the model can only attend to preceding frames in the sequence, rather than “seeing” future zero-padding or meaningless noise. This unidirectional attention structure simulates real-world temporal causality, preventing future information from contaminating current predictions. As a result, even when faced with incomplete snippets or environmental motion noise, the model correctly learns the authentic continuous manipulation logic of humans.

Powerful Zero-Shot Prediction and Generalization: Ushering in a New Scalable Paradigm for VLA Models

How effective is this novel VLA pre-training method?

The researchers conducted validations on both human hand motion prediction and real-world dexterous robots, with results exceeding expectations. Models pre-trained on human data using this method demonstrated superior zero-shot predictive capabilities in hand motion forecasting. Following fine-tuning, the robots’ execution success rates in real-world tasks improved significantly, showcasing exceptional generalization abilities.

Outstanding Zero-Shot Performance in Hand Motion Prediction: When faced with entirely unseen scenarios—such as unfamiliar kitchens or previously unencountered furniture—the model can predict plausible human hand movements as long as a natural language instruction is provided (e.g., “retrieve an object from the drawer” or “pour water into a specific cup”). Its performance far surpasses models trained solely on limited laboratory datasets.

Effective “Inheritance” of General-Purpose Visual Understanding: The researchers further discovered that a VLA model trained via full-parameter fine-tuning on human hand data can, to a large extent, retain the general visual understanding capabilities of the original Vision-Language Model (VLM), even without additional training tricks. For example, when presented with several photos of objects and tasked with predicting the hand motion required to grasp one specific photo, the model can accurately select the target and generate a plausible grasping action—even though such abstract concepts were never explicitly introduced during the VLA pre-training phase.

Significant Fine-tuning Efficiency on Physical Robots: Using only around 1,000 teleoperation data samples for fine-tuning, the average success rate for a dexterous hand across four tasks—”pick-and-place at random positions,” “functional grasping (e.g., grasping a pot handle),” “pouring water,” and “sweeping”—soared from 30%-40% (without human pre-training) to over 70%. Notably, the success rate for the “pick-and-place” task exceeded 80%. This marks a substantial improvement over previous pre-training methods, such as those relying on large-scale robotic datasets, human video prediction, or latent action pre-training.

Superior Generalization Capabilities: When encountering objects never seen during training, such as new types of thermoses or oddly shaped toys, the robot maintained a success rate of approximately 70%. Even with entirely unfamiliar object categories, such as an unseen laptop power adapter, the robot could leverage the “similar object grasping logic” learned from human videos to execute accurate operations, far surpassing the generalization limits of prior methods. The model has not merely memorized specific motions but has mastered the underlying “manipulation logic,” enabling it to handle novel situations beyond its training distribution. This is attributed to the fact that human data diversity far exceeds that of laboratory robotic data, and the motion patterns between human hands and dexterous robotic hands are more congruent, facilitating smoother knowledge transfer.

The research on VITRA provides a highly promising, scalable, and extensible pre-training paradigm for generalizable VLA models. As robotic hardware continues to advance, real-world human video data will serve as an inexhaustible training resource, allowing robots to continuously acquire a vast array of human manipulation skills. Perhaps in the near future, we will witness truly general-purpose robotic assistants entering our daily lives.

As Yu Deng puts it, “Humans represent the ultimate form that humanoid robots strive to achieve.” By enabling robots to “watch” and “understand” authentic human behavior, we are accelerating the transition of this vision from a concept to reality.

The post VITRA Redefines VLA Pre-training Paradigms via Human Video Reconstruction appeared first on Microsoft Research.

This capability, called GUI grounding, enables computer use agents to locate specific elements on a graphical user interface (GUI). GUI grounding is the agent’s perception system that translates instructions like “click the Submit button” into exact screen coordinates.

Currently, grounding models succeed only 65% of the time, far from reliable enough for everyday use. A research team from Microsoft Research Asia conducted an extensive study to understand why, examining every stage of how these models are built and trained. Their work produced Phi-Ground (opens in new tab), a new model family that achieves state-of-the-art performance across all five grounding benchmarks among comparably-sized models.

The significance of GUI grounding

Computer use agents could transform how we interact with software in the digital world. These agents operate through graphical interfaces exactly as humans do, without requiring specialized APIs and allowing straightforward human oversight. This universality gives computer use agents broader potential than traditional robotic systems or specialized web automation tools.

GUI grounding serves as the agent’s interface with the digital world, the mechanism that determines whether the system succeeds or fails at its tasks. An agent that can’t reliably find what it’s looking for on screen is fundamentally limited, regardless of how well it can reason or plan.

Building better training data

Training more capable grounding models requires large-scale, high-quality data. The research team started with web pages from CommonCrawl, a massive public repository of internet content, and rendered them as screenshots to generate training examples. Yet web data contains substantial noise that can derail model training, from broken layouts and malformed pages to irrelevant content.

To address this, the team developed a multi-stage workflow to filter and refine the data, illustrated in Figure 1.

Beyond CommonCrawl, the team incorporated open-source datasets, screenshots from web searches, and manually annotated examples for everyday scenarios, e.g. grounding for office software, where precision matters most. Together, these sources formed the training foundation for the Phi-Ground model family. Table 1 shows the final composition of the training data, including how many times each dataset was used during training (epoch) and its relative importance in the learning process (weight).

How to train grounding models

The team discovered that the order in which text and images are fed into the model significantly impacts performance. They tested two approaches: inputting text before images, and the reverse. The results in Table 2 show that text-first yields substantially better outcomes.

Why does this matter? Transformer models, the architecture underlying most modern AI systems, use causal processing, meaning earlier inputs cannot be updated using information from later ones. When images come first, the model processes visual information without receiving the user’s instructions (like “click the Submit button”). When text comes first, the model interprets visual information after receiving the instructions. It knows what to search for as it processes the image. For perception tasks like grounding, this instruction-aware process directly influences results. A simple change in input order produces this effect.

Researchers also examined computational costs during testing. These costs depend not only on model size but also on the number of image tokens—the units into which images are divided for processing. For perception tasks, more image tokens generally improve performance, but at what point do the benefits plateau?

To answer this, the team investigated the relationship among model size, image token count, and training data volume. Such experiments can guide developers balancing training efficiency with application speed, which are practical concerns beyond raw parameter counts.

Figure 2 illustrates training results for six models with different sizes and image detail levels. Based on these evaluations, inference time generally aligns with the computational relationship, shown on the x-axis. Notably, many current studies report only the number of parameters when discussing model performance, overlooking computational factors like image token count, a gap that this research addresses directly.

In the experiments where the model architecture is fixed, the team found that for advanced benchmarks like ScreenSpot-Pro and UI-Vision, image token count significantly impacts performance. When the image token count falls below a certain threshold, they become a bottleneck. The model cannot perceive small interface elements, reducing accuracy. However, beyond approximately 2,000 image tokens, the benefits plateau. Additional tokens provide little additional benefit.

Evaluation results

Table 3 presents test results comparing several open-source models with fewer than 10 billion parameters across five GUI grounding benchmarks. The upper section shows results using the benchmark’s standard reference expressions, typically brief instructions or short phrases. The lower section shows results when researchers used OpenAI’s o4-mini model to generate longer, more detailed reference expressions, which were then tested by the grounding models.

The Phi-Ground model is trained specifically for agent applications, with training data consisting primarily of various reference expressions. As a result, the model achieves state-of-the-art results across all benchmarks in the agent setting. On ScreenSpot-Pro, it reached 55% accuracy. On UI-Vision, it attained 36.2%, the highest score reported for this benchmark. The model’s performance on the Showdown benchmark surpassed that of commercial systems like OpenAI Operator and Claude Computer Use.

Enabling Copilot to understand onscreen content

The core technology of the Phi-Ground model has been integrated into the Vision Highlighting feature of Windows Copilot. As shown in Figure 3, Copilot can guide users step-by-step through visual tasks, such as helping them construct a bubble graphic.

Beyond advancing GUI grounding, this research demonstrates how systematic study of training methods can unlock performance gains across multimodal AI systems. The integration into Windows Copilot marks an early step toward computer use agents that can genuinely assist with everyday digital tasks.

The post Phi-Ground: Improving how AI agents navigate screen interfaces appeared first on Microsoft Research.

To address these limitations, researchers at Microsoft Research Asia developed Deep Video Discovery (DVD), an agentic AI framework for long-video analysis. DVD divides long videos into shorter clips for individual analysis, then uses LLM-based reasoning to plan next steps and select appropriate tools. The agent retrieves needed information and uses it to answer complex questions about the video.

How DVD works

DVD operates through a simple cycle: observe the video content, analyze what it means, and choose the next action. Current video-analysis systems follow rigid, predesigned steps that have difficulty adapting to different tasks. In contrast, DVD adjusts its approach based on information it has gathered so far. To support this flexibility, the system operates in two stages:

Stage 1: Building a searchable video database

The system converts long videos into a structured database, dividing them into five-second clips and extracting information at three levels:

• Global: Provides a topic-level summary of the video.
• Clip‑level: Includes subtitles and brief text descriptions of each segment.
• Frame‑level: Includes individual frames and visual details captured moment by moment.

Stage 2: Retrieving information and generating answers

The system uses three core tools to search the database:

• Global browse: Provides high‑level context and video summaries.
• Clip search: Retrieves clips that match a description and returns relevant results with subtitles and timestamps.
• Frame inspect: Examines a specific moment in the video and extracts fine visual details; it can also answer questions about what appears in that frame.

The LLM serves as the system’s orchestrator, running repeated observe-reason-act cycles based on gathered information. This design gives the agent autonomy, ensures that its answers stay grounded in actual video content, and allows the system to break complex questions into smaller, more manageable sub-questions.

DVD achieves state-of-the-art performance across benchmarks

DVD achieved state-of-the-art performance across multiple long‑video benchmarks (Table 1). On the challenging LVBench dataset, DVD reached 74.2% accuracy, outperforming all existing methods, a 13.4‑point gain over the previous best method, MR. Video. When transcript data was available, accuracy rose to 76.0%.

DVD also exceeded previous state-of-the-art performance on three other long‑video benchmarks: LongVideoBench, Video MME Long, and EgoSchema, surpassing human‑level accuracy (approximately 76%) on EgoSchema.

The choice of reasoning model critically affects DVD’s performance (Figure 2). Replacing the reasoning model with OpenAI o4‑mini or GPT‑4o causes sharp performance drops, indicating that limited reasoning capability breaks down agent’s process. Different models also show distinct patterns in how they use tools; how deeply they analyze videos; and how accurately they respond. For example, GPT‑4o often exhibits “overconfidence,” stopping its analysis prematurely. These observations offer practical guidance for designing future agents and developing foundational LLMs.

Toward more comprehensive video understanding

As video content becomes richer and more complex, enabling AI to interpret and reason about what it captures, not just identify individual elements, is a central challenge in video comprehension. DVD offers one path forward through an agentic approach that is interpretable, plannable, and collaborative.

Looking forward, researchers at Microsoft Research Asia are working to develop agents with stronger contextual awareness, and more advanced reasoning capabilities, advancing toward AI systems that can handle complex videos with greater depth, precision, and automation.

The post Deep Video Discovery: Using agentic search to analyze long-form video appeared first on Microsoft Research.

For Yansen Wang, a senior researcher at Microsoft Research Asia, this philosophy has guided his work at the intersection of AI and neuroscience.

Wang’s interest in science began early. While his classmates spent hours searching for information to complete an assignment, he solved it in 10 minutes by writing a few lines of code. Since then, programming became his way of tackling complex challenges, and the satisfaction of creating practical solutions fueled his passion for computer science. As his studies advanced, his goals crystallized: he wanted to create technology that genuinely serves people.

This people-centered approach has shaped Wang’s career. As an undergraduate at Tsinghua University, he witnessed AI defeat the world’s Go champion—a moment that sparked a realization: AI could help us understand how humans think. This insight led him to pursue research in multimodal AI for his master’s studies before joining Microsoft Research Asia – Shanghai. Today, in the AI/ML Group, Wang works to bridge AI and neuroscience.

A two-way journey between AI and neuroscience

Wang’s research focuses on two complementary directions: advancing neuroscience by decoding how the brain works (AI for Brain), and drawing inspiration from the brain to improve AI architectures and algorithms (Brain for AI).

For the AI for Brain project, Wang and his colleagues use noninvasive EEG(electroencephalogram) to build brain-computer interfaces, which record the electric signals from our brains without surgery, as their research platform. “Our understanding of the brain is still very limited,” he explains. “Its multitasking abilities and rapid adaptability are far beyond what AI can achieve, so we’re using AI to analyze the EEGsignals and uncover the links between perception, intention, and brain activity.”

The team has already made substantial progress. In terms of perception, they have decoded broad visual features, such as colors and simple moving scenes. Using diffusion models, they transformed neural signals into matching visual content and developed EEG2Video, a baseline model that reconstructs video clips from EEG signals. To improve generalization across different contexts, the team has built multiple datasets linking EEG signals to everyday behaviors. See the video below for an example of this work.

For controlling with commands, Yansen and his team tackled the challenge of decoding letters from brain signals. They introduced an innovative codebook approach: instead of having participants imagine letter shapes, which are difficult for devices to recognize, they guided participants to imagine body movements, mathematical calculations, and other semantic information that devices could more easily recognize. AI then mapped these signals to specific letters. With portable devices, this method has achieved 30%–40% accuracy with 36 options (26 letters and 10 digits).

“We’re now working to extend this approach to controlling mobile phones and interacting on the web, exploring new interaction modes beyond letter input,” he says.

Still, the approach of using noninvasive brain-computer interfaces comes with many challenges. EEG signals have low signal-to-noise ratios and are easily disrupted by environmental and physiological interference, such as eye movements or muscle activity, making reliable readings difficult on portable devices. EEG data is scarce and must be collected in a controlled setting. And individual differences mean that systems often don’t generalize well across users and tasks.

“We’re advancing research on EEG foundation models and hope to make them more robust with more data and larger models, much like large language models,” Wang explains.

Learning from the brain to make AI more efficient

For the Brain for AI project, Wang and his colleagues are exploring how brain function can address AI’s energy demands.

“The brain can accomplish tremendous thinking and computation with just the energy supplied from a bowl of rice, while AI requires vast resources and electricity to achieve similar results,” he observes. “Even more remarkable is that the brain efficiently handles many tasks without complex networks, like fine motor control. People only need a few examples to learn new tasks, but current AI models often need massive amounts of data to relearn. There must be design principles at play here that AI can learn from.”

The key differences lie in how neurons are structured and operate. Neurons use a spiking mechanism, firing and transmitting signals only when they reach their activation threshold. This results in extremely low energy consumption when they are at rest. Artificial neural networks, by contrast, perform large-scale computations even when there is very little information to process, using far more energy than the brain requires for similar tasks.

Using this insight, Wang and his team developed a more efficient spiking neural network (SNN) framework. In time-series prediction tasks, SNNs now perform comparably to traditional neural networks but can theoretically reduce energy consumption to a quarter of the latter, offering a new path for low-power AI (Figure 1).

“The spiking neural network research is just one part of our work,” says Wang. “Neurons in the brain are sparsely connected—each one typically links to only a few nearby neurons, while in artificial neural networks, a single neuron connects to thousands of others. The brain’s sparse connectivity also helps reduce energy consumption. If we continue learning from how the brain operates, AI will be able to generalize better and become more energy-efficient.”

Breaking boundaries: From outsider to domain expert

For Wang, cross-disciplinary research in AI and neuroscience was new territory. He had no formal training in neuroscience or EEG analysis, but through dedicated study, active collaboration, and strong team support, he developed deep expertise.

When he began EEG research, Wang studied medical textbooks and sought guidance from collaborating physicians. “I set a rule: temporarily set aside my AI perspective and approach this as a doctor would. I studied medical textbooks thoroughly and learned to read EEG signals. Only then did I consider how AI could help.” This approach helped him understand clinical problems rather than imposing familiar frameworks.

Cross-disciplinary collaboration is not just about combining knowledge; it’s about how different perspectives collide. For example, in research on epilepsy detection, Wang discovered that AI researchers and physicians approach problems differently. AI researchers often assume that with enough data, models can learn to identify features of epileptic seizures, such as spikes and slow waves. But physicians can quickly spot the rare abnormality in massive amount of hard-to-interpret EEG signals based on experience. Models can miss these abnormalities, even when trained on vast amounts of data.

“This showed me that machine learning progress cannot rely solely on brute force. In data-scarce fields like medicine, we must incorporate domain expertise and build in the right assumptions to improve model performance.”

To help researchers build cross-domain knowledge, Microsoft Research Asia – Shanghai established a neuroscience study group, with weekly classes, homework, and discussions. After six months, Yansen had learned the fundamentals of neuroscience and gained practical guidance from senior researchers. “This collective learning atmosphere means we’re no longer working in isolation but instead growing together as a community,” he says.

Microsoft Research Asia encourages open exploration and open exchange. At the Shanghai lab’s weekly “Grand Challenge” meetings, researchers rigorously challenge one another’s work. “At first, I wasn’t used to this style of questioning,” Wang admits. “But I realized that these challenges expose blind spots and allow research to improve through iterative refinement. The toughest questions often lead to the most important breakthroughs.”

Research with purpose: Building AI that serves people

For Wang, technology should serve people. Whether developing brain-computer interfaces or creating explainable AI for Go, the focus of the work should be on making AI useful and accessible.

In 2022, Yansen and his colleagues launched what they called a “human salvation project” for Go players. AI had surpassed top players, causing anxiety among professionals. Players could imitate AI moves but couldn’t understand the reasoning behind them. They memorized patterns without developing their own strategic thinking. “I thought, ‘If AI could explain its logic, players could truly understand the strategies behind the moves,’” Wang says. “We wanted to help people improve alongside AI.” Wang and the team are actively collaborating with Go lovers and professional Go players to verify the feasibility of the explanations. “That is so impressive!” one of the teachers in a Go learning institute says, “and I see the future of teaching human how to play Go”.

For Wang, this captures what drives his research: not the number of papers, but tangible impact. Perhaps it’s the moment when a player grasps a brilliant move, or when someone finds more convenient ways to interact with devices, or when researchers apply new approaches for energy-efficient AI.

“At Microsoft Research Asia, I can follow my interests and work with partners to solve meaningful problems for humanity,” he says.

The post Where AI meets neuroscience: Yansen Wang’s pursuit of human-centered innovation appeared first on Microsoft Research.

This becomes clear when comparing test conditions to real-world use. Current benchmarks use unrealistically large GUI elements—on a typical monitor, buttons and icons occupy a much smaller fraction of the screen—and test only a limited subset of element types like checkboxes and Submit buttons. Moreover, these benchmarks rely on simple, explicit instructions like “click the Save button” while neglecting the implicit language people actually use, like “click where I can change my password.”

To address this gap, a research team at Microsoft Research Asia has developed UI-E2I-Synth (opens in new tab), a large-scale data synthesis method that generates more realistic data, and UI-I2E-Bench (opens in new tab), a benchmark that better reflects actual computer use. The related paper has been accepted by The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025) Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Specifically, UI-I2E-Bench reflects typical 1080p and 1440p displays, labels whether each instruction is explicit or implicit, and includes a broader, more balanced range of UI elements compared with leading benchmarks, as shown in Figure 1.

How the system training data

UI-E2I-Synth uses large language models (LLMs) to automatically generate realistic user instructions, reducing the manual effort required to label screenshots. The system works in three stages, each building on the previous one.

Gathering and organizing interface data. First, it collects UI screenshots and accompanying metadata from various platforms, including web pages, Windows applications, and Android interfaces. An automated tool then identifies and catalogs each UI element, recording details like whether it’s a button or text field, what it displays, and where it appears on screen. This step produces an organized catalog of UI elements that serves as the foundation for generating instructions.

Generating natural descriptions. Next, OpenAI’s GPT-4o analyzes these cataloged elements to create different ways users might realistically describe them, both explicit descriptions (e.g., “the blue Submit button in the top-right corner”) and implicit ones (e.g., “the confirmation button” or “the button next to the username”). This variety captures the range of ways users might refer to the same interface element.

Creating complete instructions. Finally, GPT-4o pairs these descriptions with specific actions to create complete, natural-sounding user instructions that reflect how people actually interact with interfaces, for example, “Click Send” or “Enter my password.” The result is a diverse set of instructions that more accurately reflects user behavior.

This process is illustrated in Figure 2.

Training and testing more realistic models

From UI-E2I-Synth’s synthesized data, the team created UI-I2E-Bench, a new benchmark that reflects real-world conditions. It includes labels identifying the type of element and whether instructions are explicit or implicit, along with a realistic element-to-screen ratio—providing a rigorous test of vision-language models’ (VLMs) GUI grounding capabilities.

To evaluate the effectiveness of the proposed data synthesis pipeline, the team used almost 10 million individual instructions generated by UI-E2I-Synth to train two VLMs: UI-I2E-VLM-4B and UI-I2E-VLM-7B. UI-I2E-VLM-7B performed well across multiple benchmarks, including UI-I2E-Bench, using only 72% of the training data required by comparable state-of-the-art models.

The models performed especially well at handling indirect instructions and locating small elements, and more easily recognized challenging element types like icons and text entry fields. The results also confirmed that existing benchmarks overestimate model capabilities due to their unrealistically simple test conditions. The details of these results are shown in Table 1.

Diagnosing model strengths and weaknesses

The detailed labels of UI-I2E-Bench enabled the research team to analyze where models succeed and fail. The analysis revealed several key patterns.

Instruction complexity. Models showed the most improvement in handling implicit instructions. As shown in Table 2 leading models struggled with these realistic instructions, with  lagging by 12 percentage points, compared with on explicit instructions. Interestingly, systems powered by GPT-4o performed well on implicit instructions but struggled with explicit ones, primarily due to difficulty in locating small elements and uncommon interface components.

Element size matters. The smaller the interface element, the more accuracy dropped across all models. This confirms that small elements and high-resolution images are critical factors in model testing. Models trained with UI-E2I-Synth, which uses more training data and processes images with higher detail, performed better in locating these small elements.

Underrepresented element types. Existing models showed clear shortcomings with less common interface elements like icons and text entry fields. By balancing the distribution of element types in training data, UI-E2I-Synth directly addresses this gap and improves model performance.

Raising the bar for UI Grounding

UI-E2I-Synth and UI-I2E-Bench address fundamental gaps in how GUI grounding models are trained and evaluated. Rather than relying on oversimplified benchmarks, this approach prepares models for the messy reality of actual computer interfaces—where elements are small, diverse, and instructions are often ambiguous.

The research establishes more rigorous standards for the field and could pave the way for AI assistants that can reliably navigate real-world software, moving these tools closer to practical deployment.

The post UI-E2I-Synth: Realistic and challenging UI grounding benchmark for computer-use agents appeared first on Microsoft Research.

Yet despite their promise, these agents perform poorly in practice. They typically draw on external knowledge—information retrieved from the web that describes how to navigate the interfaces in question—and use it to interpret what’s on the screen and adapt to different environments. However, these agents often fail to translate this knowledge into successful action—a problem researchers call the “knowledge–action gap.”

A recent study shows that even when the instructions are 90% correct, agents perform tasks successfully only 41% of the time. This disconnect between having the needed information and effectively applying it, illustrated at the top of Figure 1, can lead to a frustrating user experience.

To address this, researchers at Microsoft Research Asia developed UI-Evol, a ready-to-use component that integrates into an agent’s workflow and relies on the actual user interface for guidance. UI-Evol continuously updates its interface knowledge, helping make agents more accurate and reliable when completing tasks, as shown in the bottom of Figure 1.

This work has been recognized by the research community, with the team’s findings accepted at the ICML 2025 Workshop on Computer Use Agents (opens in new tab).

How UI-Evol works

UI-Evol addresses the knowledge-action gap through a two-stage process. The first stage, called retrace, records the exact steps an agent takes to finish a task. In this way, the system captures the specific clicks, keystrokes, and other actions that led to the result.

The second stage, critique, reviews those actions against instructions drawn from outside the application. If it finds mismatches, it adjusts the knowledge so that the steps reflect what actually works in practice. Together, these two stages turn external instructions into tested, reliable guidance for agents. This process is illustrated in Figure 2.

Assessing UI-Evol’s effect on performance, reliability

The research team tested UI-Evol on Agent S2, a state-of-the-art computer-use agent. They used the OSWorld benchmark, designed to evaluate multimodal agents on open-ended computer tasks involving real software and workflows. They found that UI-Evol not only improved performance but also made the agent’s behavior more dependable.

Computer-use agents have long shown what researchers call “high behavioral standard deviation.” In plain terms, the same agent, given the same task, may act differently each time it tries to carry it out. This unpredictability has not been a central focus of earlier work, yet it is precisely what limits agents’ usefulness in real-world applications.
With UI-Evol, that pattern shifted. Experiments with agents based on leading LLMs, like GPT-4o and OpenAI-o3, showed not only higher success rates (Table 1) but also greater consistency with UI-Evol.

What this means for practical AI

The introduction of UI-Evol tackles a problem that has long challenged computer use agents since their inception: the gap between what they know and what they can reliably do. As these agents move from research labs to real-world settings such as office automation, virtual assistants, and robotic process automation on software, consistency matters as much as capability.

UI-Evol’s approach—learning from actual agent behavior rather than relying on external knowledge alone—offers a path forward. It’s not only about making agents smarter; it’s about making them dependable enough to trust with real work.

The post UI-Evol: Compute-use Agents Act on Knowledge appeared first on Microsoft Research.

One of the key challenges in this transformation lies in generating documents that are not only accurate in content but also well-structured and visually coherent. While most research has focused on improving text quality, the structural and stylistic dimensions of professional documents—layout, hierarchy, and readability—remain underexplored.

To address this gap, Microsoft Research Asia, in collaboration with The Chinese University of Hong Kong and the University of Chinese Academy of Sciences, has developed DocReward, a reward model that evaluates the structural and stylistic quality of AI-generated documents. By guiding agents to produce outputs that are clear, organized, and well-presented, DocReward provides crucial support for automated document creation.

Deep Research is an agent that gathers, analyzes, and synthesizes information from multiple sources into coherent, well-structured documents. Combined with DocReward, it completes a full workflow—from research and information integration to polished document presentation—laying the groundwork for transforming traditional office software into agent-driven systems.

Task modeling: Evaluating document structure and style

DocReward assigns quality scores to documents based on their layout and visual characteristics. For example, consider a set of documents, {D_i}, where each document includes both its text content D_{text, i} and corresponding rendered image D_{img, i}. The reward model assigns scores to these documents to reflect their quality in terms of structure and stylistic presentation.

For a group of documents with identical content, the goal is for the reward model (R_θ) to predict scores in an order consistent with the true ranking of their structural and stylistic quality (π*). In doing so, the model learns to distinguish differences in layout and design even when the text remains the same, improving the accuracy of its evaluations.

Mathematically, this is expressed as the following:

Definition of document structure and style professionalism:

• Structure: Proper use of whitespace and margins; clear section separation; consistent text alignment; paragraph spacing and indentation; standardized headers and footers; and overall logical organization.
• Style: Appropriate fonts (type, size, and color); readable heading hierarchy; effective use of bold and italics for emphasis; clear bullet and numbering formats; and consistent formatting throughout the document.

Constructing DocPair, the foundation for DocReward

To train DocReward, the research team built the DocPair dataset, which contains 117,000 document pairs spanning 32 domains and 267 document types. This diverse dataset enables the model to be optimized through preference learning to accurately assess structural and stylistic quality across a wide range of documents.

As shown in Figure 2, constructing the DocPair dataset involves three steps:

1. Curating high-quality professional documents

The team began by collecting a broad set of Microsoft Word files, ranging from formal institutional documents to routine business correspondence. Data sources include:

• Government and institutional documents, which make up the GovDocs1 and NapierOne datasets. GovDocs1 contains a wide range of U.S. government materials, including policy reports, administrative forms, statistical reports, meeting minutes, and more. NapierOne features office documents from public institutions, all characterized by strong structural and stylistic standards.
• Web documents, which consist of professionally authored files from the CommonCrawl database, spanning business, education, the nonprofit sector, and medicine. These include proposals, syllabi, newsletters, technical manuals, and policy briefs, contributing to broad diversity in document formats and presentation styles.

To ensure data quality, all documents were converted to .docx format and filtered to remove abnormal or incorrectly formatted files. The large language model (LLM) GPT-5 was then used to automatically score structure and style on a 0–10 scale, retaining only those scoring above 8.

The resulting dataset spans 32 domains and 267 document types and serves as the basis for subsequent document-pair construction. Figures 2 and 3 show the distribution of the top 10 domains and top 30 document types.

2. Expanding source documents via agents

To create document sets with identical text but varying structure and style, the team designed two types of document-generation agents:

• Text-to-document generation agent: Extracts plain text from source documents, removes all structural and stylistic information, and then uses advanced LLMs (GPT-4o, Claude Sonnet 4, GPT-5, etc.) to generate .docx documents through python-docx code.
• Structure and style optimization agent: Further refines the synthetic documents by referencing original human-written examples. This process involves two stages—first generating an optimization plan, then modifying .docx files via python-docx to improve structure and style.

3. Document ranking and annotation

Within each document group, all samples share the same text content. The team constructed two types of comparison pairs:

• Human vs. synthetic documents: When a pair includes a real human-written document, that version is labeled as more professional.
• Synthetic vs. synthetic documents: When both documents are synthetic, a human-written reference document is used as a guide, and GPT-5 annotates which synthetic version exhibits higher structural and stylistic quality.

The final DocPair dataset provides a solid foundation for training DocReward. When multi-page visual renderings are input to the vision encoder, a regression head is added to the language model. A special token is placed at the end of each image sequence, and its hidden state, processed by the regression head, predicts the document’s overall score.

Figure 4 illustrates the overall DocPair data construction process, summarizing the three main stages described above.

Training and evaluation

Training

DocReward is trained using the Bradley-Terry (BT) loss to learn from paired document preferences. Each document’s pages are input into the model, which outputs a score representing its structural and stylistic quality. The BT loss encourages DocReward to assign higher scores to preferred documents, helping it reliably distinguish differences in structure and style.

Mathematically, this is expressed as the following:

Experiments and evaluation

The research team conducted a series of experiments to test DocReward’s effectiveness in evaluating document structural and stylistic quality.

Experiment 1: Preference accuracy evaluation

Researchers randomly sampled high-quality documents to build an evaluation dataset that included both human-written and synthetic documents generated by various LLMs, ensuring diversity in structure and style.

For each group of documents with identical text but differing structure and style, experienced Word users familiar with document design ranked them by quality. These rankings were then converted into 473 document-pair comparisons, with each pair annotated to indicate which document was superior.

As shown in Table 1, DocReward achieved significant improvements over strong baselines, including GPT-4o, Claude Sonnet 4, and GPT-5.

DocReward-7B (with 7 billion parameters) achieved an overall human-preference accuracy of 89.22%, outperforming the best proprietary baseline, GPT-5 (69.77%), by 19.45 percentage points. Even in the more challenging synthetic-vs.-synthetic setting, DocReward-7B maintained 78.22% accuracy, compared with GPT-5’s 64.85%.

These results show that DocReward accurately recognizes differences in document structure and style that existing LLMs often overlook.

Experiment 2: Improving document generation with DocReward

To assess DocReward’s impact on real document-generation tasks, the team ran experiments in which AI agents produced multiple candidate documents from the same text. Different reward models were then used to select the best-structured and best-styled version as the final output.

Three reward strategies were compared: random selection, GPT-5 as the reward model, and DocReward as the reward model. Human evaluators assessed each final document for structure and style, recording win/lose/tie ratios.

As shown in Figure 5, random selection performed the worst (24.6% win rate); GPT-5 improved performance to 37.7%; and DocReward achieved a 60.8% win rate and only a 16.9% loss rate, significantly outperforming both baselines.

To visually demonstrate DocReward’s ability to assess structure and style, the team conducted sample analyses using documents with identical text but differing layouts, as shown in Figure 6.

• Sample (a): The document has poor whitespace allocation: last-name field spacing is too small, and first-name field spacing is too large. This results in an unbalanced layout. Additionally, the key fields (Faculty/Department, Country, Country Code) are misaligned, creating visual clutter. The score is 1.21.
• Sample (b): The table-like layout of the document is more organized than that of Sample (a), but the heading font is too small, lacking a clear distinction from the body text and weakening the visual hierarchy. Additionally, the input fields lack borders, making information harder to interpret. The score is 2.11.
• Sample (c): The document features a clear, standardized structure, with a larger heading font, balanced whitespace, a well-aligned layout, and strong readability. The score is 5.34.

These examples show that DocReward accurately distinguishes differences in structural and stylistic quality, with scores consistent with human evaluations. Together, the experiments and sample analyses confirm that DocReward reliably guides agents to produce documents that aligned with human expectations for accuracy and presentation quality, supporting the agentic transformation of core office software like Microsoft Office.

The post DocReward: Advancing professional document design through AI evaluation appeared first on Microsoft Research.

In response, Microsoft Research Asia has developed On-Policy Alignment DPO (OPA-DPO), a new algorithm that aligns expert feedback with the model’s own output distribution before training begins. This “on-policy” alignment slightly alters the model so that expert corrections are close to what the model would naturally produce. As a result, the model is more likely to learn from these expert demonstrations, rather than treating them as outliers to be ignored.

Until now, most attempts to curb hallucinations have involved retraining models with extra data or applying filters to clean up their answers afterwards. While these approaches can help, they’re computationally expensive and don’t address the root issue: how models learn to distinguish accurate from misleading responses.

Direct Preference Optimization (DPO) has recently emerged as a solution. It trains models to favor accurate responses by learning from pairs of good and bad examples. However, when DPO is applied to vision-language tasks, it’s often inadequate because the expert-corrected examples differ too much from what the model would naturally generate, preventing effective learning.

OPA-DPO addresses this by providing a simpler and more data-efficient way to reduce hallucinations while using less training data than previous methods. This work has been recognized with an oral presentation at CVPR 2025.

Limitations of current DPO methods

Previous approaches fall into three categories:

1. Hallucination injection, which inject hallucinated fragments into standard responses. Preference pairs are then constructed by pairing standard responses with their corresponding hallucinated versions.
2. Hallucination recognition, where models generate responses and people or GPT-4/4v identifies and correct hallucinations. Preference pairs are then constructed by pairing corrected responses with their original versions.
3. Self evolution, where models generate multiple responses and a hallucination-recognition model ranks them by severity. Preference pairs are constructed based on these ranking results.

Among these, self-evolution tends to perform best, followed by recognition and then injection. However, all three approaches face limitations. Hallucination injection is weak because the fabricated content does not reflect the model’s own tendencies. Self-evolution is more effective but computationally costly. Recognition, while seemingly the most intuitive, underperforms in practice because expert-edited responses are often too different from the model’s natural outputs. Standard DPO struggles to learn from this “off-policy” data, leading to vanishing gradients and little improvement.

These challenges highlight the need for a method that can incorporate expert corrections while staying aligned with the model’s own output distribution.

OPA-DPO: Breaking convention, reshaping alignment strategy

To address these challenges, OPA-DPO introduces an on-policy alignment step before DPO training. Using 4.8k training samples, OPA-DPO achieves state-of-the-art performance compared with the 16k required by previous state-of-the-art methods. This work was accepted as an oral presentation at CVPR 2025.

OPA-DPO aligns a model’s outputs with expert-preferred responses through a four-step process. First, it generates responses from the model using both the image and prompt. Next, expert feedback—such as that from GPT-4v—is used to finely edit these responses, correcting hallucinations while preserving accurate content.

The edited and ground-truth responses are then used to fine-tune the data-producing model via LoRA-SFT, resulting in what is referred to as the OPA model. Finally, DPO training is performed on the OPA model, incorporating language, image, and anchor preference pairs. Among these stages, the OPA step has the greatest impact on performance. This process is shown in Figure 3.

Researchers compared various DPO-based algorithms fine-tuned on LLaVA-1.5-7B and 13B. With only 4.8k training samples, OPA-DPO achieves state-of-the-art performance on 50% of hallucination metrics for LLaVA-Instruct-1.5-7B. This improves to 70% for LLaVA-Instruct-1.5-13B. OPA-DPO demonstrates particularly strong results on metrics that directly measure hallucination occurrence, such as CHAIR and HalRate. The results are shown in Table 1.

Evaluating OPA-DPO

To validate the importance of OPA and data volume, researchers conducted ablation studies. Even with 600 training samples, OPA-DPO performs better than most baseline algorithms on hallucination-related metrics. As the data volume increases, the performance of OPA-DPO steadily improves. Incorporating the OPA operation leads to a nearly 50% reduction in AMBER HalRate and Object-halCHAIRs.

They also experimented with LLaVA-OneVision as the base model. Despite its detailed but redundant outputs and numerous hallucinations, OPA-DPO significantly improved hallucination metrics with 2.4k training samples, achieving a 43.2% reduction in HalRate and a 38.7% improvement in CHAIR scores compared to baseline models.

OPA-DPO-trained models tend to adopt a conservative strategy, emphasizing salient and verifiable observations while minimizing attention to ambiguous or less relevant details. As illustrated in Figure 5, this approach focuses the description on the actions of the three individuals at the center of the image, while deliberately ignoring peripheral elements such as trees and minor details like backpacks that are speculated by base models. By avoiding speculative or overly detailed content that could introduce hallucinations, the models prioritize clarity and reliability—contributing to their improved performance on hallucination metrics.

Interestingly, base models often assume the query language is accurate, even when it contains hallucinations, leading to responses that reinforce false premises. In contrast, OPA-DPO-trained models demonstrate the ability to detect and reject hallucinated content embedded in the query itself. As shown in Figure 6, this approach can identify fabricated elements—such as the mention of “hands” in the input prompt—and respond with clarifications or corrections rather than perpetuating the hallucination.

OPA-DPO not only improves algorithm performance but also advances multimodal alignment methods. Its approach of generating on-policy data from expert feedback marks a step forward in multimodal alignment training.

The post OPA-DPO: Efficiently minimizing hallucinations in large vision-language models appeared first on Microsoft Research.

The implications extend far beyond accommodation. Only 1.7% (roughly 1,100 of 70,000) of surveyed developers (opens in new tab) are blind or have low vision. Yet the Microsoft Research study shows that AI assistants can unlock new capabilities for this group, sometimes surpassing traditional methods.

“I used to do only non-UI development because my visual impairment made UI tasks difficult,” said one blind developer. “Now, I turn user feedback into prompts for GitHub Copilot to modify code, ask it to check the generated code, and send screenshots for review. I can even review the code myself. This has greatly simplified my workflow.”

The research

The Microsoft Research Asia team recruited 16 developers with varying experience levels and degrees of visual impairment for a comprehensive three-phase study examining real-world use of GitHub Copilot in Visual Studio Code.

In the first phase, participants completed onboarding and coding tasks. Then they used Copilot in their daily work for two weeks while documenting their experience. Final interviews captured long-term feedback on participants’ performance and sentiment.

GitHub Copilot proved ideal for the study because it already incorporates accessibility features: sound cues in addition to visual prompts, text-based views for layout clarity, and multimodal capabilities that convert visual content like screenshots into textual descriptions. The tool’s features are illustrated in Figure 1.

Beyond basic accommodation

“Through real-world use, participants consistently reported that AI programming tools improved efficiency, enhanced coding skills, and lowered the barrier to learning new technologies,” said Luna Qiu, technical program manager at Microsoft Research Asia – Shanghai. “More importantly, these tools used the multimodal capabilities of large models to assist with visual elements, expanding users’ capabilities.”

The study revealed how participants were adapting the new vibe coding approach to overcome traditional limitations. One developer explained: “I like to discuss plans or ask for explanations in Ask mode before letting GitHub Copilot handle my files.” Another noted the power of natural language: “I used natural language to ask GitHub Copilot to undo an operation—and it worked.”

But the benefits went beyond simple task completion. “Accessibility isn’t just about adding labels or shortcuts,” said another developer who is blind. “More types of cues, like sound effects, help me better understand changes. Too many text prompts can actually interfere with my code comprehension.”

For newcomers to programming, the impact was particularly striking. “With a code assistant like GitHub Copilot, getting started with programming is much easier,” one participant noted. “In daily life, we have all kinds of needs, and better programming capabilities help us meet personalized requirements.”

Video 1 shows how screen readers enable users to review code four times faster than they normally would.

Video 2 shows the actual GitHub Copilot interface.

Eight critical improvements

The research team identified specific pain points and solutions across four key areas of AI programming tools.

Managing AI interactions

More consistent shortcuts and clearer feedback: Users often run into conflicting keyboard shortcuts that don’t behave consistently across sessions. Because of this, some resort to clumsy workarounds like copying content to the clipboard and pasting it elsewhere for editing. We recommend creating
 a consistent and predictable shortcut system that minimizes conflicts, reduces extra navigation, and provides timely, accessible session settings.

Guidance on prompts and model choice: AI suggestions are sometimes too brief or based on incorrect assumptions, which requires users to repeatedly refine prompts. As users gain experience, tools should help by detecting vague prompts, asking for clarification, and offering straightforward guidance on selecting suitable AI models for the task.

Reviewing AI responses

Clearer responses: For developers using screen readers, audio cues can be unclear or distracting, and intermixed code changes are difficult to follow. We recommend a system that tracks changes through clear sound cues or text indicators, provides concise text summaries, and groups related information to reduce navigation effort and cognitive load.

Smarter message navigation: Lists of messages can help organize interactions, but navigation is often linear and inefficient. Long responses and input fields that are hard to exit add to the difficulty. We recommend a more navigable format that groups related messages, uses headings or indexes for orientation, minimizes misleading content, and provides reference information to build trust.

Accessible view, optimized: A plain-text accessibility view simplifies navigation but often loses important detail, especially in formatted content like tables. A simplified UI is valuable, but it should still preserve the completeness and integrity of information.

AI response playback: Automatic playback of AI responses can reduce manual effort, but long passages can interrupt thought flow and be hard to digest. We recommend making this “autoplay” optional so that users can choose their preferred interaction style.

Staying focused across views

Improving focus with integrated views: Switching between the editor, chat panel, and terminal can break concentration and increase the risk of errors. In Agent mode, developers must divide attention across multiple views, which makes this even harder. We recommend consolidating key information and actions into a single panel, along with self-verification tools and clear feedback to reduce the need for manual cross-checking.

System status and next steps

Clear status updates: After submitting a request, users need timely updates to understand system status. In Agent mode, vague notifications make it harder to decide on next steps. We recommend providing clear status updates that separate AI-driven actions from those requiring user input, and adding a “Do Not Disturb” setting to minimize unnecessary interruptions.

“AI programming tools are expanding in functionality, but for users of screen readers, more features don’t mean better usability,” said Nan Chen, research SDE at Microsoft Research Asia – Shanghai. “Complex interfaces, convoluted workflows, and unpredictable feedback reduce efficiency. What’s needed is to deliver more value through fewer actions. Striking the right balance between added features and streamlined usability will be a key challenge for future accessibility design.”

Looking ahead to personalized AI programming

As tools evolve from passive adaptation to active customization, personalization is emerging as a new direction for accessible programming. Users of screen readers have diverse preferences: some want minimal text for quick access to information, while others need richer detail to understand code logic and structure.

“With the learning and adaptation capabilities of large models, AI programming tools can tailor interactions to each user’s traits and habits, becoming a truly personalized assistant,” said Luna Qiu.

These new interaction models and workflows expand the potential of human-AI collaboration and highlight opportunities to improve accessibility. Based on these insights, the research team proposed specific recommendations for more accessible programming.

For example, accessibility design should be built in from the start, not added as a post-launch patch. When screen reader use cases are considered early in the process, accessibility is embedded throughout the product.

Regarding developer support, the focus should go beyond documentation that relies heavily on visuals like screenshots or diagrams. Creating learning materials designed specifically for users of screen readers can lower barriers, improve efficiency, and help more people master AI programming tools, helping them participate more fully in the move toward development using AI-assistants.

The post Microsoft study shows AI assistants help with development for programmers who are blind or have low vision appeared first on Microsoft Research.