Vasilis Kontonis, Yuchen Zeng, Shivam Garg, Lingjiao Chen, Hao Tang, Ziyan Wang, Ahmed Awadallah, Eric Horvitz, John Langford, Dimitris Papailiopoulos

We taught models to compress their own chain-of-thought mid-generation. Peak KV cache drops 2–3x, throughput nearly doubles, and the erased reasoning blocks leave traces in the KV cache that the model still uses. Paper, OpenMemento dataset (228K traces), and vLLM fork all open.

If you’re too busy to read this, here’s what we found:

1. You can teach a model to segment its own chain-of-thought into blocks, compress each into a dense memento, and reason forward from that. Standard SFT on ~30K examples suffices to teach this to a model.
2. This cuts peak KV cache by 2–3× and nearly doubles serving throughput, with small accuracy gaps that shrink with scale and close with RL.
3. Erased blocks don’t fully disappear: their information leaks forward through the KV cache representations, forming an implicit second channel without which accuracy drops significantly.
4. We are releasing OpenMementos (228K annotated traces built on top of OpenThoughts-v3), the data generation pipeline, and a vLLM fork with native block masking.

The Problem: LLMs Don’t Know How to Manage their Context

It’s well established at this point that reasoning models can solve hard problems by generating a lot of tokens. Test-time compute works and has led to dramatic advances on competition-level math and coding, but it can also result in a single inference call producing hundreds of thousands of tokens. That is roughly the length of a book. All these tokens stay in memory, attended to at equal cost, whether they lead somewhere or not. The model has no built-in mechanism to compact what it has figured out, keep the conclusions, and move on.

There are ways to manage this externally, e.g., by running a separate summarizer, restart API calls with condensed context, build orchestration logic around the model. However, these are all systems bolted around the model rather than skills the model itself has learned. We think figuring out what to remember and what to forget can and should be a skill that the model learns during training.

Memento teaches language models exactly this. A Memento-trained (aka a mementified) model segments its reasoning into semantically coherent blocks. When a block is complete, the model produces a memento: a terse, information-dense compression of the block’s conclusions, key intermediate values, formulas, and strategic decisions. Think of a memento as a lemma: a minimal record of what future reasoning steps need to continue.

Once a memento is generated, the preceding thinking block is masked from attention and its KV cache entries are flushed away. From that point on, the model sees only past mementos plus whatever block it is currently working through. This means context grows while the model is reasoning through a block, but then it drops sharply once the memento is produced and the block is evicted. This gives rise to a sawtooth pattern where peak memory stays at a fraction of what a standard flat CoT trace would require. Here’s what this looks like:

Importantly, all of this happens within a single generation call, with no restarts, separate summarizers, or orchestration layers involved. The model segments, compresses, and masks its own reasoning by itself.

We applied Memento to five models: Qwen2.5-7B, Qwen3 8B and 32B, Phi-4 Reasoning (14B), and OLMo3-7B-Think. It works across all of them. Peak KV cache drops by 2-3x with small accuracy gaps that shrink with scale and close further with RL.

We also found something we did not anticipate: the erased blocks, although physically removed from the KV cache, don’t fully disappear from the model’s representations. More on this in a minute!

But before that, how do we train context management in a model?

How do you teach context management? Add it in the training data!

Teaching this behavior requires training data that isn’t quite common: large-scale, high-quality reasoning traces segmented into blocks, each paired with a memento that captures the block’s conclusions in a way the model can reason forward from. The intuition is straightforward: if you take reasoning traces, segment them, add proper summaries, and SFT on the result, maybe the model learns to do context management on its own.

It sounds simple, but as with many things there were several components that broke along the way and had to be fixed.

First, we decided to build on top of OpenThoughts (opens in new tab): reasoning traces generated by QwQ-32B that are already reasonably high-quality and widely used by the community, which saves us from generating everything from scratch. Now the question is: how do we go from raw traces to segmented, annotated ones with mementos at each block boundary? The challenge is that reasoning traces have no natural segment boundaries, i.e., ideas mix together, calculations span multiple sentences, and where to “cut” the CoT depends much more heavily on meaning rather than formatting, or some other obvious indicator.

We tried the obvious thing first: paste a trace into a frontier model and ask it to segment and summarize directly. This does not work! Not even if you cut the trace into pieces first, because you don’t quite know where to cut. Finding good partitions requires simultaneously reasoning about block coherence, size balance, and semantic boundaries, which is a tricky combinatorial optimization that LLMs (at least the ones we tried) struggle to do in one shot.

So we factored the problem into parts. First, we segment each trace into atomic units—sentences, code blocks, math equations—that can’t be meaningfully split further. Then an LLM scores each inter-sentence boundary from 0 (mid-thought, would break flow, i.e., bad) to 3 (major transition, natural stopping point, i.e., good). This is a local question and LLMs handle local questions very well. The global optimization of where to actually place boundaries given these scores is then handled by dynamic programming, which maximizes boundary quality while penalizing uneven block sizes. This is the kind of thing that’s (again, in our experience) hard for an LLM to zero-shot, but where good old dynamic programming just works.

Once we have our segmented traces, we now need to compress each block. A compressor LLM produces a memento for each one, and we explicitly explain in the prompt that the task is not vanilla summarization but state compression: produce something compact enough that the model could continue reasoning from the memento alone, without ever seeing the original block. And so, a memento is born!

Then, a separate judge LLM evaluates each memento across six dimensions (formulas extracted, values preserved, methods named, validation included, no hallucinations, result-first structure) and if the score falls short, the judge provides specific, actionable feedback (not “more details needed” but “missing formula: K² − 3K + 3” etc) and the compressor retries.

This iterative refinement turned out to be crucial. Single-pass compression barely hits a 28% pass rate on our rubric, because initial mementos typically miss exact formulas or intermediate values that downstream blocks depend on. Two rounds of judge feedback bring the pass rate to 92%.

Note: For all LLM calls in the pipeline we used GPT-5.x, but any sufficiently capable model should work. The full pipeline is open and we hope people use it, improve it, and build better datasets than ours.

Here is what the data gen pipeline looks like:

The final dataset, OpenMementos, contains 228K annotated traces consisting of 54% math, 19% code, 27% science problems. We measured that mementos resulted in roughly 6x trace-level compression: about 11k tokens of reasoning compacted to under 2k tokens of mementos per trace.

Here are some cool compression statistics on OpenMementos.

Training: How do we put pressure on the model?

We have annotated traces with block structure and mementos. The obvious next step is: let’s SFT on them; but how exactly? The goal is for the model to eventually reason forward from mementos alone, with the original blocks masked and their KV cache entries removed.

One option is to train with normal causal attention on the annotated traces and then just mask the blocks at inference time. This works to some extent, but it means training never puts any pressure on the model to actually pack information into its mementos, as it can always fall back on attending to the full block during training, and then at inference it’s basically on its own.

We want training to match inference: if blocks will be masked when the model is deployed, they should be masked during training too.

But training directly with block masking from the start also does not work well. The model is trying to learn three things at the same time: the block-memento format, how to compress under hard constraints, how to only rely on mementos for the next block generation. And it struggles with all three simultaneously.

What we found is that curriculum matters a lot.

Stage 1 uses standard causal attention with loss on all tokens. The model learns the format: when to end a block, how to write a memento, what structure looks like. It can still see everything, so there’s no compression pressure yet. Stage 2 then introduces the hard constraint: after each memento, the preceding thinking block is fully masked from subsequent attention. Now the model has to produce mementos that carry everything future reasoning needs, because the original blocks are gone. This is where the real learning seems to occur: the model is forced to pack more information into its mementos, almost like an RL-style pressure signal that pushes toward self-contained compression. Very cool!

Multi-stage SFT ablation on AIME 2024, AIME 2025, and GPQA-Diamond (Pass@1, n=8, Qwen2.5-7B). OT = OpenThoughts only; OM/Full = OPENMEMENTOS Full Attention; OM/Mem = OPENMEMENTOS Memento Attention; 2-Stg = OT → OM/Full; 3-Stg = OT → OM/Full → OM/Mem (Ours). Training directly on OPENMEMENTOS from the base model (OM variants) substantially underperforms vanilla SFT (OT). Our three-stage pipeline enables block masking while retaining strong performance.

We also found, consistent with other work on teaching skills, that training on a small subset of the OpenMementos data is enough. Even ~30K samples from the 228K pool, trained for 5 epochs per stage at 32K sequence length, were sufficient for models to pick up the skill.

For models that already reason well (Qwen3, OLMo3, Phi-4-reasoning), two stages suffice; non-reasoning base models like Qwen 2.5 7B need a preliminary round of standard reasoning SFT first. Memento doesn’t require qualitatively more data than standard reasoning SFT, it just requires different kind of data.

Training data scaling. Pass@1 accuracy on AIME24 and AIME25 for Qwen2.5-7B-Instruct fine-tuned on 1K–100K examples. All methods improve monotonically with data size.

How does compaction affect accuracy?

The first obvious concern with Memento is that attending to fewer tokens should hurt accuracy. And when we first looked at the numbers, there was indeed a drop. Where does it come from?

Our initial reaction was that it must be due to compaction and sparsity: the model is seeing far less context, so of course it gets worse. But then we ran control studies, and the picture turned out to be more interesting.

The key insight is that we train on OpenThoughts traces generated by QwQ-32B, which is a different and often weaker model than the ones we are fine-tuning. Several of our target models were released after QwQ and are arguably stronger. So we ran a control: take each base model, SFT it on the same raw OpenThoughts traces (no block structure, no mementos), and measure the accuracy drop from that alone. It turns out that just doing SFT on another model’s reasoning traces already costs you something. When we compare Memento against that control rather than the untouched baseline, the additional drop from compression is small, and in some cases negligible.

But we were still curious about whatever accuracy gap remained. So we asked: can the model still solve the same problems?

To test that, we generated 64 completions per problem across all three model families on AIME 2024/25/26, and the answer is overwhelmingly yes. The overlap between the problems solved by the base model and by Memento averages 96.4%, hitting 100% in some settings. The model retains the capability to solve these problems and what drops is the consistency of solving them on any single attempt.

This is an important distinction because it likely implies the gap is closable. For example we found that even majority voting at k=3 is enough for the Memento model to match not just the control but the original baseline. This confirms that the capability is still there in the distribution.

The natural next step was RL. And unsurprisingly it works: fine-tuning the Qwen3-8B Memento checkpoint with CISPO recovers AIME’26 and GPQA-Diamond scores (sometimes actually exceeding the vanilla baseline), while the KV savings remain substantial after RL.

Scale also helps independently, even without RL. Going from Qwen3-8B to 32B, the gap shrinks considerably even though both models are trained on the same QwQ-32B traces: the larger model handles the distribution mismatch and the effects of compression more gracefully.

So the bottom line is: compression preserves capability, any consistency loss traces primarily to training data mismatch rather than a fundamental limitation, and both RL and scale close the gap further.

The Dual Information Stream

Early in the project, there were a lot of discussions about how inference should actually work. The simplest approach, and the one that would make our lives much easier, is restarts: every time a memento is produced, kill the KV cache and start a fresh API call with just the accumulated memento text. No need to implement non-causal sparse attention inside vLLM, which turned out to be a huge pain. Just restart the call.

But we kept coming back to a concern: when a memento is generated, the model can still see the full thinking block and therefore memento tokens attend to block tokens during their own generation. The block is only masked after the memento is complete. This means the KV cache entries for the memento were computed as a function of the block’s content. So even after the block text is gone, something from it survives in the memento’s KV representations. If the next block attends to the memento, it’s attending in an indirect way to this implicit, soft representation of what came before. In a restart setup, you throw all of that away.

That is to say, there is non-trivial information about masked blocks that survives in the KV cache representations, beyond what the actual memento tokens capture. So a question kept bothering us: does this implicit information channel actually matter for accuracy, or are we overthinking it?

And so we ran an ablation. Take the same Qwen3-8B checkpoint, compare normal Memento inference (mask blocks but keep the memento KV states intact) against restart mode (recompute the entire KV cache from scratch at each memento boundary, so the mementos themselves never attended to their blocks). The restart mode drops AIME’24 from 66.1% to 50.8%. Fifteen percentage points which to any reasonable observer does not register as noise. It in fact strongly suggests that the side information channel flowing through the KV representations matters a great deal. Just to be sure, we wanted to test this hypothesis further.

So we designed a simple experiment: take a model, inject a random 5-digit passcode into a target block, mask that block, and train linear probes on the KV states of downstream mementos that never directly attended to the masked block. Can you recover a piece of information that exists only in the implicit KV channel, not in any memento text?

Oh, yes, yes you can! The probes reconstruct the passcode well above chance, precisely because of information leakage that happens through the KV states. This leakage concentrates in deeper layers, decays with distance from the target block, but remains detectable even seven blocks away, and scales with model capacity.

We also verified this on a small controlled toy transformer (4 layers, 810K parameters), where the leakage is constant across training checkpoints even as task accuracy improves from 77% to 95%. This is an architectural consequence of residual connections, causal attention, and in-place masking.

We believe this distinguishes Memento from approaches like InftyThink and Accordion-Thinking, which discard original tokens and rebuild context from summary text alone and lose this implicit channel entirely. And it is what convinced us to do the hard infrastructure work of implementing proper block masking inside vLLM rather than taking the infra-wise easier restart path.

Making Memento work in vLLM

Memento’s block masking is data-dependent and keeps changing during generation, since which tokens to mask depends on what the model produces. No production inference framework supported this out of the box, unfortunately. We started with a HuggingFace backend, which was enough to validate that block masking and keeping everything in a single inference call actually helps, but once we were convinced, it was clear we needed to build this properly inside vLLM.

That turned out to be painful but, in the end, doable. The key design choice was physical KV cache compaction rather than logical masking: when a block completes, its KV entries are physically flushed and the freed slots are returned to the KV pool. This means standard FlashAttention and paged-attention kernels work completely unmodified as they never see the evicted tokens. The implementation operates purely at the vLLM Python level and can be installed as a patch on top of an existing vLLM installation.

On a single B200 GPU with 240 concurrent requests (Qwen3-8B, 32K max tokens), Memento sustains 4,290 tok/s versus 2,447 for vanilla (1.75× throughput) and completes the batch in 693s versus 1,096s. The gains come from freeing KV entries as blocks complete, allowing the engine to sustain higher batch sizes in regimes where vanilla vLLM becomes KV-cache-bound.

This infrastructure also turned out to be essential for RL: generating 32K-token training rollouts requires block masking during generation, with each rollout producing and compacting blocks on the fly. Without the vLLM fork, RL at this scale would not have been feasible.

What’s next?

Two things seem natural from here. First, scaling the RL recipe: our results with Qwen3-8B are early, and the pass@64 analysis makes it clear there is a lot of headroom for improvement. Larger models with more RL compute should take us to interesting places.

Second, and more importantly to us: agents. Memento was built for mathematical, coding, and science reasoning as a test case, not because we think single-turn math and coding are the most interesting applications. The block-and-compress pattern maps onto any setting where a model accumulates a long trajectory of intermediate state and limited context windows become the bottleneck. Terminal and CLI agents are naturally multi-turn, where each action-observation cycle is laid out as a natural block, and the ability to selectively remember and forget is exactly what seems missing (at least from OSS models/agents). Recent work on context compaction in agentic settings (e.g., from Anthropic and OpenAI) points in the same direction, and we think there is a ton of room to explore here.

Coda

Memento started as an attempt to teach models to compact their own reasoning. That indeed works: 2-3x KV reduction, accuracy largely preserved while throughput nearly doubled. But we came away from this project with two insights that feel more important than the efficiency gains.

The first is that context management can be taught through standard training on the right data. A model that had no concept of blocks or summaries can, after SFT on ~30K examples, learn to segment its own reasoning, compress each segment, and continue from the compressed version. This is a non-trivial, non-causal skill involving sparse attention, selective forgetting, state compression, that was acquired through entirely conventional training. We think there in fact is a much wider space of unconventional capabilities that can be taught this way.

The second is the dual information stream supported by both hard tokens and their KV representations. When you mask a block inside a single forward pass, the block’s information doesn’t quite vanish: it persists in the KV representations of the mementos that were computed while the block was still visible. This is both useful and architecturally unavoidable, and we don’t yet know how far this implicit channel can be pushed, especially with RL.

These two pieces point in the same direction: memory management should be a learned capability, and models can learn with less effort than we expected.

We think Memento is a first step, and there is a long way to go, with better training data, stronger RL, and agent applications. We are continuing work across all of these, and along the way we are releasing OpenMementos (228K annotated reasoning traces), our full data generation pipeline, and the vLLM fork with native block masking.

In the meantime, stop flushing your KV cache. Your model remembers more than you think.

Paper (opens in new tab) · Code (opens in new tab) · Dataset (opens in new tab)

The post Memento: Teaching LLMs to Manage Their Own Context appeared first on Microsoft Research.

Despite their smaller size (14B parameters), Phi-4-reasoning and Phi-4-reasoning-plus are competitive with or exceeding much larger open weight (QwQ-32B, DeepSeek R1- Distill-Llama-70B, DeepSeek-R1) and closed (o1-mini, Claude Sonnet 3.7) reasoning models across several benchmarks as shown in Figures 1, 3 and Tables 1, 2. Our extensive benchmarks span math and scientific reasoning, coding, algorithmic problem solving, planning, and spatial understanding. Interestingly, we observe a non-trivial transfer of improvements to general-purpose benchmarks as well. 

Figure 1. Performance comparison on representative reasoning benchmarks spanning mathematics (HMMT, AIME 25, OmniMath), scientific (GPQA), and coding (LiveCodeBench 8/24-1/25) domains. 

Notably, Phi-4-reasoning and Phi-4-reasoning-plus achieve better performance than o1-mini, and DeepSeek-R1-Distill-Llama-70B at most benchmarks and achieve performance comparable to the full DeepSeek-R1 model (with 671B parameters) on AIME 2025¹ (the 2025 qualifier for the USA Math Olympiad). They also outperform Claude 3.7 Sonnet and Gemini 2 Flash Thinking on all tasks except GPQA (PhD-level STEM questions) and Calendar Planning.

More Potential with Parallel Test-time Scaling: As shown in Figure 2, our small-ish model nearly saturates performance on AIME 2025 with increasing parallel test-time compute (e.g., Majority @N), surpassing the pass@1 of the teacher (o3-mini). 

Figure 2: Effects of parallel test-time compute on AIME 2025

Table 1. Average Pass@1 accuracy on selected reasoning benchmarks. Bold denotes best model per benchmark and model class (open versus closed-weight), and underline denotes the second best. We report the standard deviation in parentheses where available. 

Key contributors to best-in-class performance 

Below we summarize the core contributions that led to the superior performance of Phi-4-reasoning models. We provide more comprehensive technical details and experimentations surrounding each bullet point in our tech repot [1]. 

• Careful Data Curation: our reasoning prompts are specifically filtered to cover a range of difficulty levels and to lie at the boundary of the base model capabilities. Our approach aligns closely with data-centric methods of earlier Phi and Orca models [2,3,4,5,6,7,8], demonstrating that meticulous data curation and high-quality synthetic datasets allow smaller models to compete with larger counterparts. The datasets used in supervised finetuning include topics in STEM (science, technology, engineering, and mathematics), coding, and safety-focused tasks. Our reinforcement learning is conducted on a small set of high-quality math-focused problems with verifiable solutions. 

• Benefits of Supervised Finetuning (SFT): Phi-4-reasoning after the SFT stage already performs strongly across diverse benchmarks. Interestingly, the improvement in performance generalizes tasks not directly targeted in the training data—such as calendar planning and general-purpose benchmarks (Table 2). We highlight the critical role of data mixture and training recipe in unlocking reasoning capabilities during the SFT stage, which goes hand-in-hand with our data selection and filtering.  

• Boost with Reinforcement Learning: we are encouraged by the gains achieved through a short round of outcome-based reinforcement learning (RL) and the potential of combining distillation/SFT and reinforcement learning. We observe that the model after RL provides higher accuracy on math while using approximately 1.5x more tokens than the SFT model on average, offering a trade-off between accuracy and inference-time compute. 

Reasoning is a meta skill 

We think that reasoning is a transferable meta-skill that can be learned through supervised finetuning alone and further enhanced with reinforcement learning. To test the generalization of the models’ reasoning capabilities, we evaluate them on multiple new reasoning benchmarks that require algorithmic problem solving and planning, including 3SAT (3-literal Satisfiability Problem), TSP (Traveling Salesman Problem), and BA-Calendar planning. These reasoning tasks are nominally out-of-domain for the models as the training process did not target these skills, but the models show strong generalization to these tasks as shown in Figure 2. 

Table 2. Average pass@1 accuracy on general-purpose benchmarks, averaged across five runs. 

This generalized improvement in capabilities also goes beyond reasoning. Without explicit training on non-reasoning tasks, we saw significant improvements on IFEval, FlenQA, and internal PhiBench as shown in Table 2. And despite limited coding data during the SFT stage (and none during RL), the model performs well, scoring at o1-mini level on LiveCodeBench (LCB) and Codeforces as shown in Table 1. We plan to emphasize coding further in our future versions.

Figure 3. Average Pass@1 performance on reasoning benchmarks, averaged across five runs. Except for GPQA, other benchmarks are out-of-distribution with respect to Phi-4-reasoning’s training data. 

Lessons on Evaluating Reasoning Models 

Language models exhibit large generation nondeterminism, i.e., they may produce substantially different answers given the same prompts and inference hyperparameters (e.g., temperature). To account for this stochastic nature, we study the accuracy distribution on AIME 2025, approximated by kernel density estimation of 50 independent runs with the same prompt and temperature. We have found several interesting observations as illustrated in Figure 4: 

1. All models show a high accuracy variance. For example, accuracy of answers generated by DeepSeek-R1- Distill-Llama-70B ranges from 30% to 70%, while o3-mini’s accuracy ranges from 70% to 100%. This suggests that any comparison among models using a single run can easily produce misleading conclusions.  

2. Models on the two extremes of average accuracy demonstrate more robust accuracy. For example, Phi-4-reasoning-plus and Phi-4 have relatively narrower accuracy ranges compared to DeepSeek-R1-Distill-Llama-70B and Phi-4-reasoning.  

3. The accuracy distribution further indicates the competitive performance of Phi-4-reasoning-plus, largely intersecting with o3-mini’s distribution and being almost disjoint from DeepSeek-R1-Distill-Llama-70B’s distribution.  

Figure 4. Distribution of pass@1 accuracy on AIME 2025, approximated by kernel density estimation over 50 runs with the same prompt and temperature. The accuracy distribution further shows the competitive performance of Phi-4-reasoning-plus, largely intersecting with o3-mini’s distribution and being almost disjoint from the DeepSeek-R1-Distill-Llama-70B.  

Phi-4-Reasoning in action

Below we provide some interesting example responses from Phi-4-reasoning that showcases its intelligent behavior. 

Prompt: “Generate a website for steves pc repairs using a single html script”

Prompt: “write a Python program that shows a ball bouncing inside a spinning triangle. The ball must bounce off the rotating walls realistically and should not leave the triangle”

References

[1] “Phi-4-reasoning Technical Report.” arXiv preprint arXiv:2504.21318 (2025). [link (opens in new tab)] 

[2] “Phi-4 technical report.” arXiv preprint arXiv:2412.08905 (2024). 

[3] “Phi-3 technical report: A highly capable language model locally on your phone.” arXiv preprint arXiv:2404.14219 (2024).  

[4] “Phi-2: The surprising power of small language models.” Microsoft Research Blog (2023). 

[5] “Textbooks are all you need.” arXiv preprint arXiv:2306.11644 (2023). 

[6] “Agentinstruct: Toward generative teaching with agentic flows.” arXiv preprint arXiv:2407.03502 (2024).  

[7] “Orca 2: Teaching small language models how to reason.” arXiv preprint arXiv:2311.11045 (2023).   

[8] “Orca: Progressive learning from complex explanation traces of gpt-4.” arXiv preprint arXiv:2306.02707 (2023). 

The post Phi-Reasoning: Once again redefining what is possible with small and efficient AI  appeared first on Microsoft Research.

Do reasoning capabilities of large reasoning models extend to complex reasoning skills beyond math? What is their advantage when compared to conventional, autoregressive models? What is left to harvest in the reasoning space and how far can we go from here? Do longer and extended CoT scratchpads always translate to higher accuracy? This blog summarizes answers to these questions by using insights from the recent Eureka report on inference-time scaling: “Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Ahead (opens in new tab)”.

For extracting these insights, the study uses experiments on eight diverse complex reasoning tasks on nine state-of-the-art models at the frontier of Artificial Intelligence today. The tasks include:

• Math reasoning (Benchmarks: AIME 2025, AIME 1983-2024, OmniMATH)  
• Science reasoning (Benchmarks: GPQA)
• Planning and scheduling (Benchmarks: BA Calendar)
• NP-hard algorithmic reasoning (Benchmarks: TSP for traveling salesman minimal paths and 3SAT on 3-literal satisfiability)
• Spatial understanding (Benchmarks: Spatial Understanding and Maze)

All these tasks were used to test conventional models like: Claude 3.5 Sonnet, Gemini 2.0 Pro, GPT-4o, and Llama 3.1 405B, as well as reasoning models: Claude 3.7 Sonnet, DeepSeek R1, Gemini 2.0 Flash Thinking, O1, and O3-mini.

To estimate the future potential of all models we ran all experiments several times following two different scaling approaches. In the parallel approach, we make N independent calls to the model and aggregate the results via different aggregators: average, majority vote, best of N, worst of N. In the sequential approach, the model is set to sequentially attempt to solve the problem and if it is incorrect, it receives feedback from another model inference call until the context budget is exhausted or N trials are done.

All experiment implementations and data are available on Eureka ML Insights (opens in new tab), which is an open-source framework for standardizing evaluations of large foundation models, and for extracting insights beyond single-score reporting and rankings.

In math benchmarks, reasoning models surpass their conventional counterparts often by more than 50 percentage points in accuracy. It is also interesting to see major improvements in algorithmic problems such as NP-hard problems like Satisfiability (3SAT) and Traveling Salesman Path Optimization (TSP), as well as calendar planning. Improvements in spatial understanding (Maze and SpatialMap) and scientific reasoning however are less pervasive across model families but still often over 20 percentage points.

As shown in Figure 2, an in-depth analysis on the GPQA benchmark for scientific problems, reveals that while reasoning models all achieve an accuracy of higher than 90% for Physics, they still lag behind on Biology and Chemistry. In algorithmic problems and other problems that have a notion of difficulty, model accuracy drops even for the best models as difficulty increases and the length of reasoning traces saturates.

There is high variability in token use, even across models with similar accuracies on a task. For example, in Figure 3, we can observe that often there exist pairs of models that have similar accuracy but one of them uses a lot more tokens (e.g. for AIME 25, DeepSeek-R1 and Claude 3.7 Sonnet Thinking have an average accuracy across five repeats within a < 3% range, but Claude 3.7 Sonnet Thinking uses at least 2.5 times more tokens).

Figure 4 illustrates the average accuracy over generation lengths for the DeepSeek R1 model and O3-mini high on the GPQA task.

Horizontal whiskers in Figure 3 are a measure of cost nondeterminism as they show the variability within a single prompt (data instance). In Table 1, we summarize these charts and show the actual cost in dollars on average for 1000 prompts, with today’s prices per provider. This shows that the variability on token length can translate to up to 40% variability in actual cost for almost all reasoning models.

To conduct this analysis, we run all our experiments 5 times and see whether a correct inference path exists by checking with a “perfect” verifier which has access to ground truth. See examples of results in Figure 6. The existence of the inference path shows that it is possible to extract that skill or knowledge from the model with better fine tuning and RL techniques. This emphasizes the importance of building improved and generalizable verifiers that can be used for further development. In fact, investments in better verifiers for different domains can become the distinguishing factor in the current AI space that determine the speed of progress in generalizing reasoning for a broad number of use cases.

Figure 7 shows results on experiments that simulate sequential iterations on O1 and GPT-4o, where the model first attempts a solution and then receives feedback from another judge (of the same type) to make another attempt on the solution, if the previous one was incorrect, until the context length is depleted. Here, O1 improves much faster than GPT-4o, and its improvements are even faster with sequential feedback.

The post Eureka Inference-Time Scaling Insights: Where We Stand and What Lies Ahead appeared first on Microsoft Research.

“When we released AutoGen, one of the first things that the developers absolutely loved about it was its simplicity and the many pre-built agents and teams that it provided, such as the user proxy agent and the assistant agent, and the group chat between multiple agents. With the AutoGen AgentChat layer, we are maintaining these features and adding tons of more essential features such as streaming support, serialization, state management and memory for agents, and finally full-time support for a better development experience.”

– Gagan Bansal, Senior Researcher, Microsoft Research AI Frontiers

Microsoft research copilot experience How has the new update to the AutoGen framework enhanced its capabilities for agentic AI applications? 

Related resources

The post AutoGen v0.4: Reimagining the foundation of agentic AI for scale, extensibility, and robustness appeared first on Microsoft Research.

“That ability to condition on generation, rather than evaluate the generation, ends up being amazingly useful in terms of giving you a more honest valuation of the generated text.”

– John Langford, Partner Research Manager, Microsoft Research AI Frontiers

Microsoft research copilot experience What advantages do belief state transformers offer over traditional GPT-style models in planning and decision-making tasks? 

Related resources

The post Belief state transformers appeared first on Microsoft Research.

Graphic User interface (GUI) automation requires agents with the ability to understand and interact with user screens. However, using general purpose LLM models to serve as GUI agents faces several challenges: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associating the intended action with the corresponding region on the screen. OmniParser closes this gap by ‘tokenizing’ UI screenshots from pixel spaces into structured elements in the screenshot that are interpretable by LLMs. This enables the LLMs to do retrieval based next action prediction given a set of parsed interactable elements.

OmniParser V2 takes this capability to the next level. Compared to its predecessor, it achieves higher accuracy in detecting smaller interactable elements and faster inference, making it a useful tool for GUI automation. In particular, OmniParser V2 is trained with a larger set of interactive element detection data and icon functional caption data. By decreasing the image size of the icon caption model, OmniParser V2 reduces the latency by 60% compared to the previous version. Notably, Omniparser+GPT-4o achieves state-of-the-art average accuracy of 39.6 on a recently released grounding benchmark ScreenSpot Pro (opens in new tab), which features high resolution screen and tiny target icons. This is a substantially improvement on GPT-4o’s original score of 0.8.

To enable faster experimentation with different agent settings, we created OmniTool, a dockerized Windows system that incorporates a suite of essential tools for agents. Out of the box, we enable OmniParser to be used with a variety of state-of-the-art LLMs: OpenAI (4o/o1/o3-mini), DeepSeek (R1), Qwen (2.5VL) and Anthropic (Sonnet) combining the screen understanding, grounding, action planning and execution steps.

Risks and Mitigations

To align with the Microsoft AI principles and Responsible AI practices, we conduct risk mitigation by training the icon caption model with Responsible AI data, which helps the model avoid inferring sensitive attributes (e.g.race, religion etc.) of the individuals which happen to be in icon images as much as possible. At the same time, we encourage user to apply OmniParser only for screenshot that does not contain harmful content. For the OmniTool, we conduct threat model analysis using Microsoft Threat Modeling Tool overview – Azure | Microsoft Learn (opens in new tab). We provide a sandbox docker container, safety guidance and examples in our GitHub Repository. And we advise a human to stay in the loop in order to minimize the risk.

The post OmniParser V2: Turning Any LLM into a Computer Use Agent appeared first on Microsoft Research.

Contributors: Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang (Eric) Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, Saleema Amershi

The future of AI is agentic. AI systems are evolving from having conversations to getting things done—this is where we expect much of AI’s value to shine. It’s the difference between generative AI recommending dinner options to agentic assistants that can autonomously place your order and arrange delivery. It’s the shift from summarizing research papers to actively searching for and organizing relevant studies in a comprehensive literature review.

Modern AI agents, capable of perceiving, reasoning, and acting on our behalf, are demonstrating remarkable performance in areas such as software engineering, data analysis, scientific research, and web navigation. Still, to fully realize the long-held vision of agentic systems that can enhance our productivity and transform our lives, we need advances in generalist agentic systems. These systems must reliably complete complex, multi-step tasks across a wide range of scenarios people encounter in their daily lives.

Introducing Magentic-One (opens in new tab), a high-performing generalist agentic system designed to solve such tasks. Magentic-One employs a multi-agent architecture where a lead agent, the Orchestrator, directs four other agents to solve tasks. The Orchestrator plans, tracks progress, and re-plans to recover from errors, while directing specialized agents to perform tasks like operating a web browser, navigating local files, or writing and executing Python code.

Magentic-One achieves statistically competitive performance to the state-of-the-art on multiple challenging agentic benchmarks, without requiring modifications to its core capabilities or architecture. Built on AutoGen (opens in new tab), our popular open-source multi-agent framework, Magentic-One’s modular, multi-agent design offers numerous advantages over monolithic single-agent systems. By encapsulating distinct skills in separate agents, it simplifies development and reuse, similar to object-oriented programming. Magentic-One’s plug-and-play design further supports easy adaptation and extensibility by enabling agents to be added or removed without needing to rework the entire system—unlike single-agent systems, which often struggle with inflexible workflows.

We’re making Magentic-One open-source (opens in new tab) for researchers and developers. While Magentic-One shows strong generalist capabilities, it’s still far from human-level performance and can make mistakes. Moreover, as agentic systems grow more powerful, their risks—like taking undesirable actions or enabling malicious use-cases—can also increase. While we’re still in the early days of modern agentic AI, we’re inviting the community to help tackle these open challenges and ensure our future agentic systems are both helpful and safe. To this end, we’re also releasing AutoGenBench (opens in new tab), an agentic evaluation tool with built-in controls for repetition and isolation to rigorously test agentic benchmarks and tasks while minimizing undesirable side-effects.

How it works

Magentic-One work is based on a multi-agent architecture where a lead Orchestrator agent is responsible for high-level planning, directing other agents and tracking task progress. The Orchestrator begins by creating a plan to tackle the task, gathering needed facts and educated guesses in a Task Ledger that is maintained. At each step of its plan, the Orchestrator creates a Progress Ledger where it self-reflects on task progress and checks whether the task is completed. If the task is not yet completed, it assigns one of Magentic-One other agents a subtask to complete. After the assigned agent completes its subtask, the Orchestrator updates the Progress Ledger and continues in this way until the task is complete. If the Orchestrator finds that progress is not being made for enough steps, it can update the Task Ledger and create a new plan. This is illustrated in the figure above; the Orchestrator work is thus divided into an outer loop where it updates the Task Ledger and an inner loop to update the Progress Ledger.

Magentic-One consists of the following agents:

• Orchestrator: The lead agent responsible for task decomposition, planning, directing other agents in executing subtasks, tracking overall progress, and taking corrective actions as needed
• WebSurfer: An LLM-based agent proficient in commanding and managing the state of a Chromium-based web browser. For each request, the WebSurfer performs actions such as navigation (e.g., visiting URLs, performing searches), interacting with webpages (e.g., clicking, typing), and reading actions (e.g., summarizing, answering questions). It then reports on the new state of the webpage. The WebSurfer relies on the browser’s accessibility tree and set-of-marks prompting to perform its tasks.
• FileSurfer: An LLM-based agent that commands a markdown-based file preview application to read local files. It can also perform common navigation tasks such as listing directory contents and navigating through them.
• Coder: An LLM-based agent specialized in writing code, analyzing information collected from the other agents, and creating new artifacts.
• ComputerTerminal: Provides access to a console shell for executing programs and installing new libraries.

Together, Magentic-One’s agents equip the Orchestrator with the tools and capabilities it needs to solve a wide range of open-ended problems and autonomously adapt to, and act in, dynamic and ever-changing web and file-system environments.

While the default multimodal LLM used for all agents is GPT-4o, Magentic-One is model-agnostic, allowing the integration of heterogeneous models to support different capabilities or meet different cost requirements. For example, different LLMs and SLMs or specialized versions can power different agents. For the Orchestrator, we recommend a strong reasoning model, like GPT-4o. In a different configuration, we also experimented with using OpenAI o1-preview for the Orchestrator’s outer loop and for the Coder, while other agents continued to use GPT-4o.

Evaluation

To rigorously evaluate Magentic-One’s performance, we introduce AutoGenBench, an open-source standalone tool for running agentic benchmarks that allows repetition and isolation, e.g., to control for variance of stochastic LLM calls and side-effects of agents taking actions in the world. AutoGenBench facilitates agentic evaluation and allows adding new benchmarks. Using AutoGenBench, we can evaluate Magentic-One on a variety of benchmarks. Our criterion for selecting benchmarks is that they should involve complex multi-step tasks, with at least some steps requiring planning and tool use, including using web browsers to act on real or simulated webpages. We consider three benchmarks in this work that satisfy this criterion: GAIA, AssistantBench, and WebArena.

In the Figure below we show the performance of Magentic-One on the three benchmarks and compare with GPT-4 operating on its own and the per-benchmark highest-performing open-source baseline and non open-source benchmark specific baseline according to the public leaderboards as of October 21, 2024. Magentic-One (GPT-4o, o1) achieves statistically comparable performance to previous SOTA methods on both GAIA and AssistantBench and competitive performance on WebArena. Note that GAIA and AssistantBench have a hidden test set while WebArena does not, and thus WebArena results are self-reported. Together, these results establish Magentic-One as a strong generalist agentic system for completing complex tasks.

Risks and mitigations

Agentic systems like Magentic-One mark a significant shift in both the opportunities and risks associated with AI. Magentic-One interacts with a digital world designed for humans, taking actions that can change states and potentially lead to irreversible consequences. These inherent and undeniable risks were evident during our testing, where several emerging issues surfaced. For example, during development, a misconfiguration led agents to repeatedly attempt and fail to log into a WebArena website. This resulted in the account being temporarily suspended. The agents then tried to reset the account’s password. Even more concerning were cases in which agents, until explicitly stopped, attempted to recruit human assistance by posting on social media, emailing textbook authors, or even drafting a freedom of information request to a government entity. In each case, the agents were unsuccessful due to a lack of the required tools or accounts, or because human observers intervened.

Aligned with the Microsoft AI principles and Responsible AI practices, we worked to identify, measure, and mitigate potential risks before deploying Magentic-One. Specifically, we conducted red-teaming exercises to assess risks related to harmful content, jailbreaks, and prompt injection attacks, finding no increased risk from our design. Additionally, we provide cautionary notices and guidance for using Magentic-One safely, including examples and appropriate default settings. Users are advised to keep humans in the loop for monitoring, and ensure that all code execution examples, evaluations, and benchmarking tools are run in sandboxed Docker containers to minimize risks.

Recommendations and looking forward

We recommend using Magentic-One with models that have strong alignment, pre- and post-generation filtering, and closely monitored logs during and after execution. In our own use, we follow the principles of least privilege and maximum oversight. Minimizing risks associated with agentic AI will require new ideas and extensive research, as much work is still needed to understand these emerging risks and develop effective mitigations. We are committed to sharing our learnings with the community and evolving Magentic-One in line with the latest safety research.

As we look ahead, there are valuable opportunities to improve agentic AI, particularly in safety and Responsible AI research. Agents acting on the public web may be vulnerable to phishing, social engineering, and misinformation threats, much like human users. To counter these risks, an important direction is to equip agents with the ability to assess the reversibility of their actions—distinguishing between those that are easily reversible, those that require effort, and those that are irreversible. Actions like deleting files, sending emails, or filing forms are often difficult or impossible to undo. Systems should therefore be designed to pause and seek human input before proceeding with such high-risk actions.

We invite the community to collaborate with us in ensuring that future agentic systems are both helpful and safe.

For further information, results and discussion, please see our technical report. (opens in new tab)

Azure AI Foundry Labs

Get a glimpse of potential future directions for AI, with these experimental technologies from Microsoft Research.

Explore Azure AI Foundry

The post Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks appeared first on Microsoft Research.

Recent advancements in large vision-language models (VLMs), such as GPT-4V and GPT-4o, have demonstrated considerable promise in driving intelligent agent systems that operate within user interfaces (UI). However, the full potential of these multimodal models remains underexplored in real-world applications, particularly when it comes to acting as general agents across diverse operating systems and applications with only vision input. One of the primary limiting factors is the absence of a robust technique for screen parsing which is capable of 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen.

Meet OmniParser, a compact screen parsing module that can convert UI screenshots into structured elements. OmniParser can be used with a variety of models to create agents capable of taking actions on UIs. When used with GPT-4V, it significantly improves the agent capability to generate precisely grounded actions for interface regions.

An agent using OmniParser and GPT-4V achieved the best performance on the recently released WindowsAgentArena (opens in new tab) benchmark.

We are making OmniParser publicly available on GitHub, along with a report describing the training procedure to encourage research on creating agents that can act on different applications and environments.

Creating OmniParser

Curating Specialized Datasets–The development of OmniParser began with the creation of two datasets:

• An interactable icon detection dataset, which was curated from popular web pages and annotated to highlight clickable and actionable regions.
• An icon description dataset, designed to associate each UI element with its corresponding function. This dataset serves as a key component for training models to understand the semantics of detected elements.

Fine-Tuning Detection and Captioning Models–OmniParser leverages two complementary models:

• A detection model, fine-tuned on the interactable icon dataset, which reliably identifies actionable regions within a screenshot.
• A captioning model, trained on the icon description dataset, which extracts the functional semantics of the detected elements, generating contextually accurate descriptions of their intended actions.

Benchmark performance

We demonstrate that with the parsed results, the performance of GPT-4V is greatly improved on ScreenSpot benchmarks. On Mind2Web, OmniParser +GPT-4V achieves better performance compared to GPT-4V agent that uses extra information extracted from HTML. And on AITW benchmark, OmniParser outperforms GPT-4V augmented with specialized Android icon detection model that is trained with view hierarchy. It also achieves the best performance on a new benchmark WindowsAgentArena (opens in new tab)!

To further demonstrate OmniParser is a plugin choice for off-the-shelf vision language models, we show the ScreenSpot benchmark performance of OmniParser combined with recently announced vision language models: Phi-3.5-V and Llama-3.2-V. We hope OmniParser can serve as a general and easy-to-use tool that has the capability to parse general user screen across both PC and mobile platforms without any dependency on extra information such as HTML and view hierarchy in Android.

The post OmniParser for pure vision-based GUI agent appeared first on Microsoft Research.

“The traditional way to fine-tune an LLM for post-training … basically tells the model to emulate good behaviors, but it does not target or correct any mistakes or bad behaviors that it makes explicitly. … Self-improving post-training explicitly identifies and tries to correct bad behaviors or mistakes that the model makes.”

– Corby Rosset, Senior Researcher, Microsoft Research AI Frontiers

Microsoft research copilot experience What is Direct Nash Optimization, and how does it enable language models to self-improve using general preferences? 

Related resources

The post Direct Nash Optimization: Teaching language models to self-improve with general preferences appeared first on Microsoft Research.

“Agents are a very, very powerful abstraction over things like task decomposition, specialization, tool use, etc. Really, you think about which roles you need on your team, and you put together your team of agents, and you get them to talk to one another, and then you start making progress on your task.”

– Adam Fourney, Principal Researcher, Microsoft Research AI Frontiers

Microsoft research copilot experience What updates did Adam Fourney provide on AutoGen, and how does it handle complex tasks with multiple agents? 

Related resources

The post AutoGen Update: Complex Tasks and Agents appeared first on Microsoft Research.