{"id":1059900,"date":"2024-07-25T12:04:25","date_gmt":"2024-07-25T19:04:25","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/tracing-the-path-to-self-adapting-ai-agents\/"},"modified":"2024-08-01T14:02:38","modified_gmt":"2024-08-01T21:02:38","slug":"tracing-the-path-to-self-adapting-ai-agents","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/tracing-the-path-to-self-adapting-ai-agents\/","title":{"rendered":"Tracing the path to self-adapting AI agents"},"content":{"rendered":"\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1400\" height=\"788\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1.jpg\" alt=\"white line icons on blue and green gradient background\" class=\"wp-image-1060101\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1.jpg 1400w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-1280x720.jpg 1280w\" sizes=\"(max-width: 1400px) 100vw, 1400px\" \/><\/figure>\n\n\n\n<p>The games industry has long been a frontier of innovation for AI. In the early 2000s, programmers hand-coded neural networks to breathe life into <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/galciv3.fandom.com\/wiki\/History_of_the_Galactic_Civilizations_franchise\" target=\"_blank\" rel=\"noreferrer noopener\">virtual worlds<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, creating <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.alanzucconi.com\/2020\/07\/27\/the-ai-of-creatures\/\" target=\"_blank\" rel=\"noreferrer noopener\">engaging AI characters<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> that interact with players. Fast forward two decades, neural networks have grown from their humble beginnings to colossal architectures with billions of parameters, powering real-world applications like <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/arxiv.org\/abs\/2303.08774\" target=\"_blank\" rel=\"noreferrer noopener\">ChatGPT<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><strong> <\/strong>and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.bing.com\/chat?q=Microsoft+Copilot&FORM=hpcodx\" target=\"_blank\" rel=\"noreferrer noopener\">Microsoft Copilots<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. The catalyst for this seismic shift in AI scale and capability is the advent of automatic optimization. AutoDiff frameworks like <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/pytorch.org\/docs\/stable\/index.html\" target=\"_blank\" rel=\"noreferrer noopener\">PyTorch<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><strong> <\/strong>and <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.tensorflow.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">Tensorflow<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><strong> <\/strong>have democratized scalable gradient-based end-to-end optimization. This breakthrough has been instrumental in the development of Large Foundation Models (LFMs) that now sit at the core of AI.<\/p>\n\n\n\n<p>Today, the AI systems we interact with are more than just neural network models. They contain intricate workflows that seamlessly integrate customized machine learning models, orchestration code, retrieval modules, and various tools and functions. These components work in concert to create the sophisticated AI experiences that have become an integral part of our digital lives.&nbsp;Nonetheless, up to now, we do not have tools to automatically train these extra components. They are handcrafted through extensive engineering, just like how neural networks were engineered in the early 2000s.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"end-to-end-automatic-optimization-of-ai-systems-1\">End-to-end automatic optimization of AI systems<\/h2>\n\n\n\n<p>The latest research from Microsoft and Stanford University introduces <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/microsoft.github.io\/Trace\/\" target=\"_blank\" rel=\"noreferrer noopener\">Trace<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, a groundbreaking framework poised to revolutionize the automatic optimization of AI systems. Here are three highlights of the transformative potential of Trace:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>End-to-end optimization<\/em>: Trace treats AI systems as computational graphs, akin to neural networks, and optimizes them end-to-end through a generalized back-propagation approach.<\/li>\n\n\n\n<li><em>Dynamic adaptation<\/em>: It handles the dynamic nature of AI systems, where the graph can change with varying inputs and parameters and needs to adapt to various kinds of feedback.<\/li>\n\n\n\n<li><em>Versatile applications<\/em>: Trace can optimize heterogenous parameters (such as prompts and codes) in AI systems. Empirical studies showcase Trace\u2019s ability to optimize diverse problems, including hyperparameter tuning, large language model (LLM) agents, and robot control, often outperforming specialized optimizers.<\/li>\n<\/ul>\n\n\n\n<p>In a nutshell, Trace is a new AutoDiff-like tool for training AI systems without using gradients. This generalization is made possible by a new mathematical formulation of optimization, Optimization with Trace Oracle (OPTO), which can describe end-to-end optimization of AI systems with general feedback (such as numerical losses, natural language, and errors). Instead of propagating gradients, which are not well-defined for AI systems beyond neural networks, Trace propagates Minimal Subgraphs which can then be used to also recover gradients where applicable. Trace is implemented as a PyTorch-like Python library with which users can easily create AI systems and refine them, akin to training neural networks.<\/p>\n\n\n\n<p>In this blog post, we are excited to announce the release of the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/microsoft\/Trace\" target=\"_blank\" rel=\"noreferrer noopener\">Trace Python library<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>. With the help of demos, we\u2019ll show you how this powerful tool can be used to build AI agents that learn and adapt from their experiences, eliminating the need for specialized engineering. <\/p>\n\n\n\n\t<div class=\"border-bottom border-top border-gray-300 mt-5 mb-5 msr-promo text-center text-md-left alignwide\" data-bi-aN=\"promo\" data-bi-id=\"1045002\">\n\t\t\n\n\t\t<p class=\"msr-promo__label text-gray-800 text-center text-uppercase\">\n\t\t<span class=\"px-4 bg-white display-inline-block font-weight-semibold small\">Spotlight: Event<\/span>\n\t<\/p>\n\t\n\t<div class=\"row pt-3 pb-4 align-items-center\">\n\t\t\t\t\t\t<div class=\"msr-promo__media col-12 col-md-5\">\n\t\t\t\t<a class=\"bg-gray-300\" href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/cvpr-2024\/\" aria-label=\"Microsoft at CVPR 2024\" data-bi-cN=\"Microsoft at CVPR 2024\" target=\"_blank\">\n\t\t\t\t\t<img decoding=\"async\" class=\"w-100 display-block\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/05\/CVPR_WebBanner_1920x720.png\" alt=\"teal background triangular pattern\" \/>\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t<div class=\"msr-promo__content p-3 px-5 col-12 col-md\">\n\n\t\t\t\t\t\t\t\t\t<h2 class=\"h4\">Microsoft at CVPR 2024<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<p class=\"large\">Microsoft is a proud sponsor and active participant of\u00a0<a href=\"https:\/\/cvpr2023.thecvf.com\/Conferences\/2024\" target=\"_blank\" rel=\"noreferrer noopener\">CVPR 2024<\/a>, which focuses on advancements in computer vision and pattern recognition. <\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t<div class=\"wp-block-buttons justify-content-center justify-content-md-start\">\n\t\t\t\t\t<div class=\"wp-block-button\">\n\t\t\t\t\t\t<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/event\/cvpr-2024\/\" class=\"btn btn-brand glyph-append glyph-append-chevron-right\" aria-label=\"Learn more\" data-bi-cN=\"Microsoft at CVPR 2024\" target=\"_blank\">\n\t\t\t\t\t\t\tLearn more\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div><!--\/.msr-promo__content-->\n\t<\/div><!--\/.msr-promo__inner-wrap-->\n\t<\/div><!--\/.msr-promo-->\n\t\n\n\n<h2 class=\"wp-block-heading\" id=\"warm-up-building-a-battleship-game-ai-agent-through-learning\">Warm up: Building a Battleship game AI agent through learning<\/h2>\n\n\n\n<p>To start, consider building an AI agent for the classic Battleship board game. In Battleship, a player needs to devise strategies to cleverly locate and attack the opponent\u2019s ships on a hidden board as fast as possible. To build an AI agent with Trace, one simply needs to program the workflow and declare the parameters, like programming a neural network architecture. Here we will design an agent with two components: a <strong>reason<\/strong> function and an <strong>act <\/strong>function, as illustrated in Figure 1a.&nbsp;We provide a basic description of what these two functions should do as docstrings. We leave the functions\u2019 content to be blank and set them to be trainable. At this point, the agent doesn\u2019t know how the Battleship API works. It must not only learn how to play the game, but also learn how to use the unknown API.<\/p>\n\n\n\n<div class=\"wp-block-columns are-vertically-aligned-center is-layout-flex wp-container-core-columns-is-layout-1 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-vertically-aligned-center is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1440\" height=\"1464\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-a-TRACE.png\" alt=\"The agent\u2019s policy is defined as the composition of a reason step and an act step. The codes of both steps are marked as trainable and are initialized as trivial functions. A basic description of what each function is supposed to behave is provided as docstrings in the function definition. \" class=\"wp-image-1059996\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-a-TRACE.png 1440w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-a-TRACE-295x300.png 295w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-a-TRACE-1007x1024.png 1007w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-a-TRACE-768x781.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-a-TRACE-177x180.png 177w\" sizes=\"(max-width: 1440px) 100vw, 1440px\" \/><figcaption class=\"wp-element-caption\">Figure 1a: Write a Trace-trainable policy.<\/figcaption><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-vertically-aligned-center is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1440\" height=\"1464\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-b-TRACE.png\" alt=\"The agent\u2019s policy is optimized by a simple but generic training loop, which mimics neural network training. First the agent\u2019s policy and an iterative optimizer for it are declared. In each iteration, the agent\u2019s policy takes a board configuration as input and outputs a target location. The environment returns feedback on whether the target successfully hits a ship or not. Alternatively, when the agent\u2019s policy triggers any execution error, the error is used as feedback. Then the feedback is propagated to the parameters in the trainable policy for updates. \" class=\"wp-image-1059999\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-b-TRACE.png 1440w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-b-TRACE-295x300.png 295w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-b-TRACE-1007x1024.png 1007w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-b-TRACE-768x781.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-1-b-TRACE-177x180.png 177w\" sizes=\"(max-width: 1440px) 100vw, 1440px\" \/><figcaption class=\"wp-element-caption\">Figure 1b: Optimize using a PyTorch-like API.<\/figcaption><\/figure>\n<\/div>\n<\/div>\n\n\n\n<p>We iteratively train this AI agent to play the game through a simple Python <em><u>for<\/u><\/em> loop, seen in Figure 1b. In each iteration, the agent (that is, policy) sees the board configuration and tries to shoot at a target location on a training board. The environment returns in text whether it\u2019s a hit or a miss. Then, we run Trace to propagate this environment feedback through agent\u2019s decision logic to update the parameters (for example, the policy is like a two-layer network with a <strong>reason<\/strong> layer and an <strong>act <\/strong>layer). These iterations mimic how a human programmer might approach the problem. They run the policy and change the code based on the observed feedback, try different heuristics to solve this problem, and may rewrite the code a few times to fix any execution errors by using stack traces.<\/p>\n\n\n\n<p>In Figure 2, we show the results of this learning agent, where the agent is trained by an LLM-based optimizer <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/publication\/trace-is-the-new-autodiff-unlocking-efficient-optimization-of-computational-workflows\/\">OptoPrime<\/a> in Trace. The performance is measured as the scores of the agent playing on new randomly generated games (different from the training board). We see that the agent understands the Battleship game and proposes the enumeration strategy after one iteration; then, after a few more tries, it starts to develop complex strategies for playing the game.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2276\" height=\"1094\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-2.png\" alt=\"The experimental results show that Trace can quickly learn complex behaviors for Battleship in a few iterations. At iteration 0, the agent is initialized to output a constant coordinate. At iteration 1, the agent learns the simple strategy of enumerating the board. After a few more iterations (e.g., iteration 7), the agent learns a complex strategy to balance unexplored squares vs. adjacent squares to previous hits. In comparison, the state-of-the-art LLM optimizer OPRO only achieves less than 1\/3 of Trace\u2019s performance in this problem. \" class=\"wp-image-1059918\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-2.png 2276w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-2-300x144.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-2-1024x492.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-2-768x369.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-2-1536x738.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-2-2048x984.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-2-240x115.png 240w\" sizes=\"(max-width: 2276px) 100vw, 2276px\" \/><figcaption class=\"wp-element-caption\">Figure 2: Trace optimizes Code-as-Parameter to create a complex Battleship AI from scratch, compared with state-of-the-art LLM-based optimizer OPRO.<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"super-fast-reinforcement-learning-agent-for-robot-control\">Super-fast reinforcement learning agent for robot control<\/h2>\n\n\n\n<p>We can extend the same idea of end-to-end optimization to train more complicated AI systems. In this example, we want to learn a policy code to control a robotic manipulator. Compared to the Battleship example, the problem here has a longer horizon, since the policy would need to drive the robot for multiple time steps before receiving any feedback. Traditionally, such a problem is framed as a reinforcement learning (RL) problem, and usually learning a policy with RL requires tens of thousands of training episodes. We show Trace can be used to effectively solve such a problem, with just dozens of episodes \u2014 a 1,000 times speed-up. We trace an entire episode and perform end-to-end updates through these steps (using the same OptoPrime optimizer). In this way, effectively, Trace performs back-propagation through time (<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/en.wikipedia.org\/wiki\/Backpropagation_through_time\" target=\"_blank\" rel=\"noreferrer noopener\">BPTT<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>).<\/p>\n\n\n\n<p>We conduct experiments using a simulated Sawyer robot arm in the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/meta-world.github.io\/\" target=\"_blank\" rel=\"noreferrer noopener\">Meta-World<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> environment of <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/microsoft.github.io\/LLF-Bench\/\" target=\"_blank\" rel=\"noreferrer noopener\">LLF-Bench<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, as shown in Figure 3. The agent needs to decide a target pose for the robot, which will then be used as a set point for a position controller, to perform a pick-and-place task. Each episode has 10 timesteps, which results in a graph of depth around 30. The agent receives language feedback as intermediate observations (from LLF-Bench) and finally feedback about success and episode return (i.e. cumulative reward for RL) in texts at the end. Like the Battleship example, we initialize the policy code to be a dummy function and let it adapt through interactions, demonstrated in Figure 4. We repetitively train the agent starting from one initial condition, then test it on 10 new held-out initial conditions for generalization. Very quickly, after 13 episodes, we see that the agent learns complex rules to solve the problem, as shown in Figure 3 and Figure 4.<\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-2 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"480\" height=\"480\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-4-iter-0.gif\" alt=\"The video shows how the robot agent performs on new configurations which are not seen during training. At iteration 0, the robot\u2019s policy is initialized to stay at its initial position. \" class=\"wp-image-1060080\"\/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"480\" height=\"480\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-4-iter-1.gif\" alt=\"The video shows how the robot agent performs on new configurations which are not seen during training. At iteration 1, the robot learns to reach the goal but does not grasp the object, which leads to failure in this pick and place task.\" class=\"wp-image-1060083\"\/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"480\" height=\"480\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-4-iter-3.gif\" alt=\"The video shows how the robot agent performs on new configurations which are not seen during training. The robot learns to grasp the object starting from iteration 3 but fails to successfully place and drop the object at the goal correctly. Nonetheless, after dropping the object incorrectly, the robot would attempt to pick up the object and try again. This behavior continues until iteration 12. \" class=\"wp-image-1060086\"\/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"480\" height=\"480\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-4-iter-9.gif\" alt=\"The video shows how the robot agent performs on new configurations which are not seen during training. The robot learns to grasp the object starting from iteration 3 but fails to successfully place and drop the object at the goal correctly. Nonetheless, after dropping the object incorrectly, the robot would attempt to pick up the object and try again. This behavior continues until iteration 12. \" class=\"wp-image-1060089\"\/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"480\" height=\"480\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-4-iter-13.gif\" alt=\"The video shows how the robot agent performs on new configurations which are not seen during training. At iteration 13, the robot learns a generalizable policy to perform pick and place successfully.  \" class=\"wp-image-1060092\"\/><\/figure>\n<\/div>\n<\/div>\n\n\n\n<p class=\"has-text-align-center\"><sub>Figure 3: Trace rapidly learns a generalizable robot controller in the MetaWorld simulated environment. The learning starts from a trivial policy and the robot does not know the exact rule and dynamics of the task. The video shows Trace learns a policy to successfully perform the pick-place task after 13 iterations, through language feedback (e.g., &#8220;The previous step&#8217;s reward was 0.008. The latest arm movement was in a wrong direction. Finishing the task is now more distant than previously. Moving to [-0.07\u00a0 0.68\u00a0 0.12\u00a0 0.\u00a0 ] now is a good idea.\u201d). The video shows the robot tested on 10 held-out initial configurations not seen in training. From left to right: iteration 0 (initial policy which does not move the robot; the video shows the 10 testing configurations), iteration 1 (the robot learned to reach the goal but forgot to pick up the object first), iteration 3 and iteration 9 (The robot learned to pick up the object, attempted to place it to the goal location, but failed), iteration 13 (The robot learned to successfully perform pick-place for all unseen 10 initial configurations, which is the desired behavior.)<\/sub><\/p>\n\n\n\n<p><\/p>\n\n\n\n<figure class=\"wp-block-image size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1832\" height=\"436\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-a.png\" alt=\"The robot\u2019s control policy is initialized to simply output a zero vector, which would make the robot stay at the initial configuration. \" class=\"wp-image-1060008\" style=\"width:900px;height:auto\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-a.png 1832w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-a-300x71.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-a-1024x244.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-a-768x183.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-a-1536x366.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-a-240x57.png 240w\" sizes=\"(max-width: 1832px) 100vw, 1832px\" \/><figcaption class=\"wp-element-caption\"><strong>Initial control code<\/strong><\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1832\" height=\"1250\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-b.png\" alt=\"The control policy learned after 13 iterations is complex decision logic, with many rules to decide when to grasp, how to grasp, and when to released. The decision boundary is never told to the robot and is learned through trial and error in the environment.\" class=\"wp-image-1060011\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-b.png 1832w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-b-300x205.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-b-1024x699.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-b-768x524.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-b-1536x1048.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-5-b-240x164.png 240w\" sizes=\"(max-width: 1832px) 100vw, 1832px\" \/><figcaption class=\"wp-element-caption\"><strong>Learned control code after 13 episodes&nbsp;<\/strong><br><br><center>Figure 4. Trace adapts an initial dummy control policy into a complex, generalizable control policy.<\/center><\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"finale-self-adapting-multi-agent-llm-systems\">Finale: Self-adapting multi-agent LLM systems<\/h2>\n\n\n\n<p>Trace is not limited to code optimization. The Trace framework supports optimizing heterogenous parameters, including codes, prompts, and hyperparameters. Here we demonstrate Trace\u2019s ability to optimize prompts of multiple LLM agents in solving complex household tasks in the <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"http:\/\/virtual-home.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">VirtualHome<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> simulated environment.&nbsp;<\/p>\n\n\n\n<p>Many tasks require multi-agent collaboration to solve efficiently. But crafting the right prompts for multiple LLM agents requires careful engineering. Trace can seamlessly optimize agents\u2019 behaviors based on environmental feedback. Trace automatically constructs the interaction graph of agents and updates each agent\u2019s behavior factoring in the behavior of other agents. Then the agents can automatically evolve to acquire specialized capabilities such as behavioral roles, freeing system designers from the painstaking process of hand-tuning multiple LLM prompts.<\/p>\n\n\n\n<p>We use Trace and OptoPrime to improve ReAct agents that have been <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/github.com\/tobeatraceur\/Organized-LLM-Agents\" target=\"_blank\" rel=\"noreferrer noopener\">carefully orchestrated<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> to complete the VirtualHome tasks. In each step, the agent can interact with the environment (like opening a cabinet) or send a message to another agent when they see each other. We declare the plan of each LLM-based agent (a part of their prompt) as a trainable parameter and use reward as feedback. The experimental results are shown in Figure 5 where agents optimized by Trace can complete the tasks using fewer actions and environment interactions. We observed fascinating emergent pro-social behaviors from agents without being explicitly told to communicate as illustrated in Figure 6. This pro-social interaction behavior changes with different tasks. For example, agents did not communicate with each other for the task of \u201cbook reading,\u201d but they collaborated when asked to \u201cput forks and plates into a dishwasher,\u201d which we show in Figure 7. We also observed other patterns such as role specialization, where one agent became the lead in a given task, and was followed by another agent to assist.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1168\" height=\"1166\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/TRACE_fig-6.png\" alt=\"The multi agent system optimized by Trace requires a smaller number of steps to complete each task (Read Book from 22 to 10 steps; Put Dishwasher from 21 to 19 steps; Prepare Food from 21 to 18 steps).\" class=\"wp-image-1059942\" style=\"width:406px;height:auto\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/TRACE_fig-6.png 1168w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/TRACE_fig-6-300x300.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/TRACE_fig-6-1024x1022.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/TRACE_fig-6-150x150.png 150w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/TRACE_fig-6-768x767.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/TRACE_fig-6-180x180.png 180w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/TRACE_fig-6-360x360.png 360w\" sizes=\"(max-width: 1168px) 100vw, 1168px\" \/><figcaption class=\"wp-element-caption\">Figure 5: We show the number of environmental interaction actions taken to succeed in each task. Trace optimized agent takes fewer steps to succeed, thus more efficient in this environment.<\/figcaption><\/figure>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-3 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"480\" height=\"360\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-7-a.gif\" alt=\"The video shows example behaviors of the agents in the three tasks in VirtualHome.\" class=\"wp-image-1060035\"\/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"480\" height=\"360\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-7-b.gif\" alt=\"The video shows example behaviors of the agents in the three tasks in VirtualHome.\" class=\"wp-image-1060038\"\/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"480\" height=\"360\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/fig-7-c.gif\" alt=\"The video shows example behaviors of the agents in the three tasks in VirtualHome.\" class=\"wp-image-1060041\"\/><\/figure>\n<\/div>\n<\/div>\n\n\n\n<p class=\"has-text-align-center figcaption\"><sup>Figure 6: Demo videos of how Trace agents behave to finish each of the three tasks.<\/sup><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;send_message] < Agent 1 >&nbsp;to < Agent 2 >: I am handing you the < cutleryfork >. Please grab another piece of cutlery or plate to help! \n&#91;send_message] < Agent 2 > to < Agent 1 >: Can you also hand me the < plate > you are holding?\n&#91;send_message] < Agent 1 > to < Agent 2 >: Here's the < cutleryfork >. I'll go grab the < plate > now. \n...\n&#91;send_message] < Agent 1 > to < Agent 2 >: Let's head to the kitchen and put the < cutleryfork > and < plate > into the dishwasher.<\/code><\/pre>\n\n\n\n<p class=\"has-text-align-center\"><sup>Figure 7: Trace learns pro-social behavior in the Dishwasher task. Trace optimized agents send messages to attempt to collaborate while simple ReAct agent will only carry out the tasks.<\/sup><\/p>\n\n\n\n<p>Trace heralds a new era of interactive agents that adapt automatically using various feedback types. This innovation could be the key to unlocking the full potential of AI systems, making them more efficient and responsive than ever before. After witnessing the awesome power of Deep Neural Networks, stay tuned for the next revolution in AI design \u2014&nbsp;<strong>Deep Agent Networks<\/strong>!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introducing Trace, Microsoft and Stanford University&#8217;s novel AI optimization framework, now available as a Python library. Trace adapts dynamically and optimizes a wide range of applications from language models to robot control.<\/p>\n","protected":false},"author":37583,"featured_media":1060101,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[1],"tags":[],"research-area":[13561,13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243984],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-1059900","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-algorithms","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-blog-homepage-featured"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199565,992148],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[144672],"related-projects":[],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Ching-An Cheng","user_id":38991,"display_name":"Ching-An Cheng","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/chinganc\/\" aria-label=\"Visit the profile page for Ching-An Cheng\">Ching-An Cheng<\/a>","is_active":false,"last_first":"Cheng, Ching-An","people_section":0,"alias":"chinganc"},{"type":"guest","value":"allen-nie","user_id":"1060335","display_name":"Allen Nie","author_link":"<a href=\"https:\/\/anie.me\/about\" aria-label=\"Visit the profile page for Allen Nie\">Allen Nie<\/a>","is_active":true,"last_first":"Nie, Allen","people_section":0,"alias":"allen-nie"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-960x540.jpg\" class=\"img-object-cover\" alt=\"white line icons on blue and green gradient background\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2024\/07\/Trace-2024-BlogHeroFeature-1400x788-1.jpg 1400w\" sizes=\"(max-width: 960px) 100vw, 960px\" \/>","byline":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/chinganc\/\" title=\"Go to researcher profile for Ching-An Cheng\" aria-label=\"Go to researcher profile for Ching-An Cheng\" data-bi-type=\"byline author\" data-bi-cN=\"Ching-An Cheng\">Ching-An Cheng<\/a>, Adith Swaminathan, and <a href=\"https:\/\/anie.me\/about\" title=\"Go to researcher profile for Allen Nie\" aria-label=\"Go to researcher profile for Allen Nie\" data-bi-type=\"byline author\" data-bi-cN=\"Allen Nie\">Allen Nie<\/a>","formattedDate":"July 25, 2024","formattedExcerpt":"Introducing Trace, Microsoft and Stanford University&#039;s novel AI optimization framework, now available as a Python library. Trace adapts dynamically and optimizes a wide range of applications from language models to robot control.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1059900"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/37583"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1059900"}],"version-history":[{"count":60,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1059900\/revisions"}],"predecessor-version":[{"id":1063920,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1059900\/revisions\/1063920"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1060101"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1059900"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1059900"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1059900"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1059900"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1059900"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1059900"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1059900"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1059900"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1059900"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1059900"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1059900"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}