{"id":1174157,"date":"2026-06-08T08:21:26","date_gmt":"2026-06-08T15:21:26","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-blog-post&#038;p=1174157"},"modified":"2026-07-10T13:03:19","modified_gmt":"2026-07-10T20:03:19","slug":"sentinelbench-a-benchmark-for-long-running-monitoring-agents","status":"publish","type":"msr-blog-post","link":"https:\/\/www.microsoft.com\/en-us\/research\/articles\/sentinelbench-a-benchmark-for-long-running-monitoring-agents\/","title":{"rendered":"SentinelBench, a Benchmark for Long-Running Monitoring Agents"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><em>By Matheus Kunzler Maldaner, Adam Fourney, Amanda Swearngin, Hussein Mozannar, Gagan Bansal, Maya Murad, Rafah Hosn, Saleema Amershi<\/em><\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"yt-consent-placeholder\" role=\"region\" aria-label=\"Video playback requires cookie consent\" data-video-id=\"BtXkjKwmDjA\" data-poster=\"https:\/\/img.youtube.com\/vi\/BtXkjKwmDjA\/maxresdefault.jpg\"><iframe aria-hidden=\"true\" tabindex=\"-1\" title=\"SentinelBench: MSR Blogpost Video\" width=\"500\" height=\"281\" data-src=\"https:\/\/www.youtube-nocookie.com\/embed\/BtXkjKwmDjA?feature=oembed&rel=0&enablejsapi=1\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><div class=\"yt-consent-placeholder__overlay\"><button class=\"yt-consent-placeholder__play\"><svg width=\"42\" height=\"42\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" aria-hidden=\"true\" focusable=\"false\"><g fill=\"none\" fill-rule=\"evenodd\"><circle fill=\"#000\" opacity=\".556\" cx=\"21\" cy=\"21\" r=\"21\"\/><path stroke=\"#FFF\" d=\"M27.5 22l-12 8.5v-17z\"\/><\/g><\/svg><span class=\"yt-consent-placeholder__label\">Video playback requires cookie consent<\/span><\/button><\/div><\/div>\n<\/div><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Modern AI agents are increasingly capable of long and complex work. <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" target=\"_blank\" href=\"https:\/\/metr.org\/time-horizons\/\">METR <span class=\"sr-only\"> (opens in new tab)<\/span><\/a>has been tracking how this scales, and the 50% task-completion time horizon for frontier models has roughly doubled every seven months, climbing from four seconds in 2019 to more than sixteen hours in 2026. In practice, that means <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" target=\"_blank\" href=\"https:\/\/www.anthropic.com\/engineering\/effective-harnesses-for-long-running-agents\">hours-long codebase migrations and deep research reports<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, where an agent grinds away at a problem in a single sustained sitting, have become increasingly feasible.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But these workflows embed the implicit assumption that meaningful state changes in the environment come only from the agent&#8217;s own actions. In other words, the agent believes it must keep acting to trigger new events to progress with its objective.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That&#8217;s the wrong model for a large class of tasks. For example, no amount of reloading a webpage will make concert tickets go on sale faster, and no amount of search broadening helps when those tickets are exclusive to a single site. Here, the correct behavior is to watch, wait, and act only when the environment changes on its own. But current agents are not optimized for this. They either churn obsessively, burning context and tokens on every page refresh (and risking an error on every extra action), or they give up after a few attempts.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In a <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/tell-me-when-building-agents-that-can-wait-monitor-and-act\/\">previous post<\/a>, we presented designs for agents that are better able to handle monitoring tasks. We referred to such tasks as Sentinel Tasks and previewed a benchmark for measuring agent performance on these workloads. In this post, we formally present that benchmark, <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" target=\"_blank\" href=\"https:\/\/github.com\/microsoft\/sentinel_environments\">SentinelBench<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>, as open source on GitHub, and as a <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" target=\"_blank\" href=\"https:\/\/arxiv.org\/abs\/2606.05342\">detailed technical report<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n<h2 id=\"overview\" class=\"wp-block-heading\">Overview<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">SentinelBench contains 100 tasks spread across 10 high-fidelity synthetic web environments. Each environment mimics a familiar consumer product:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td><strong>MicroMail<\/strong><\/td><td>An email client<\/td><td><strong>MicroGram<\/strong><\/td><td>A photo sharing site<\/td><\/tr><tr><td><strong>MicroChat<\/strong><\/td><td>A team messaging client<\/td><td><strong>MicroTube<\/strong><\/td><td>A online video platform<\/td><\/tr><tr><td><strong>MicroDin<\/strong><\/td><td>A professional network<\/td><td><strong>MicroFy<\/strong><\/td><td>A music streaming platform<\/td><\/tr><tr><td><strong>MicroHub<\/strong><\/td><td>A code hosting website<\/td><td><strong>MicroLendar<\/strong><\/td><td>A calendar application<\/td><\/tr><tr><td><strong>MicroHood<\/strong><\/td><td>A stock trading website<\/td><td><strong>MicroScholar<\/strong><\/td><td>Academic Search<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Table 1: The ten environments in SentinelBench. Each resembles a popular class of web application, and includes both a multi-screen interactive UI, and a catalog of synthetic data from which content and events are sampled.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What makes these environments useful for benchmarking is that they can evolve over time, regardless of which actions an agent takes. Each task ships with a scripted timeline of events that the server plays back as the simulation runs. New emails arrive, stock prices drift, songs land on the trending feed, and so forth. The agent is then tasked to perform a task on a live page, whose state is changing underfoot. Here is one example from the MicroFy environment, a music streaming platform:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">&#8220;Watch the trending feed. When a song drops whose lyrics mention &#8216;subway&#8217;, like it for me.&#8221;<\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">To make the simulations feel coherent, all 10 apps are populated from a shared catalog of 100 synthetic personas and 201 entities (e.g., companies, music bands, news organizations etc.) The principal user (the account the agent is acting on behalf of) is a 29-year-old product associate named Chris Taylor, who has the same identity, social network, and content history across all 10 environments.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"486\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/persona-chris-preview-1-1-1024x486.png\" alt=\"table\" class=\"wp-image-1174863\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/persona-chris-preview-1-1-1024x486.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/persona-chris-preview-1-1-300x142.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/persona-chris-preview-1-1-768x364.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/persona-chris-preview-1-1-1536x729.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/persona-chris-preview-1-1-240x114.png 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/persona-chris-preview-1-1.png 1760w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Figure 2: SentinelBench includes 100 synthetic user personas. Each persona consists of core demographic and biographical attributes plus per-environment sub-profiles that ground the same identity coherently across all 10 simulated applications (MicroGram, MicroScholar, MicroFy, MicroHub, MicroDin shown). Chris Taylor, shown here, is the principal user. I.e., the user whose profile is accessed by the agent.<\/em><\/p>\n\n\n\n<h2 id=\"task-design\" class=\"wp-block-heading\">Task Design<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">SentinelBench tasks can be organized along two independent axes of task design, plus a third category of no-operation tasks. These penalize agents that game the benchmark by unconditionally declaring success moments before the simulation ends.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td><\/td><td class=\"has-text-align-center\" data-align=\"center\"><strong>Absolute<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\"><strong>Relative<\/strong><\/td><\/tr><tr><td><strong>Passive<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\">18<\/td><td class=\"has-text-align-center\" data-align=\"center\">20<\/td><\/tr><tr><td><strong>Active<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\">23<\/td><td class=\"has-text-align-center\" data-align=\"center\">19<\/td><\/tr><tr><td><strong>No-operation<\/strong> (No-op)<\/td><td class=\"has-text-align-center\" data-align=\"center\" colspan=\"2\">20<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Table 2: Task counts by action requirement and criterion type.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Active versus passive.<\/strong> Some monitoring tasks are purely about detection. The agent notices that the condition has occurred and lets the user know. For example, \u201cLet me know when my post about brunch gets 100 likes\u201d, is passive, and can be solved by watching a social feed. Others require the agent to periodically do something to reveal hidden information. In a team messaging app, a task that says &#8220;let me know the moment Charles Davis sends design mockups in any channel&#8221; is active because it forces the agent to keep opening chat channels as new messages arrive, to check for Charles\u2019 message.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Absolute versus relative.<\/strong> The phrasing of the success criterion changes the cognitive demand on the agent. &#8220;Tell me when CHIP exceeds $300 per share&#8221;, an absolute task, is solvable from a single screenshot. &#8220;Tell me when CHIP is up 10%\u201d, a relative task, forces the agent to remember the starting condition across an arbitrary number of polls. The latter can be a challenge for agents with aggressive context management.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>No-operation tasks.<\/strong> Finally, there is an obvious adversarial strategy that any benchmark of this type must address. If an agent &#8220;knows&#8221; when the simulation ends (in our case, 10 minutes by default), it can take increasingly urgent action as the clock nears its end. For example, an agent performing a passive task could unconditionally declare the task complete within seconds of termination, if the task has not already done by that time. To penalize such strategies, 20 of the 100 tasks are deliberately designed so the target condition never fires. The only way to pass a no-op task is to still be watching when the simulation ends.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, by default each task is designed to be solvable inside a 10-minute window, but a single <code>speed_factor<\/code> knob can stretch them out to arbitrary lengths. That&#8217;s useful when we want to amplify the cost of bad waiting strategies, which we&#8217;ll do later.<\/p>\n\n\n\n<h2 id=\"task-life-cycle\" class=\"wp-block-heading\">Task Life Cycle<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We wanted SentinelBench to work with whatever web agent you already have. From the user&#8217;s perspective, the evaluation harness launches an agent from the command shell, with two parameters: the starting page, and the task prompt. The harness doesn&#8217;t require a particular browser framework like Playwright, and it doesn&#8217;t assume any particular observation modality, whether screenshots, accessibility trees, or the DOM. Instead, any web-capable agent that can be launched from a terminal can be evaluated (agents must also exit when their work is done, rather than interactively prompting the user for more work).<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"160\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/lifecycle-1024x160.png\" alt=\"The simulation life cycle for SentinelBench environments. The evaluation harness interacts with\nthe server to transition between life cycle states: Pre-init, Ready, Running and Completed\" class=\"wp-image-1174323\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/lifecycle-1024x160.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/lifecycle-300x47.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/lifecycle-768x120.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/lifecycle-1536x241.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/lifecycle-2048x321.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/lifecycle-240x38.png 240w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Figure 3: The simulation life cycle for SentinelBench environments. The evaluation harness interacts with the server to transition between life cycle states.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once a SentinelBench evaluation is initiated, an evaluation harness enumerates the tasks, and manages them through a short, four state, life cycle. <strong>Pre-initialization<\/strong> is the starting state. The harness calls <code>\/init<\/code> with a scenario (the event timeline, and an SQL query that will be used to score the task), and the server moves to <strong>Ready<\/strong>. The agent is then invoked in the terminal. Here it is given the starting page <code>\/redirect<\/code>, and the task prompt with an extra benchmark instruction appended to the end: to visit <code>\/contact<\/code> \u201cOnce the necessary conditions are met and\/or sufficient actions are taken\u201d. As the name suggests, <code>\/redirect<\/code> serves an HTTP 301 to the actual starting page, at which point the server moves to <strong>Running<\/strong>. From this moment, scheduled events fire at their wall-clock times, the database updates, and the UI reflects those updates as the agent navigates.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The agent signals task completion by accessing <code>\/contact<\/code>, a simple &#8220;contact the user&#8221; web form. That transitions the server to <strong>Completed <\/strong>and cancels any remaining scheduled events. The harness detects the transition and hits <code>\/evaluate<\/code>, which runs a task evaluation SQL query against the live database state to determine success. Notably, since success is measured against the database, not against the agent&#8217;s output, any web-capable agent can be evaluated<\/p>\n\n\n\n<h2 id=\"what-we-measured\" class=\"wp-block-heading\">What We Measured<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To validate that the benchmark distinguishes meaningful differences, and to establish some baseline for comparisons, we ran each of the benchmark tasks under six conditions. Specifically, we chose three multimodal models (GPT-5.4 in low-reasoning mode, GPT-4o, and Qwen 3.5:9B), each paired with two versions of a simple browser agent adapted from <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" target=\"_blank\" href=\"https:\/\/github.com\/microsoft\/magentic-ui\">Magentic-UI<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The two versions differ in only one place, namely what the agent does when it decides to wait. The first variant has a familiar <code>sleep(time)<\/code> tool that unconditionally blocks the agent for a specified number of seconds. The second is a purpose-built <code>wait_for(condition, timeout)<\/code> tool. When invoked, <code>wait_for<\/code> takes a baseline textual snapshot of the page, then loops once per second, computing a textual diff against the baseline. For new changes, not previously evaluated, wait_for invokes the LLM to evaluate the change against the condition. This turns out to be both token efficient and responsive as seen below.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"556\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/microtube-video-total-relative-passive-teaser-1024x556.png\" alt=\"Screenshots and event timelines for an example task in the micro MicroTube environment: \"I can see a few videos on the home page already. Let me know when 5 more new ones have shown up\".\n\nThe timeline shows that the agent using sleep uses many more tool calls before detecting the target event vs. the agent using wait_for,\" class=\"wp-image-1174334\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/microtube-video-total-relative-passive-teaser-1024x556.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/microtube-video-total-relative-passive-teaser-300x163.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/microtube-video-total-relative-passive-teaser-768x417.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/microtube-video-total-relative-passive-teaser-1536x833.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/microtube-video-total-relative-passive-teaser-2048x1111.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/microtube-video-total-relative-passive-teaser-240x130.png 240w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Figure 4: Agent execution timelines in SentinelBench for a representative scenario. Each row visualizes one agent strategy over simulated time as events are played back in the simulation (screenshots show the state of the app after selected events). The <code>sleep<\/code> agent spends most of its time executing fixed interval polling<br>while the <code>wait_for<\/code> agent waits for a condition for an extended period and resumes once the environment changes to meet the condition. SentinelBench enables systematic investigation of agent implementation choices for measuring progress on time-evolving monitoring tasks.<\/em><\/p>\n\n\n\n<h2 id=\"what-we-found\" class=\"wp-block-heading\">What We Found<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">From the baseline evaluation, two findings stood out:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Models are cleanly distinguishable.<\/strong> Overall pass rates ranged from 0.46 (GPT-4o + <code>sleep<\/code>) to 0.75 (GPT-5.4 + <code>wait_for<\/code>). Within each model, passive tasks were easier than active tasks, as expected. Likewise, GPT-4o and Qwen both struggled noticeably more on relative tasks than on absolute tasks, whereas GPT-5.4 proved to be robust on solving relative tasks (actually performing slightly better on relative tasks!)<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2000\" height=\"482\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/success-rate-table-1.png\" alt=\"table\" class=\"wp-image-1174865\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/success-rate-table-1.png 2000w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/success-rate-table-1-300x72.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/success-rate-table-1-1024x247.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/success-rate-table-1-768x185.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/success-rate-table-1-1536x370.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/success-rate-table-1-240x58.png 240w\" sizes=\"auto, (max-width: 2000px) 100vw, 2000px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Table 3: Success rates for three models and two agent configurations (six conditions total). As expected, GPT-5.4 performs better overall than GPT-4o and Qwen 3.5:9B. Likewise, agents configured to use <code>wait_for<\/code>perform about as well as, or better than, agents configured to use <code>sleep<\/code>. We also observe that no-operation<br>tasks are generally easier than passive tasks, and that active tasks are harder still. The one exception is GPT-5.4 with sleep, which performs unexpectedly poorly on no-operation tasks.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">While GPT-5.4 fared extremely well, it exhibited a surprising number of failures on the no-operation tasks, when using the sleep tool (0.70 task completion, vs 0.95 \u2013 1.00 for all other conditions). Inspecting the logs, we found GPT-5.4 with <code>sleep<\/code> giving up on tasks too early, filing the contact form with messages like &#8220;<em>I checked the chats and did not find any conversation where Diana Miller @mentioned you.<\/em>&#8221; The model was effectively acknowledging that the condition had not been met but submitting anyway.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Waiting strategy matters as much as model choice.<\/strong> For GPT-5.4 at the default 10-minute setting, the median task cost was 5.1\u00d7 higher with <code>sleep<\/code> than with wait_for ($1.17 vs. $0.23), with comparable or worse task completion. A similar pattern was observed for the other models, with sleep costing more than <code>wait_for<\/code>, while performing no better.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"352\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/sentinel_api_costs-1-1024x352.png\" alt=\"Box-whisker plot of task API cost in USD for each model and tool configuration. The horizontal axis is logarithmic; Qwen 3.5:9B is roughly two orders of magnitude cheaper than the GPT models, and within every model sleep is consistently more expensive than wait_for. Means landing outside Q3 (GPT-5.4 wait_for, GPT-4o sleep) reflect right-skewed distributions with high-cost outliers visible at the upper whisker.\" class=\"wp-image-1174339\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/sentinel_api_costs-1-1024x352.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/sentinel_api_costs-1-300x103.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/sentinel_api_costs-1-768x264.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/sentinel_api_costs-1-1536x529.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/sentinel_api_costs-1-2048x705.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/sentinel_api_costs-1-240x83.png 240w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Figure 5: Per-task API cost in USD for each model and tool configuration. Box outlines show the interquartile range, the colored bar inside each box marks the median (blue for <code>wait_for<\/code>, orange for <code>sleep<\/code>), whiskers<br>extend to the observed minimum and maximum, and the diamond marks the mean. Dashed horizontal lines separate model groups. The horizontal axis is logarithmic; Qwen 3.5:9B is roughly two orders of magnitude cheaper than the GPT models, and within every model <code>sleep<\/code> is consistently more expensive than <code>wait_for<\/code>.<br>Means landing outside Q3 (GPT-5.4 <code>wait_for<\/code>, GPT-4o <code>sleep<\/code>) reflect right-skewed distributions with high-cost outliers visible at the upper whisker.<\/em><\/p>\n\n\n\n<h2 id=\"evaluation-stretch-goal\" class=\"wp-block-heading\">Evaluation \u201cStretch\u201d Goal<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, we re-ran the best-performing model (GPT-5.4) with <code>speed_factor=0.25<\/code>, which stretches the maximum task length from 10 minutes to 40 minutes. At 40 minutes, the gap between the two waiting strategies becomes dramatic. The median task cost under sleep rises to $4.65, while wait_for remains low, and around $0.48. That&#8217;s a 9.7\u00d7 difference.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"434\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/gpt54-cost-vs-target-time-1-1024x434.png\" alt=\"chart, scatter chart\" class=\"wp-image-1174867\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/gpt54-cost-vs-target-time-1-1024x434.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/gpt54-cost-vs-target-time-1-300x127.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/gpt54-cost-vs-target-time-1-768x326.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/gpt54-cost-vs-target-time-1-1536x652.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/gpt54-cost-vs-target-time-1-240x102.png 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2026\/06\/gpt54-cost-vs-target-time-1.png 2011w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Figure 6: Per-task API cost as a function of target event time for GPT-5.4 under the <code>wait_for<\/code> and <code>sleep<\/code>tool configurations. Successful tasks appear as blue dots. Failed tasks appear as red \u2018X\u2019s. Here, tasks are scaled to take up to 40 minutes to complete (2400 seconds). When using <code>wait_for<\/code>, costs remain relatively<br>stable and low, with some outliers as high as $10.59. When using <code>sleep<\/code>, costs trend upward with time, especially for successful tasks, with costs as high as $31.15.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This trend is even more evident when plotting cost against the ideal task completion time (the time when the target event occurs). Looking at successful tasks (blue dots in the above figure), <code>wait_for<\/code> cost stays nearly flat, while sleep cost climbs steadily. At the high end, one sleep task cost $31.15 to complete. Pass rate doesn&#8217;t compensate either. <code>wait_for<\/code> finishes 69 of 100 tasks correctly, while <code>sleep<\/code> finishes only 56.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Even with the same model, choices made about how an agent waits can dominate everything else about its performance over long-running tasks. SentinelBench surfaces those choices cleanly.<\/p>\n\n\n\n<h2 id=\"availability-and-looking-ahead\" class=\"wp-block-heading\">Availability and Looking Ahead<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We&#8217;re releasing <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" target=\"_blank\" href=\"https:\/\/github.com\/microsoft\/sentinel_environments\">SentinelBench <span class=\"sr-only\"> (opens in new tab)<\/span><\/a>on GitHub. That includes the ten environments, all 100 scenarios with their event timelines. We also include the generation pipeline that produced them, and a <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" rel=\"noopener noreferrer\" target=\"_blank\" href=\"https:\/\/arxiv.org\/abs\/2606.05342\">full technical report on arXiv.<span class=\"sr-only\"> (opens in new tab)<\/span><\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Like any benchmark, SentinelBench makes deliberate tradeoffs. Event timing is artificial rather than sampled from real distributions. The environments are convincing facades but not production systems, and prolonged exploration will find their edges. Most success criteria are still objective rather than judgment-based, and the current tasks watch for persistent conditions &#8212; we haven&#8217;t yet captured the harder category of ephemeral, must-act-now tasks like &#8220;<em>buy the stock if it dips below $500.<\/em>&#8220;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But the broader point holds. As agents take on longer-running work, more of their time will be spent waiting than acting. Building always-on assistants requires being able to measure them, and a benchmark whose primary task is to wait is a useful place to start.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>By Matheus Kunzler Maldaner, Adam Fourney, Amanda Swearngin, Hussein Mozannar, Gagan Bansal, Maya Murad, Rafah Hosn, Saleema Amershi Modern AI agents are increasingly capable of long and complex work. METR (opens in new tab)has been tracking how this scales, and the 50% task-completion time horizon for frontier models has roughly doubled every seven months, climbing [&hellip;]<\/p>\n","protected":false},"author":43879,"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-content-parent":992148,"msr_hide_image_in_river":0,"footnotes":""},"research-area":[13556],"msr-locale":[268875],"msr-post-option":[269148,269142],"class_list":["post-1174157","msr-blog-post","type-msr-blog-post","status-publish","hentry","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-include-in-river"],"msr_assoc_parent":{"id":992148,"type":"lab"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1174157","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-blog-post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/43879"}],"version-history":[{"count":30,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1174157\/revisions"}],"predecessor-version":[{"id":1178483,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1174157\/revisions\/1178483"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1174157"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1174157"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1174157"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1174157"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}