{"id":948789,"date":"2023-06-16T13:30:00","date_gmt":"2023-06-16T20:30:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=948789"},"modified":"2023-06-16T13:35:16","modified_gmt":"2023-06-16T20:35:16","slug":"improving-subseasonal-forecasting-with-machine-learning","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/improving-subseasonal-forecasting-with-machine-learning\/","title":{"rendered":"Improving Subseasonal Forecasting with Machine Learning"},"content":{"rendered":"\n

This content was previously published by Nature Portfolio and Springer Nature Communities (opens in new tab)<\/span><\/a><\/em> on Nature Portfolio Earth and Environment Community (opens in new tab)<\/span><\/a>.<\/em><\/p>\n\n\n\n

Improving our ability to forecast the weather and climate is of interest to all sectors of the economy and to government agencies from the local to the national level. Weather forecasts zero to ten days ahead and climate forecasts seasons to decades ahead are currently used operationally in decision-making, and the accuracy and reliability of these forecasts has improved consistently in recent decades (Troccoli, 2010). However, many critical applications \u2013 including water allocation, wildfire management, and drought and flood mitigation \u2013 require subseasonal forecasts with lead times in between these two extremes (Merryfield et al., 2020; White et al., 2017).<\/p>\n\n\n\n

While short-term forecasting accuracy is largely sustained by physics-based dynamical models, these deterministic methods have limited subseasonal accuracy due to chaos (Lorenz, 1963). Indeed, subseasonal forecasting has long been considered a \u201cpredictability desert\u201d<\/em> due to its complex dependence on both local weather and global climate variables (Vitart et al., 2012). Recent studies (opens in new tab)<\/span><\/a>, however, have highlighted important sources of predictability on subseasonal timescales, and the focus of several recent large-scale research efforts has been to advance the subseasonal capabilities of operational physics-based models (Vitart et al., 2017; Pegion et al., 2019; Lang et al., 2020). Our team has undertaken a parallel effort to demonstrate the value of machine learning methods in improving subseasonal forecasting.<\/p>\n\n\n\n

The Subseasonal Climate Forecast Rodeo<\/h2>\n\n\n\n

To improve the accuracy of subseasonal forecasts, the U.S. Bureau of Reclamation (USBR) and the National Oceanic and Atmospheric Administration (NOAA) launched the Subseasonal Climate Forecast Rodeo (opens in new tab)<\/span><\/a>, a yearlong real-time forecasting challenge in which participants aimed to skillfully predict temperature and precipitation in the western U.S. two-to-four weeks and four-to-six weeks in advance. Our team developed a machine learning approach to the Rodeo and a SubseasonalRodeo dataset (opens in new tab)<\/span><\/a> for training and evaluating subseasonal forecasting systems.<\/p>\n\n\n\n

\"Week
Week 3-4 temperature forecasts and observations for February 5th, 2018. Upper left: Our <\/em>Rodeo<\/em> (opens in new tab)<\/span><\/a> submission. Upper right: Realized temperature anomalies. Bottom left: Forecast of the U.S. operational dynamical model, Climate Forecasting System v2. Bottom right: A standard meteorological forecasting method used as a Rodeo baseline<\/em>.<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n\t
\n\t\t\n\n\t\t

\n\t\tSpotlight: Blog post<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"White\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

Eureka: Evaluating and understanding progress in AI<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

How can we rigorously evaluate and understand state-of-the-art progress in AI? Eureka is an open-source framework for standardizing evaluations of large foundation models, beyond single-score reporting and rankings. Learn more about the extended findings.\u00a0<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tRead more\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

Our final Rodeo solution was an ensemble of two nonlinear regression models. The first integrates a diverse collection of meteorological measurements and dynamic model forecasts and prunes irrelevant predictors using a customized multitask model selection procedure. The second uses only historical measurements of the target variable (temperature or precipitation) and introduces multitask nearest neighbor features into a weighted local linear regression. Each model alone outperforms the debiased operational U.S. Climate Forecasting System version 2 (CFSv2), and, over 2011-2018, an ensemble of our regression models and debiased CFSv2 improves debiased CFSv2 skill by 40%-50% for temperature and 129%-169% for precipitation. See our write-up Improving Subseasonal Forecasting in the Western U.S. with Machine Learning<\/a> for more details. While this work demonstrated the promise of machine learning models for subseasonal forecasting, it also highlighted the complementary strengths of physics- and learning-based approaches and the opportunity to combine those strengths to improve forecasting skill.<\/p>\n\n\n\n

Adaptive Bias Correction (ABC)<\/h2>\n\n\n\n

To harness the complementary strengths of physics- and learning-based models, we next developed a hybrid dynamical-learning framework for improved subseasonal forecasting. In particular, we learn to adaptively correct the biases of dynamical models and apply our novel adaptive bias correction <\/em>(ABC) to improve the skill of subseasonal temperature and precipitation forecasts.<\/p>\n\n\n\n

\"At
At subseasonal lead times, weeks 3-4 and 5-6, ABC doubles or triples the forecasting skill of leading operational dynamical models from the U.S. (CFSv2) and Europe (ECMWF).<\/em><\/figcaption><\/figure>\n\n\n\n

ABC is an ensemble of three new low-cost, high-accuracy machine learning models: Dynamical++, Climatology++, and Persistence++. Each model trains only on past temperature, precipitation, and forecast data and outputs corrections for future forecasts tailored to the site, target date, and dynamical model. Dynamical++ and Climatology++ learn site- and date-specific offsets for dynamical and climatological forecasts by minimizing forecasting error over adaptively-selected training periods. Persistence++ additionally accounts for recent weather trends by combining lagged observations, dynamical forecasts, and climatology to minimize historical forecasting error for each site.<\/p>\n\n\n\n

ABC can be applied operationally as a computationally inexpensive enhancement to any dynamical model forecast, and we use this property to substantially reduce the forecasting errors of eight operational dynamical models, including the state-of-the-art ECMWF model.<\/p>\n\n\n\n

\"ABC
ABC can be applied operationally as a computationally inexpensive enhancement to any dynamical model forecast.<\/em><\/figcaption><\/figure>\n\n\n\n

A practical implication of these improvements for downstream decision-makers is an expanded geographic range for actionable skill, defined here as spatial skill above a given sufficiency threshold. For example, we vary the weeks 5-6 sufficiency threshold from 0 to 0.6 and find that ABC consistently boosts the number of locales with actionable skill over both raw and operationally-debiased CFSv2 and ECMWF.<\/p>\n\n\n\n

\"ABC
ABC consistently boosts the number of locales with forecasting accuracy above a given skill threshold, an important property for operational decision-making in water allocation, wildfire management, and drought and flood mitigation. <\/em><\/figcaption><\/figure>\n\n\n\n

We couple these performance improvements with a practical workflow for explaining ABC skill gains using Cohort Shapley (Mase et al., 2019) and identifying higher-skill windows of opportunity (Mariotti et al., 2020) based on relevant climate variables.<\/p>\n\n\n\n

\"a.)
Our \u201cforecast of opportunity\u201d workflow explains ABC skill gains in terms of relevant climate variables observable at forecast time.<\/em><\/figcaption><\/figure>\n\n\n\n

To facilitate future deployment and development, we also release our model and workflow code through the subseasonal_toolkit (opens in new tab)<\/span><\/a> Python package.<\/p>\n\n\n\n

The SubseasonalClimateUSA dataset<\/h3>\n\n\n\n

To train and evaluate our contiguous US models, we developed a SubseasonalClimateUSA dataset (opens in new tab)<\/span><\/a> housing a diverse collection of ground-truth measurements and model forecasts relevant to subseasonal timescales. The SubseasonalClimateUSA dataset is updated regularly and publicly accessible via the subseasonal_data (opens in new tab)<\/span><\/a> package. In SubseasonalClimateUSA: A Dataset for Subseasonal Forecasting and Benchmarking (opens in new tab)<\/span><\/a>, we used this dataset to benchmark ABC against operational dynamical models and seven state-of-the-art deep learning and machine learning methods from the literature. For each subseasonal forecasting task, ABC and its component models provided the best performance.<\/p>\n\n\n\n

\"Percentage
Percentage improvement in accuracy over operationally-debiased dynamical CFSv2 forecasts. ABC consistently outperforms standard meteorological baselines (Persistence and Climatology) and 7 state-of-the-art machine learning and deep learning methods from the literature.<\/em><\/figcaption><\/figure>\n\n\n\n

Online learning with optimism and delay<\/h2>\n\n\n\n

To provide more flexible and adaptive model ensembling in the operational setting of real-time climate and weather forecasting, we developed three new optimistic online learning algorithms \u2014 AdaHedgeD, DORM, and DORM+ \u2014 that require no parameter tuning and have optimal regret guarantees under delayed feedback.<\/p>\n\n\n\n

\"online
Each year, the <\/em>PoolD<\/em> (opens in new tab)<\/span><\/a> online learning algorithms produce ensemble forecasts with accuracy comparable to the best individual model in hindsight despite observing only 26 observations per year.<\/em><\/figcaption><\/figure>\n\n\n\n

Our open-source Python implementation, available via the PoolD<\/a> library, provides simple strategies for combining the forecasts of different subseasonal forecasting models, adapting the weights of each model based on real-time performance. See our write-up Online Learning with Optimism and Delay<\/a> for more details.<\/p>\n\n\n\n

Looking forward<\/h2>\n\n\n\n

We\u2019re excited to continue exploring machine learning applied to subseasonal forecasting on a global scale, and we hope that our open-source packages will facilitate future subseasonal development and benchmarking. If you have ideas for model or dataset development, please contribute to our open-source Python code<\/a> or contact us!<\/p>\n","protected":false},"excerpt":{"rendered":"

This content was previously published by Nature Portfolio and Springer Nature Communities (opens in new tab) on Nature Portfolio Earth and Environment Community (opens in new tab). Improving our ability to forecast the weather and climate is of interest to all sectors of the economy and to government agencies from the local to the national level. […]<\/p>\n","protected":false},"author":42183,"featured_media":948897,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"categories":[1],"tags":[],"research-area":[13556,198583],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243984],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-948789","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-ecology-environment","msr-locale-en_us","msr-post-option-blog-homepage-featured"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199563],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[847351],"related-projects":[608757],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Lester Mackey","user_id":36161,"display_name":"Lester Mackey","author_link":"Lester Mackey<\/a>","is_active":false,"last_first":"Mackey, Lester","people_section":0,"alias":"lmackey"},{"type":"guest","value":"soukayna-mouatadid","user_id":"666945","display_name":"Soukayna Mouatadid","author_link":"Soukayna Mouatadid<\/a>","is_active":true,"last_first":"Mouatadid, Soukayna","people_section":0,"alias":"soukayna-mouatadid"},{"type":"guest","value":"genevieve-flaspohler","user_id":"666984","display_name":"Genevieve Flaspohler","author_link":"Genevieve Flaspohler<\/a>","is_active":true,"last_first":"Flaspohler, Genevieve","people_section":0,"alias":"genevieve-flaspohler"},{"type":"guest","value":"judah-cohen","user_id":"608778","display_name":"Judah Cohen","author_link":"Judah Cohen<\/a>","is_active":true,"last_first":"Cohen, Judah","people_section":0,"alias":"judah-cohen"}],"msr_type":"Post","featured_image_thumbnail":"\"Nature","byline":"Lester Mackey<\/a>, Soukayna Mouatadid<\/a>, Genevieve Flaspohler<\/a>, and Judah Cohen<\/a>","formattedDate":"June 16, 2023","formattedExcerpt":"This content was previously published by Nature Portfolio and Springer Nature Communities (opens in new tab) on Nature Portfolio Earth and Environment Community (opens in new tab). Improving our ability to forecast the weather and climate is of interest to all sectors of the economy and…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/948789"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/42183"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=948789"}],"version-history":[{"count":10,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/948789\/revisions"}],"predecessor-version":[{"id":949938,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/948789\/revisions\/949938"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/948897"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=948789"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=948789"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=948789"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=948789"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=948789"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=948789"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=948789"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=948789"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=948789"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=948789"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=948789"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}