(opens in new tab)<\/span><\/a><\/p>\n\n\n\nParadigm differences: Methodological analysis of time series forecasting<\/h2>\n\n\n\n The benchmark study using ProbTS highlights two crucial methodological differences found in contemporary research: the forecasting paradigms for point and distributional estimation, and the decoding schemes for variable-length forecasting across different horizons.<\/p>\n\n\n\n
Forecasting paradigms for point and distributional estimation<\/strong><\/p>\n\n\n\n\nPoint forecasting only: Approaches that support only point forecasting, providing expected estimates without uncertainty quantification.<\/li>\n\n\n\n Predefined distribution heads: Methods that use predefined distribution heads to generate distributional forecasts, offering a fixed structure for uncertainty estimation.<\/li>\n\n\n\n Neural distribution estimation modules: Techniques employing neural network-based modules to estimate distributions, allowing for more flexible and potentially more accurate uncertainty quantification.<\/li>\n<\/ul>\n\n\n\nDecoding schemes for variable-length forecasting across different horizons<\/strong><\/p>\n\n\n\n\nAutoregressive (AR) methods: These methods generate forecasts step-by-step, using previous predictions as inputs for future time steps. They are suitable for scenarios where sequential dependencies are crucial.<\/li>\n\n\n\n Non-Autoregressive (NAR) methods: These methods produce forecasts for all time steps simultaneously, offering faster predictions and potentially better performance for long-term forecasting.<\/li>\n<\/ul>\n\n\n\nFigure 1. An overview of ProbTS. illustrating its coverage across diverse forecasting scenarios, including typical models developed in different research branches and comprehensive evaluation metrics.<\/figcaption><\/figure>\n\n\n\n<\/p>\n\n\n\n
The research results under the ProbTS framework reveals several key insights:<\/p>\n\n\n\n
Firstly, due to customized neural architectures, long-term point forecasting approaches excel in long-term scenarios but struggle in short-term cases and with complex data distributions. The lack of uncertainty quantification leads to significant performance gaps compared to probabilistic models when dealing with complex data distributions. Conversely, short-term probabilistic forecasting methods are proficient in short-term distributional forecasting but exhibit performance degradation and efficiency issues in long-term scenarios.<\/p>\n\n\n\n
Secondly, regarding the characteristics of different decoding schemes, NAR decoding is predominantly used in long-term point forecasting models, while short-term probabilistic forecasting models do not show such a biased preference. Meanwhile, AR decoding suffers from error accumulation over extended horizons but may perform better with strong seasonal patterns.<\/p>\n\n\n\n
Lastly, for current time-series foundation models, the limitations of AR decoding are reaffirmed in long-term forecasting. Additionally, foundation models show limited support for distributional forecasting, highlighting the need for improved modeling of complex data distributions.<\/p>\n\n\n\n
<\/h2>\n\n\n\nDetailed results and analysis on classical time-series models<\/h2>\n\n\n\n Researchers benchmark classical time-series models across a wide range of forecasting scenarios, encompassing both short and long prediction horizons. The evaluation includes both point forecasting metrics (Normalized Mean Absolute Error, NMAE) and distributional forecasting metrics (Continuous Ranked Probability Score, CRPS). Additionally, researchers calculate a non-Gaussianity score to quantify the complexity of data distribution for each forecasting scenario.<\/p>\n\n\n\n
Based on the data presented in Figure 2, several noteworthy observations emerge:<\/p>\n\n\n\n
\nLimitations of long-term point forecasting models: <\/strong>Customized neural architectures for time-series, primarily designed for long-term point forecasting, excel in long-term scenarios. However, their architectural benefits significantly diminish in short-term cases (see Figure 2(a) and 2(c)). Furthermore, their inability to quantify forecasting uncertainty results in larger performance gaps compared to probabilistic models, especially when the data distribution is complex (see Figure 2(c) and 2(d)).<\/li>\n\n\n\nWeaknesses of short-term probabilistic forecasting models: <\/strong>Current probabilistic forecasting models, while proficient in short-term distributional forecasting, face challenges in long-term scenarios, as evidenced by significant performance degradations (see Figure 2(a) and 2(b)). In addition to unsatisfactory performance, some models experience severe efficiency issues as the prediction horizon increases.<\/li>\n<\/ul>\n\n\n\nFigure 2. Benchmark classical time-series models with ProbTS<\/figcaption><\/figure>\n\n\n\n<\/p>\n\n\n\n
These observations yield several important implications. Firstly, effective architecture designs for short-term forecasting remain elusive and warrant further research. Secondly, the ability to characterize complex data distributions is crucial, as long-term distributional forecasting presents significant challenges in both performance and efficiency.<\/p>\n\n\n\n
Following that, researchers compare Autoregressive (AR) and Non-Autoregressive (NAR) decoding schemes across various forecasting scenarios, highlighting their respective pros and cons in relation to forecasting horizons, trend strength, and seasonality strength.<\/p>\n\n\n\nFigure 3. Comparing AR and NAR schemes with ProbTS<\/figcaption><\/figure>\n\n\n\n<\/p>\n\n\n\n
Researchers find that nearly all long-term point forecasting models use the NAR decoding scheme for multi-step outputs, whereas probabilistic forecasting models exhibit a more balanced use of AR and NAR schemes. Researchers aim to elucidate this disparity and highlight the pros and cons of each scheme, as shown in Figure 3.<\/p>\n\n\n\n
\nError accumulation in AR decoding: <\/strong>Figure 3(a) shows that existing AR models experiences a larger performance gap compared to NAR methods as the prediction horizon increases, suggesting that AR may suffer from error accumulation.<\/li>\n\n\n\nImpact of trend strength: <\/strong>Figure 3(b) connects the performance gap with trend strength, indicating that strong trending effects can lead to significant performance differences between NAR and AR models. However, there are exceptions where strong trends do not cause substantial performance degradation in AR-based models.<\/li>\n\n\n\nImpact of seasonality strength: <\/strong>Figure 3(c) explains these exceptions by introducing seasonality strength as a factor. Surprisingly, AR-based models perform better in scenarios with strong seasonal patterns, likely due to their parameter efficiency in such contexts.<\/li>\n\n\n\nCombined effects of trend and seasonality: <\/strong>Figure 3(d) demonstrates the combined effects of trend and seasonality on performance differences.<\/li>\n<\/ul>\n\n\n\nBased on these analyses, researchers point out that the choice between AR and NAR decoding schemes in different research branches is primarily driven by the specific data characteristics in their focused forecasting scenarios. This explains the preference for the NAR decoding paradigm in most long-term forecasting models. However, this preference for NAR may overlook the advantages of AR, particularly its effectiveness in handling strong seasonality. Since both NAR and AR have their own strengths and weaknesses, future research should aim for a more balanced exploration, leveraging their unique advantages and addressing their limitations.<\/p>\n\n\n\n
Detailed results and analysis on time-series foundation models<\/h2>\n\n\n\n Researchers then extend the analysis framework to include recent time-series foundation models and examine their distributional forecasting capabilities. <\/p>\n\n\n\nFigure 4. Benchmark\u00a0 recent time-series foundation models with ProbTS<\/figcaption><\/figure>\n\n\n\n<\/p>\n\n\n\n
The results show that:<\/p>\n\n\n\n
\nAR vs. NAR decoding in long-term forecasting: <\/strong>Figure 4(a) reaffirms the limitations of AR decoding over extended forecasting horizons. This suggests that time-series data, due to its continuous nature, may require special adaptations beyond those used in language modeling (which operates in a discrete space). Additionally, it is confirmed that AR-based and NAR-based models can deliver comparable performance in short-term scenarios, with AR-based models occasionally outperforming their NAR counterparts.<\/li>\n\n\n\nDistributional forecasting capabilities: <\/strong>Figure 4(b) compares the distributional forecasting capabilities of foundation models with CSDI, underscoring the importance of capturing complex data distributions. Current foundation models demonstrate limited support for distributional forecasting, typically using predefined distribution heads (e.g., MOIRAI) or approximated distribution modeling in a value-quantized space (e.g., Chronos).<\/li>\n<\/ul>\n\n\n\nThese observations lead to several important conclusions: While AR-based models can be effective in short-term scenarios, their performance diminishes over longer horizons, highlighting the need for further refinement. Time-series data may require unique treatments to optimize AR decoding, particularly for long-term forecasting. The ability to accurately model complex data distributions remains a critical area for improvement in time-series foundation models.<\/p>\n\n\n\n
<\/h2>\n\n\n\n Based on the evaluation and analysis of existing methods, researchers have proposed several important future directions for time series prediction models. These directions, if pursued, could significantly impact key scenarios across various industry sectors.<\/p>\n\n\n\n
Future direction 1: Adopting a comprehensive perspective of forecasting demands. <\/strong>One primary future direction is to adopt a holistic perspective of essential forecasting demands when developing new models. This approach can help rethink the methodological choices of different models, understand their strengths and weaknesses, and foster more diverse research explorations.<\/p>\n\n\n\nFuture direction 2: Designing a universal model. <\/strong>A fundamental question raised by these results is whether we can develop a universal model that fulfills all essential forecasting demands or if we should treat different forecasting demands separately, introducing specific techniques for each. While it is challenging to provide a definitive answer, the ultimate goal could be to create a universal model. When developing such a model, it is necessary to consider issues such as input representation, encoding architecture, decoding scheme, and distribution estimation module, etc. Additionally, future research is needed to address the challenge of distributional forecasting in high-dimensional and noisy scenarios, particularly for long horizons, and to leverage the advantages of both AR and NAR decoding schemes while avoiding their weaknesses.<\/p>\n\n\n\nFuture direction 3: Developing tools for future research. <\/strong>To support future research in these directions, researchers have made the ProbTS tool publicly available, hoping this tool will facilitate advancements in the field and encourage collective efforts from the research community.<\/p>\n\n\n\nBy addressing these future directions, researchers aim to push the boundaries of time-series forecasting, ultimately developing models that are more robust, versatile, and capable of handling a wide range of forecasting challenges. This progress holds the potential to significantly impact numerous industries, leading to better decision-making, optimized operations, and improved outcomes across critical sectors.<\/p>\n","protected":false},"excerpt":{"rendered":"
Author: Machine Learning Group Time-series forecasting is crucial across various industries, including health, energy, commerce, climate, etc. Accurate forecasts over different prediction horizons are essential for both short-term and long-term planning needs across these domains. For instance, during a public health emergency such as the COVID-19 pandemic, projections of infected cases and fatalities over one to […]<\/p>\n","protected":false},"author":34512,"featured_media":1088916,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-content-parent":199560,"footnotes":""},"research-area":[],"msr-locale":[268875],"msr-post-option":[],"class_list":["post-1088886","msr-blog-post","type-msr-blog-post","status-publish","has-post-thumbnail","hentry","msr-locale-en_us"],"msr_assoc_parent":{"id":199560,"type":"lab"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1088886"}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-blog-post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/34512"}],"version-history":[{"count":6,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1088886\/revisions"}],"predecessor-version":[{"id":1088922,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1088886\/revisions\/1088922"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1088916"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1088886"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1088886"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1088886"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1088886"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}