Exploring the effects of different metric weightings available in AutoTS

One of the most unique features of the forecasting package AutoTS is the ability to definite a metric weighting object that controls how the algorithm chooses an optimal model.

While deep learning models typically use a single loss function for optimization, AutoTS's approach is more flexible. It allows for a multi-objective optimization that can balance various aspects of forecast quality. This is particularly useful in business settings where forecasts may need to meet multiple criteria simultaneously.

Most local time series models on their own lack built-in capabilities for multi-metric optimization. AutoTS's approach provides a significant advantage in this regard, allowing for more nuanced model selection that can be tailored to specific business needs without requiring custom implementation for each model type.

Let's take a look at how different metric weightings choose different models and how that may produce forecasts with different characteristics. Building this intuition is a valuable skill in being able to generate forecasts that align with your needs.

Overview of some supported metrics

1. sMAPE (Symmetric Mean Absolute Percentage Error):

   • Most versatile metric across multiple series as it's scaled

   • May lead to more balanced forecasts across different scales

   • Not ideal for series with many zero values

2. MAE (Mean Absolute Error):

   • Unscaled metric, favors larger scale series in multivariate forecasting

   • May lead to forecasts that minimize absolute errors

3. RMSE (Root Mean Square Error):

   • Unscaled metric, more sensitive to large errors

   • May result in forecasts that avoid large deviations

4. MADE (Mean Absolute Differential Error):

   • Measures how well forecast changes match actual changes

   • May encourage more dynamic, "wavy" forecasts rather than flat lines

5. MAGE (Mean Absolute Aggregate Error):

   • Measures error of rollup forecasts

   • Useful for hierarchical/grouped forecasts

   • May lead to forecasts that minimize over/underestimation when summed

6. MLE (Mean Logarithmic Error):

   • Penalizes under-prediction more than over-prediction

   • May result in slightly higher forecasts to avoid under-prediction

7. iMLE (Inverse Mean Logarithmic Error):

   • Opposite of MLE, penalizes over-prediction more

   • May lead to slightly lower forecasts to avoid over-prediction

8. SPL (Scaled Pinball Loss):

   • Optimal for upper/lower quantile forecast accuracies

   • May improve the quality of probabilistic forecasts

9. Containment:

   • Measures percent of test data within upper/lower forecasts

   • May lead to more accurate prediction intervals

10. Contour:

    • Measures point-wise directional accuracy of forecasts

    • May result in forecasts that better match the visual pattern of actuals

11. Runtime:

    • Considers model execution speed

    • May favor faster models if given significant weight

Developing a visual intuition

Let's take a look at how adjusting the weighting of some of these metrics in the AutoTS model search will affect the final forecast.

In this example, we will be forecasting the next 30 days of Wikipedia page views for the "United States" page. Each of these forecasts ran with the following configuration with adjustments made to the metric_weighting dict:

metric_weighting = {
    'smape_weighting': 5,
    'mae_weighting': 2,
    'rmse_weighting': 2,
    'made_weighting': 0.5,
    'mage_weighting': 1,
    'mle_weighting': 0,
    'imle_weighting': 0,
    'spl_weighting': 3,
    'containment_weighting': 0,
    'contour_weighting': 1,
    'runtime_weighting': 0.05,
}

model = AutoTS(
    forecast_length=30,
    frequency='infer',
    ensemble=None,
    max_generations=5,
    num_validations=2,
    metric_weighting=metric_weighting
)
model = model.fit(df, date_col='datetime', value_col='value', id_col='series_id')

Balanced weighting

Let's start off with the default metric weighting that you'd find on AutoTS's documentation. This should give us a relatively balanced forecast in terms of shape, accuracy, and usability.

Increased MLE weighting

By increasing MLE, we find that the model avoids under-predictions. This is a useful feature in demand forecasting where out-of-stock events are costly and the cost to hold inventory is manageable. Tuning this metric further can help you find the right balance of inventory held and OOS events.

Increased MADE weighting

By increasing MADE, we are telling the algorithm to find a model that moves in a similar manner to the original data. As a result, this forecast adheres to the dynamics of the historical data better than the others. This is useful in generating forecasts that look good to the human eye - a function important in catering to stakeholders that are more abstracted from the forecast generation process.

Increased RMSE, decreased sMAPE

In this example, we explore how these two modifications effect the chosen model and best forecast. The higher weight on RMSE will prioritize models that minimize large errors. This could result in:

• Forecasts that are more conservative and less likely to have significant deviations from the actual values.

• Potentially smoother forecasts, as models that produce occasional large errors will be penalized more heavily.

• In cases with outliers or high volatility, the selected models might prioritize capturing these extreme events at the expense of overall fit.

Find what works for your use case

The metric weighting system in AutoTS provides a powerful and flexible approach to time series forecasting that goes beyond traditional single-metric optimization. By allowing users to adjust the importance of various performance metrics, AutoTS can produce forecasts that balance multiple objectives, tailored to specific business needs or data characteristics.

Our exploration revealed that different weighting configurations can significantly influence the nature of the selected forecasts. Emphasizing scale-invariant metrics like sMAPE ensures balanced performance across series of varying magnitudes, while increasing the weight of error-focused metrics like RMSE can lead to more conservative forecasts that minimize large deviations. Notably, boosting the importance of dynamics-focused metrics such as MADE can result in forecasts that more closely mimic the short-term patterns and changes in the underlying data, potentially at the cost of overall error minimization.

This customizable approach offers several advantages over both traditional time series models and typical deep learning approaches. Unlike most local models that optimize for a single criterion, AutoTS can adapt to complex, multi-faceted forecasting requirements. Compared to deep learning models that typically use a single loss function, AutoTS's weighted metric system provides greater flexibility in model selection and evaluation.

The metric weighting system in AutoTS represents a sophisticated solution to the challenges of automated time series forecasting. It allows for nuanced, context-specific model selection that can adapt to diverse forecasting scenarios, from prioritizing overall accuracy to capturing fine-grained dynamics or ensuring reliable probabilistic forecasts. This flexibility, combined with AutoTS's extensive model library and automated feature search, positions it as a powerful tool for tackling a wide range of time series forecasting tasks in both research and practical business applications.

If you want help or guidance implementing this approach, reach out to our friends at Nousot. They're forecasting experts that have applied AutoTS to real-world applications many times. You can message the creator of the package himself @sirith-llc or data science director at Nousot @mahyar with any inquires as well.