Multivariate Probabilistic Time Series Forecasting with Informer

Hugging Face has integrated the Informer model into the Transformers library, enabling efficient multivariate probabilistic time series forecasting. The Informer model addresses the scalability issues of vanilla Transformers when handling long sequence time-series forecasting (LSTF) by reducing both time and space complexity.

Solving Transformer Scalability for Long Sequences

Vanilla Transformers suffer from quadratic computational complexity $O(T^2 D)$ and memory usage $O(N T^2)$ when stacking $N$ layers for a sequence of length $T$. Informer introduces two primary mechanisms to mitigate these bottlenecks:

ProbSparse Attention

ProbSparse attention reduces time and space complexity to $O(T \log T)$ by identifying "active" queries that contribute significantly to the attention mechanism and ignoring "lazy" queries that generate trivial attention.

  • Mechanism: The model uses a Query Sparsity Measurement $M(q_i, K)$ based on the KL divergence between the actual query distribution and a uniform distribution.
  • Implementation: In practice, the measurement is calculated as the difference between the maximum dot-product and the average dot-product of a query against a randomly sampled subset of keys. Only the top $u$ active queries (where $u = c \cdot \log L_Q$) are used to calculate attention weights.

Distilling Operation

To reduce memory bottlenecks when stacking layers, Informer employs a distilling operation that reduces the input size between encoder layers by half. This is achieved by applying 1D convolution layers followed by max pooling between each encoder layer, reducing total memory usage to $O(N \cdot T \log T)$.

Multivariate Probabilistic Forecasting

Informer supports multivariate probabilistic forecasting, which involves predicting the distribution of a future vector of time-series target values rather than a single 1D distribution.

  • Modeling Approach: The core Transformer/Informer architecture remains unchanged for multivariate data; the modification occurs on the output (emission) side.
  • Distribution Handling: To manage the computational expense of high-dimensional joint conditional distributions, the implementation uses independent (diagonal) emissions, supported by the distribution families implemented in the Transformers library.

Practical Implementation and Training

Using the InformerForPrediction model, users can train on multivariate datasets such as the traffic_hourly dataset from the Monash Time Series Forecasting repository. Key implementation details include:

  • Feature Engineering: The model utilizes "lags" (look-back mechanisms) and temporal features (e.g., hour of day, day of week) to provide positional encoding and context.
  • Data Pipeline: Integration with GluonTS allows for the use of MultivariateGrouper to convert individual series into 2D matrices and InstanceSplitter to sample context and prediction windows.
  • Inference: Autoregressive generation is handled via the generate() method, which samples values from the predicted distribution to produce forecasts.

Performance and Benchmarks

When tested on the Traffic Hourly dataset, the multivariate Informer implementation achieved a Mean Absolute Scaled Error (MASE) of 1.191 and a symmetric Mean Absolute Percentage Error (sMAPE) of 0.532.

Comparing these results to other models on the Monash Time Series Repository, the authors note that multivariate forecasts can sometimes be less accurate than univariate ones due to the difficulty of estimating cross-series correlations and the risk of learning spurious correlations. In the specific case of the Traffic Hourly dataset, the vanilla univariate Transformer performed best with a MASE of 0.821.

Multivariate models tend to work well when trained on a lot of data.

Resources

Sources