Autoformer – Transformers are effective for time series forecasting

Introduction

Hugging Face announced that the Autoformer model is now available in the 🤗 Transformers library. The post demonstrates that Transformer‑based models are effective for time‑series forecasting, countering claims that simple linear models such as DLinear are superior.

Benchmarking - Transformers vs. DLinear

On the Traffic, Exchange‑Rate, and Electricity datasets, the Autoformer model achieves lower MASE than the DLinear model. Specifically, Autoformer MASE values are 0.910 on Traffic, 1.087 on Exchange‑Rate, and 0.751 on Electricity, while DLinear MASE values are 0.965, 1.690, and 0.831 respectively. This shows Autoformer outperforms DLinear on all three datasets.

Autoformer - Under The Hood

Autoformer improves time‑series forecasting by incorporating a Decomposition Layer and an Autocorrelation Mechanism.

Decomposition Layer

The Decomposition Layer splits an input series into trend and seasonal components using moving average pooling. For an input $\mathcal{X} \in \mathbb{R}^{L \times d}$, the trend is $\mathcal{X}{\text{trend}} = \text{AvgPool}(\text{Padding}(\mathcal{X}))$ and the seasonal part is $\mathcal{X}{\text{seasonal}} = \mathcal{X} - \mathcal{X}_{\text{trend}}$. A PyTorch implementation pads the series, applies average pooling, and returns the two components.

Attention (Autocorrelation) Mechanism

Autoformer replaces standard self‑attention with an autocorrelation mechanism that operates in the frequency domain. Autocorrelation of queries and keys is computed via FFT, yielding $O(L \log L)$ complexity. The resulting weights are aggregated by time delay: top‑k delays are selected, softmax‑normalized, and used to weight rolled value states. This mechanism captures period‑based dependencies.

DLinear - Under The Hood

DLinear is a simple feed‑forward network that applies the same Decomposition Layer as Autoformer, then passes the seasonal and trend parts through separate linear layers and sums their outputs. In the probabilistic setting, the linear layers project to prediction-length * hidden dimensions before a probabilistic head maps to distribution parameters.

Example: Traffic Dataset

The Traffic dataset contains 862 hourly time series of San Francisco freeway occupancy (range [0,1]) from 2015‑2016. For experiments, the prediction length is set to 24, context length to 48, batch size to 128, and models are trained for 50 epochs with small transformer configurations (encoder_layers=2, decoder_layers=2, d_model=16). Pre‑trained checkpoints for Autoformer and other models are available on the Hugging Face Hub.

Load Dataset

The dataset is loaded via gluonts.dataset.repository.datasets.get_dataset("traffic". Frequency and prediction length are taken from the dataset metadata.

Define Transformations

A transformation chain built with GluonTS components prepares the data: removes unused static/dynamic fields, casts to NumPy, adds observed indicator, time features, age feature, stacks temporal features, and renames fields to match Hugging Face expectations.

Define InstanceSplitter

An InstanceSplitter samples windows of context_length and prediction_length from the data. It can operate in train, validation, or test mode, returning appropriate past/future keys for model input.

Create PyTorch DataLoaders

Training, validation, and test DataLoaders are created by applying the transformation chain and instance splitter, then batching with as_stacked_batches. The training loader shuffles and cycles data; the validation and test loaders use the last context window for back‑testing and prediction.

Evaluate on Autoformer

Using the pre‑traced Autoformer checkpoint (kashif/autoformer-traffic-hourly), forecasts are generated with the model’s generate() method. The median forecast across samples is compared to ground truth using the MASE metric from the 🤗 Evaluate library. The average MASE over the test set is 0.910.

Evaluate on DLinear

A DLinear estimator is configured with the same hyperparameters as the Autoformer experiment (prediction length 24, context length 48, scaling std, hidden dimension 2). After training on the training set, evaluation on the test set yields a MASE of 0.965.

Conclusion

Transformer‑based models, including the vanilla Transformer, Informer, and Autoformer, achieve lower MASE than DLinear on the Traffic dataset (vanilla Transformer 0.876, Autoformer 0.910, DLinear 0.965). Multivariate variants tend to perform worse than their univariate counterparts on this small dataset due to difficulty estimating cross‑series correlations. The results confirm that Transformers remain effective for time‑series forecasting, especially when large‑scale generic time‑series datasets become available for pre‑training.

Acknowledgements

Thanks to Lysandre Debut and Pedro Cuenca for their insightful comments and help during this project.