Time Series Transformer probabilistic forecasting with 🤗 Transformers

TL;DR

Hugging Face introduced the Time Series Transformer, a vanilla encoder‑decoder Transformer that learns global probabilistic forecasts for univariate series and outperforms traditional baselines on the Tourism Monthly dataset.

Why a Global Probabilistic Model?

Training a single model on many related series (a global model) lets the network capture shared patterns and latent representations, unlike classical “local” methods that fit each series independently. Probabilistic forecasting—predicting a full distribution rather than a point estimate—provides uncertainty quantification that is essential for downstream decision making.

Architecture Overview

The Time Series Transformer reuses the standard Transformer (Vaswani et al., 2017) in an encoder‑decoder configuration:

  • Encoder consumes a fixed‑size context window of past observations.
  • Decoder autoregressively generates future values using causal masking, analogous to text generation.
  • Distribution head (default: Student‑t) outputs parameters of a probabilistic distribution for each forecast step.

Key benefits:

  • Handles missing values via an attention_mask‑like mechanism.
  • Supports arbitrary context and prediction lengths through windowed training.
  • Leverages the same API as NLP models, enabling generate() for inference.

Model Configuration Details

from transformers import TimeSeriesTransformerConfig, TimeSeriesTransformerForPrediction

config = TimeSeriesTransformerConfig(
    prediction_length=24,               # forecast horizon (months)
    context_length=48,                  # encoder window (2Ă— horizon)
    lags_sequence=[1,2,3,4,5,6,7,11,12,13,23,24,25,35,36,37],
    num_time_features=2,                # month‑of‑year + age feature
    num_static_categorical_features=1, # series ID
    cardinality=[366],                  # 366 regions in the dataset
    embedding_dimension=[2],
    encoder_layers=4,
    decoder_layers=4,
    d_model=32,
)

model = TimeSeriesTransformerForPrediction(config)
  • The model learns a Student‑t distribution (model.config.distribution_output == "student_t").
  • Static categorical embeddings encode the identity of each series, allowing a single model to serve all 366 series.

Data Pipeline (GluonTS + 🤗 Datasets)

  1. Load the Monash tourism_monthly dataset (train/validation/test splits, 366 series).
  2. Convert start timestamps to pandas.Period for easy time‑feature generation.
  3. Define a GluonTS transformation chain that:
    • Removes unused static/dynamic fields.
    • Converts fields to NumPy arrays.
    • Adds an observed‑mask for missing values.
    • Generates time features (month_of_year) and an age feature.
    • Stacks temporal features and renames fields to match the Transformer API.
  4. Create an InstanceSplitter that samples windows of size context_length + max(lags) for the encoder and prediction_length for the decoder. It supports three modes: train (random windows), validation (last window), and test (last context only).
  5. Build DataLoaders that batch the transformed instances into tensors (past_values, past_time_features, future_time_features, etc.).

Training Loop (Accelerate)

from accelerate import Accelerator
from torch.optim import AdamW

accelerator = Accelerator()
model.to(accelerator.device)
optimizer = AdamW(model.parameters(), lr=6e-4, betas=(0.9, 0.95), weight_decay=1e-1)
model, optimizer, train_loader = accelerator.prepare(model, optimizer, train_loader)

model.train()
for epoch in range(40):
    for batch in train_loader:
        optimizer.zero_grad()
        outputs = model(**batch)
        accelerator.backward(outputs.loss)
        optimizer.step()
  • The decoder automatically shifts future_values to compute a likelihood loss.
  • No hyper‑parameter sweep was performed; 40 epochs sufficed to achieve strong results.

Inference with Autoregressive Generation

model.eval()
forecasts = []
for batch in test_loader:
    out = model.generate(**batch)
    forecasts.append(out.sequences.cpu().numpy())
forecasts = np.vstack(forecasts)   # shape: (366, 100, 24)
  • generate() samples from the learned distribution, producing 100 Monte‑Carlo trajectories per series.
  • The median of the samples is used for point‑forecast evaluation.

Evaluation Metrics

Using the evaluate library:

  • MASE (Mean Absolute Scaled Error) = 1.256 (average across 366 series).
  • sMAPE (Symmetric Mean Absolute Percentage Error) = 0.161. These figures outperform a wide range of classical and deep baselines on the same benchmark.

Benchmark Comparison

Model MASE
SES 3.306
Theta 1.649
TBATS 1.751
ETS 1.526
(DHR‑)ARIMA 1.589
PR 1.678
CatBoost 1.699
FFNN 1.582
DeepAR 1.409
N‑BEATS 1.574
WaveNet 1.482
Transformer (this work) 1.256

The Transformer achieves the lowest MASE without any dataset‑specific tuning, suggesting that global attention mechanisms can capture seasonality and trend patterns effectively.

Practical Takeaways

  • Global probabilistic forecasting can be implemented with a few lines of code using the 🤗 Transformers library.
  • The same API used for language models (generate, forward, loss) applies to time‑series data, lowering the barrier for practitioners.
  • Missing data handling is native via attention masks, eliminating the need for imputation.
  • Quadratic attention cost limits context length; future work may adopt efficient attention variants.

Next Steps for the Community

  • Multivariate extensions – support diagonal‑independent and full‑covariance distribution heads.
  • Time‑series classification – add classification heads for anomaly detection and other tasks.
  • Pre‑trained checkpoints – explore large‑scale pre‑training on heterogeneous time‑series corpora, analogous to NLP/vision.
  • Optional date‑time inputs – adapt the pipeline for datasets lacking explicit timestamps (e.g., neuroscience recordings).
  • Efficient attention – integrate sparse or linear‑complexity attention to enlarge feasible context windows.

The release demonstrates that vanilla Transformers, when paired with a proper probabilistic head and data pipeline, are competitive for univariate forecasting. Researchers and engineers are encouraged to experiment with other datasets from the Hugging Face Hub, adapt the frequency‑specific parameters, and contribute additional models to the library.

Sources