PatchTST Integration in Hugging Face

Hugging Face has integrated PatchTST, a Transformer-based model designed for long-term time series forecasting. By segmenting time series into subseries-level patches and employing a channel-independent architecture, PatchTST reduces computational overhead while retaining local semantic information and extending the model's historical context window.

PatchTST Architecture and Core Mechanisms

PatchTST improves upon traditional Transformer applications in time series forecasting through two primary technical components:

Time Series Patching

Instead of treating individual time points as tokens, PatchTST vectorizes time series into patches of a given size. This design provides three specific advantages:

  • Local Semantic Retention: Local patterns are preserved within the embedding.
  • Computational Efficiency: The memory and computation requirements for attention maps are reduced quadratically relative to the look-back window due to the use of strides between patches.
  • Extended Context: The model can attend to longer historical sequences by balancing patch length (vector size) and context length (number of sequences).

Channel-Independence

PatchTST treats each channel as a single univariate time series. All univariate series share the same embedding and Transformer weights, effectively operating as a global univariate model.

Model Capabilities and Training Strategies

PatchTST features a modular design that supports both supervised forecasting and self-supervised masked time series pre-training.

Direct Forecasting

In a supervised setting, the model uses patched vectors to output a forecast for a specific prediction length. When trained on the Electricity dataset, the model achieved a Mean Squared Error (MSE) of 0.131, aligning with results reported in the original research paper.

Transfer Learning and Zero-Shot Performance

PatchTST demonstrates strong transfer learning capabilities, allowing a model pre-trained on one source domain to be applied to a target domain. In tests transferring from the Electricity dataset to the ETTh1 dataset, the following performance tiers were observed:

  • Zero-Shot Forecasting: Evaluating the pre-trained model on the target domain without any additional training resulted in an MSE of 0.370.
  • Linear Probing: Training only a linear layer on top of a frozen pre-trained backbone reduced the MSE to 0.357.
  • Full Fine-Tuning: Performing a full model fine-tune on the target data resulted in an MSE of 0.354.

Implementation and Integration

PatchTST is available via the Hugging Face Transformers library. For data pre-processing, the implementation utilizes the IBM tsfm package.

Key configuration parameters for the PatchTSTConfig include:

  • context_length: The amount of historical data used as input.
  • patch_length: The length of patches extracted from the context window.
  • patch_stride: The stride used when extracting patches.
  • d_model: The dimension of the transformer layers.
  • num_attention_heads: The number of attention heads per layer.

Sources