Graph Classification with Transformers

Hugging Face has provided a technical guide on implementing graph classification using the Transformers library. The primary model currently available for this task within the library is Microsoft's Graphormer, which allows users to apply transformer-based architectures to graph-structured data for tasks such as binary classification, multi-class classification, or regression.

Graph Data Formatting and Loading

Graph datasets on the Hugging Face Hub are primarily stored in jsonl format as lists of graphs. Each graph is represented as a dictionary with the following required and optional fields:

  • edge_index: A list containing two parallel lists of integers representing the indices of nodes in edges (e.g., [[1, 1, 3], [2, 3, 1]] for edges 1–2, 1〓3, and 3–1).
  • num_nodes: An integer indicating the total number of nodes in the graph, ensuring that isolated nodes are accounted for.
  • y: The target value for prediction, which can be an integer for multi-class classification, a float for regression, or a list of binary labels for multi-task classification.
  • node_feat (Optional): A list of lists of integers containing features for each node, ordered by node index.
  • edge_attr (Optional): A list of lists of integers containing attributes for each edge, following the edge_index ordering.

Users can load these datasets directly from the Hub using the datasets library, such as the ogbg-molhiv dataset from the Open Graph Benchmark (OGB).

Preprocessing for Graphormer

Graph transformer frameworks require specific preprocessing to generate features that assist the learning process. For Graphormer, this includes generating shortest path matrices between nodes and in/out degree information.

Preprocessing can be handled in two ways:

  1. Static Preprocessing: Using dataset.map(preprocess_item, batched=False) to process the data before training.
  2. On-the-fly Processing: Setting on_the_fly_processing=True within the GraphormerDataCollator to process data during training, which is more memory-efficient for large graphs.

Model Implementation and Fine-Tuning

Graph classification is implemented using the GraphormerForGraphClassification class. The workflow involves loading a pretrained checkpoint and adapting the classification head to the specific downstream task.

Loading and Configuration

When loading a model via from_pretrained, users can specify num_classes to match their specific task (e.g., num_classes=2 for binary classification). By setting ignore_mismatched_sizes=True, the library creates a custom classification head, replacing the original decoder head from the pretrained checkpoint.

Training Workflow

The training process is managed via the Trainer API. Key technical considerations include:

  • Data Collator: A GraphormerDataCollator must be passed to the Trainer to convert individual graphs into batches.
  • Memory Management: Due to the model size, adjusting per_device_train_batch_size and gradient_accumulation_steps is critical to prevent out-of-memory (OOM) errors.
  • Hardware: While training is possible on CPU, the guide notes that 20 epochs can take approximately one day on an Intel Core i7, making GPU acceleration highly recommended for practical use.

Sources