Federated Learning with Hugging Face and Flower

Hugging Face has provided a technical guide on integrating the Flower framework to enable federated learning (FL) for Transformer models. This integration allows for the fine-tuning of pre-trained models across multiple clients without the need to share raw data, enhancing privacy and data security.

Federated Learning Architecture with Flower

Federated Learning enables the training of a global model across multiple decentralized clients and a central server. Instead of aggregating raw data in a single location, each client trains the model locally on its own data and transmits only the model parameters back to the server. The server then aggregates these parameters using a predefined strategy to update the global model.

In the provided implementation, the process follows these steps:

  1. Local Training: Clients perform local training using their own datasets.
  2. Parameter Exchange: Clients send their updated parameters to the server via the get_parameters method.
  3. Global Aggregation: The server aggregates parameters from all clients using a strategy such as FedAvg (Federated Averaging), which defines global weights as the average of all client weights for each round.
  4. Model Distribution: The server sends the updated global parameters back to the clients via the set_parameters method.

Technical Implementation Details

Model and Dataset

The implementation uses distilBERT (distilbert-base-uncased) as the base model, loaded via Hugging Face's AutoModelForSequenceClassification for a binary sequence classification task. The target task is sentiment analysis on the IMDB dataset, where the model is trained to detect whether a movie rating is positive or negative.

The Hugging Face Workflow

The standard Hugging Face pipeline is used for data preparation and training:

  • Data Handling: The datasets library is used to fetch the IMDB dataset, which is then tokenized using AutoTokenizer and loaded into PyTorch DataLoader objects.
  • Training Loop: A standard PyTorch training loop is implemented using the AdamW optimizer.
  • Evaluation: The evaluate library is used to calculate accuracy and loss metrics during the testing phase.

The Flower Client (IMDBClient)

To bridge the Hugging Face model with the Flower framework, a custom client class is created by inheriting from flwr.client.NumPyClient. This class implements four critical methods:

  • get_parameters: Extracts model parameters as NumPy arrays for transmission to the server.
  • set_parameters: Updates the local model's state dictionary with parameters received from the server.
  • fit: Executes the local training function (train) and returns the updated parameters and the number of examples used.
  • evaluate: Runs the local test function (test) and returns the loss and accuracy metrics.

Server Configuration and Aggregation

To coordinate the federated process, a Flower server is initialized with a specific aggregation strategy. The implementation uses fl.server.strategy.FedAvg, configured with fraction_fit=1.0 and fraction_evaluate=1.0, meaning all clients participate in every round of training and evaluation.

To handle the distributed metrics, a weighted_average function is implemented. This function calculates the global accuracy and loss by weighting the metrics from each client based on the number of examples they contributed, ensuring a representative global performance measure.

Framework Compatibility

While the example provided utilizes PyTorch, the guide notes that the same federated learning workflow can be implemented using TensorFlow. The simulation functionality of Flower (flwr['simulation']) can also be used to emulate a federated environment within a single environment, such as Google Colab, for testing purposes.

Sources