KarhouTam/FL-bench

What is FL‑bench?

FL‑bench is an open‑source benchmarking suite for federated learning (FL) research. It provides a ready‑to‑run framework that lets you train and evaluate dozens of classic and recent FL algorithms on a variety of image classification datasets. The code is organized around a common server‑client abstraction, so each method only needs to implement a few hooks while re‑using the bulk of the training pipeline.

Why it matters

Fast experimentation – Switch between algorithms (FedAvg, FedProx, SCAFFOLD, pFedSim, etc.) with a single command‑line flag.
Reproducible settings – All hyper‑parameters live in YAML config files; defaults are documented and can be overridden via CLI.
Scalable execution – Built‑in support for parallel training with Ray, allowing multi‑process or multi‑node runs without extra code.
Rich ecosystem – Includes many standard vision models (ResNet, EfficientNet, MobileNet, LeNet5…) and a wide range of datasets (MNIST, CIFAR‑10/100, FEMNIST, DomainNet, medical image sets, etc.) with configurable partition strategies.
Extensible – New FL methods, models, or datasets can be added by subclass‑ing the provided FedAvgServer/FedAvgClient base classes or by extending the dataset/model registries.

Quick start (one‑liner)

# Install dependencies
pip install -r .env/requirements.txt

# Generate a synthetic MNIST split for 100 clients
data/generate_data.py -d mnist -a 0.1 -cn 100

# Run the default FedAvg experiment
python main.py method=fedavg

The command reads the default configuration from config/defaults.yaml, launches a server that coordinates 100 clients, trains a ResNet (or any model you pick) on the partitioned MNIST data, and logs metrics to TensorBoard or Visdom.

How to customise

What	How
Algorithm	Change `method=` on the CLI or edit a YAML file (e.g., `method=fedprox fedprox.mu=0.01`).
Hyper‑parameters	Override any argument via CLI (`optimizer.lr=0.1`), a custom config file, or by editing the method’s `get_hyperparams()` function.
Model	Choose a built‑in model (`model: resnet18`) or create a `CustomModel` class in `src/utils/models.py` and reference it with `model: custom`.
Dataset / Partition	Pick a dataset from the list (MNIST, CIFAR‑10, DomainNet, etc.) and a partition strategy defined in `data/README.md`.
Parallelism	Set `mode: parallel` and `parallel.num_workers` in the config; Ray will spin up workers automatically.
Monitoring	Set `common.monitor` to `tensorboard` or `visdom`, then launch the corresponding server (`tensorboard --logdir=…` or `python -m visdom.server`).

Adding a new FL method (developer tip)

Create server and client classes that inherit from FedAvgServer / FedAvgClient (or another existing method if you need its utilities).
Override any of the following hooks as needed:
- package() – what parameters are sent between server and client.
- aggregate_client_updates() – custom aggregation logic on the server.
- fit(), set_parameters() – client‑side training adjustments.
Register the new method by adding a .py file under src/server/ (the filename becomes the method name used on the CLI).
Expose hyper‑parameters via a static get_hyperparams() that returns an argparse.Namespace – these can be set in config files.

License & citation

The project is released under GPL‑3.0. If you use FL‑bench in a paper, cite the provided BibTeX entries:

@software{Tan_FL-bench,
  author = {Tan, Jiahao and Wang, Xinpeng},
  license = {GPL-3.0},
  title = {{FL-bench: A federated learning benchmark for solving image classification tasks}},
  url = {https://github.com/KarhouTam/FL-bench}
}

Bottom line

FL‑bench is a practical, well‑documented platform for anyone who wants to compare federated learning algorithms on realistic image data, experiment with personalization or domain‑generalization extensions, and scale up runs with Ray. Its modular design makes it easy to plug in new methods, models, or datasets while keeping experiments reproducible.