HewlettPackard/swarm-learning
A simplified library for decentralized, privacy preserving machine learning
What it solves
Swarm Learning addresses the challenge of training machine learning models on distributed data sources without compromising data privacy. It eliminates the need to centralize raw data in a single location, which is often impossible due to security, privacy regulations, or the sheer volume of data at the edge.
How it works
The framework uses a decentralized architecture where training occurs locally at the edge (near the data source). Instead of sharing raw data, nodes share only the learned insights (model weights) with their peers.
To coordinate this process, the system employs several modular components:
- SL Nodes: Handle the distribution and merging of model weights between peers.
- SN Nodes: Use a blockchain platform (Ethereum) to track training progress and maintain a shared state of metadata, ensuring coordination without a central authority.
- SWOP Nodes: Act as agents to manage operations like starting/stopping runs and upgrading containers.
- SWCI/SLM-UI: Provide command-line and graphical interfaces for monitoring and managing the framework.
Users can integrate existing Keras, PyTorch, or HuggingFace Trainer models by adding a SwarmCallback API to their training code.
Who it’s for
It is designed for organizations or researchers who need to collaborate on ML model training across multiple distributed data owners while keeping raw data strictly local and secure.
Highlights
- Privacy-Preserving: Only model insights are shared, never raw data.
- Decentralized Coordination: Uses blockchain for metadata and progress tracking instead of a central server.
- ** uma-modular Architecture**: Components run in separate containers for flexibility.
- Broad Framework Support: Compatible with PyTorch, Keras, and HuggingFace Trainer.
- Secure Communication: Uses X.509 certificates for identity and secure communication between nodes.