higgsfield – Distributed training made painless

What it is – higgsfield is an open‑source Python package that acts as a GPU‑node workload manager and a thin training framework for very large neural networks (billions‑to‑trillions of parameters). It plugs into the standard PyTorch ecosystem and supports the ZeRO‑3 DeepSpeed API as well as PyTorch’s Fully‑Sharded Data Parallel (FSDP) sharding, letting you run LLMs such as LLaMA‑70B across a cluster of machines.

---

Core capabilities (as described in the README)

Feature	What it does
Node allocation	Provides exclusive or shared access to compute nodes, letting users reserve GPUs for a training run.
Zero‑3 / FSDP support	Wraps DeepSpeed’s ZeRO‑3 and PyTorch FSDP so model parameters are sharded across GPUs, enabling trillion‑parameter models.
Experiment lifecycle	Simple Python decorator (@experiment) creates a run, launches it on the allocated nodes, and streams logs/metrics via a GitHub‑based UI.
Queue & contention handling	Maintains a job queue so multiple users can share a cluster without stepping on each other’s toes.
GitHub‑centric CI/CD	Generates GitHub Actions workflows that automatically deploy code to the cluster, run the experiment, and push checkpoints back to a hub.
Environment reproducibility	Installs Docker, drivers and the higgsfield binary on each node, avoiding “environment hell”.
Minimal config	No massive argument lists or YAML files; the framework supplies a tiny Python‑only interface for experiments.

---

Quick‑start installation

pip install higgsfield==0.0.3   # pulls the latest released package from PyPI

The package brings in the command‑line tools needed to register nodes, generate GitHub workflows, and launch experiments.

---

Minimal training example (from the README)

from higgsfield.llama import Llama70b
from higgsfield.loaders import LlamaLoader
from higgsfield.experiment import experiment
import torch.optim as optim
from alpaca import get_alpaca_data

@experiment("alpaca")
def train(params):
    # 70‑billion‑parameter LLaMA model, ZeRO‑3 sharding, bf16 precision
    model = Llama70b(zero_stage=3, fast_attn=False, precision="bf16")
    optimizer = optim.AdamW(model.parameters(), lr=1e-5)
    dataset = get_alpaca_data(split="train")
    train_loader = LlamaLoader(dataset, max_words=2048)

    for batch in train_loader:
        optimizer.zero_grad()
        loss = model(batch)
        loss.backward()
        optimizer.step()

    model.push_to_hub('alpaca-70b')   # saves the final checkpoint to the HuggingFace hub

Running this script on a machine that has been registered with higgsfield will:

1. Spin up the required number of GPU nodes (via the generated GitHub Action).
2. Allocate them to the experiment.
3. Execute the training loop.
4. Stream logs/metrics to the GitHub UI.
5. Push the final model checkpoint.

---

How the automation works

1. Node bootstrap – higgsfield installs Docker, the required drivers and its own binary on each node you point it at.
2. Workflow generation – It creates a GitHub Actions workflow that clones your repo, sets up the environment, and runs the experiment inside Docker.
3. GitHub‑driven deployment – Pushing a commit triggers the workflow, which automatically deploys the code to the allocated nodes.
4. UI & monitoring – The GitHub interface shows experiment status, logs, and lets you download checkpoints.

---

Supported environments

• OS: Ubuntu (any recent LTS)
• Access: SSH with a non‑root user that has password‑less sudo
• Clouds tested: Azure, LambdaLabs, FluidStack (any cloud that gives you raw Ubuntu VMs works as long as you can SSH in).

---

Documentation pointers (README links)

• Setup guide – setup.md walks through node registration, environment preparation, and first‑run deployment.
• Tutorial – tutorial.md covers distributed model handling, data loading, optimizer tricks, checkpointing, stability tricks, and monitoring.
• Support channels – GitHub Issues (response < 1 day), Twitter, and the project website.

---

Who would benefit?

• Researchers or engineers training LLMs that exceed a single‑GPU memory budget.
• Teams that already use GitHub for code collaboration and want training jobs to be launched automatically from PRs/commits.
• Anyone frustrated by managing multiple CUDA / PyTorch versions across a cluster; higgsfield’s Docker‑based approach isolates those dependencies.

---

Things to keep in mind

• The framework assumes you have your own GPU nodes (on‑prem or cloud VMs). It does not provide managed GPU instances.
• It currently targets Ubuntu only; other Linux distros would need manual adaptation.
• The README showcases LLaMA‑70B; while the API is generic, you’ll need to supply your own model classes for architectures not already wrapped.
• The package is at version 0.0.3, indicating early‑stage development; expect occasional breaking changes.

---

TL;DR

higgsfield is a lightweight orchestration layer that turns a set of Ubuntu GPU servers into a fault‑tolerant, queue‑aware training cluster for massive LLMs. By leveraging DeepSpeed ZeRO‑3 or PyTorch FSDP under the hood and wiring everything to GitHub Actions, it lets you launch, monitor, and checkpoint trillion‑parameter experiments with just a few lines of Python.