HorizonRobotics/alf

Agent Learning Framework https://alf.readthedocs.io

What is ALF?

ALF (Agent Learning Framework) is an open‑source reinforcement‑learning (RL) library built on PyTorch. It provides a modular, configurable code base for researchers and engineers to prototype, train, and evaluate a wide variety of RL algorithms—from classic on‑policy methods like A2C and PPO to modern off‑policy and model‑based approaches such as SAC, MuZero, and RLPD. The framework emphasizes flexibility (you can mix and match components such as networks, loss functions, and data transforms) and ease of use (configuration is done with plain Python files rather than a separate DSL).

Key Features (as described in the README)

Feature Details
Broad algorithm coverage Implements dozens of algorithms, including A2C, PPO, PPG, DDQN, DDPG, SAC, RLPD, MuZero, BC, IQL, DIAYN, ICM, RND, and many more. Each algorithm lives in its own module (e.g., alf/algorithms/ppo_algorithm.py).
Modular design Components such as actors, critics, replay buffers, and data transformers are separate, making it easy to replace or extend parts of an algorithm.
PyTorch‑native Uses torch.utils.cpp_extension for fast parallel environments and supports distributed training via PyTorch’s torchrun.
Configuration system Jobs are configured with Python‑based “conf” files (*_conf.py) that can express arithmetic and conditional logic, superseding the older gin‑based configs.
Multi‑GPU / multi‑node support Single‑node multi‑GPU via the --distributed multi-gpu flag; multi‑node training using PyTorch’s torchrun launcher with NCCL settings.
Extensive documentation & tutorials Online docs at https://alf.readthedocs.io/ and a draft tutorial page; the README lists many example scripts.
Ready‑to‑run examples Over a dozen ready‑made example configs for classic control (CartPole), Atari, MuJoCo tasks (Fetch, Humanoid), robotics simulators (MetaDrive, SocialRobot), and even game‑like environments (Super Mario).
Docker & Nix support Official Docker image (horizonrobotics/cuda:11.8.0-py3.11‑torch2.2‑ubuntu22.04) and a Nix flake for reproducible development environments.
Installation flexibility Supports manual pip install -e ., the modern uv tool for dependency management, or a Nix environment.
Continuous integration CI badge shows tests run on the pytorch branch.

Who Might Use It?

  • Researchers exploring new RL ideas who need a solid, well‑tested baseline implementation of many algorithms.
  • Robotics engineers who want to train policies in simulation (e.g., PyBullet, MuJoCo, MetaDrive) and then export them.
  • Practitioners looking for a single library that can handle on‑policy, off‑policy, offline, and model‑based RL without switching frameworks.

Getting Started (quick steps)

  1. Clone & set up
    git clone https://github.com/HorizonRobotics/alf
cd alf
# using uv (recommended)
curl -Ls https://astral.sh/uv/install.sh | sh
uv sync
  2. Run an example (e.g., CartPole with A2C)
    uv run python -m alf.bin.train --conf=alf/examples/ac_cart_pole.gin --root_dir=./logs/cartpole
  3. Monitor with TensorBoard:
    tensorboard --logdir=./logs/cartpole
  4. Scale up – add --distributed multi-gpu or use the torchrun command shown in the README for multi‑node training.

What the README doesn't say

  • No mention of a stable 1.0 release; the project appears actively maintained (CI badge, recent Docker image).
  • No built‑in support for large‑scale language‑model RL (e.g., RLHF), but the modularity would allow adding custom components.
  • The tutorial is still a draft, so newcomers may need to rely on the example configs and source code.

Bottom line: ALF is a genuine, actively maintained reinforcement‑learning framework that covers a wide spectrum of algorithms and provides the tooling needed for both research experiments and more applied robotics/embodied‑AI projects.

Related

  • Project
    yihaosun1124/OfflineRL-Kit

    A PyTorch library for offline reinforcement learning that provides a standardized framework and implementations of state-of-the-art model-free and model-based algorithms.

  • Project
    FareedKhan-dev/train-llm-from-scratch

    A complete from-scratch PyTorch implementation of an LLM pipeline, covering everything from raw text tokenization and Transformer architecture to SFT and advanced RLHF alignment techniques like PPO and GRPO.

  • Project
    meta-pytorch/torchforge

    A PyTorch-native agentic RL library that separates infrastructure concerns from model concerns to simplify scalable reinforcement learning experimentation.

  • Project
    AgileRL/AgileRL

    AgileRL is a Python library for deep reinforcement learning that adds RLOps‑style tooling. It bundles many RL algorithms and provides evolutionary hyper‑parameter optimisation, population‑based distributed training, multi‑agent support, offline/bandit methods, and a cloud platform (Arena) for scaling experiments. Install via `pip install agilerl` (optional extras for Box2D, Arena, LLM fine‑tuning). Licensed under Apache‑2.0.

  • Project
    kozistr/pytorch_optimizer

    A PyTorch‑compatible library that aggregates 100+ research optimizers, several LR schedulers and loss functions behind a single, consistent API, with optional integrations for low‑precision training.