LucasAlegre/morl-baselines

Multi-Objective Reinforcement Learning algorithms implementations.

MORL‑Baselines – Multi‑Objective Reinforcement‑Learning library

What it is – A Python package (PyTorch) that implements a growing collection of state‑of‑the‑art multi‑objective reinforcement‑learning (MORL) algorithms. It follows the MO‑Gymnasium environment API, which is a drop‑in replacement for Gymnasium where the reward is returned as a vector (one entry per objective).

Why it matters – MORL tackles problems where an agent must balance several, often conflicting, goals (e.g., speed vs. safety, energy use vs. performance). Researchers need reliable reference implementations to compare new ideas, and MORL‑Baselines provides a curated, tested, and benchmarked set of such implementations.

Core features (as listed in the README)

  • Single‑ and multi‑policy algorithms covering both Scalarized Expected Return (SER) and Expected Scalarized Return (ESR) criteria.
  • Strict MO‑Gymnasium compliance – you can swap any MO‑Gymnasium environment without code changes.
  • Automatic experiment logging to Weights & Biases dashboards for reproducible performance tracking.
  • Pre‑commit hooks enforce black formatting, isort imports and linting.
  • Utility modules (Pareto pruning, replay buffers, neural‑net helpers) to avoid boiler‑plate.
  • Hyper‑parameter optimisation support out‑of‑the‑box.
  • Full test suite and continuous‑integration badge.

Implemented algorithms (snapshot)

Algorithm Policy type Criterion Observation / Action space Reference
GPI‑LS / GPI‑PD Multi SER Continuous obs, Discrete/Continuous actions [arXiv:2301.07784]
GPI‑LS (Jax) Multi SER Continuous [Springer link]
MORL/D Multi SER / ESR Discrete / Continuous [JAIR 2022]
Envelope Q‑Learning Multi SER Continuous [arXiv:1908.08342]
CAPQL Multi SER Continuous [OpenReview]
PGMORL Multi SER Continuous [MIT paper]
Pareto Conditioned Networks (PCN) Multi SER/ESR* Continuous [AAMAS 2022]
Lorenz Conditioned Networks (LCN) Multi SER/ESR* Continuous [JAIR]
Pareto Q‑Learning Multi SER Discrete [JMLR 2015]
MO Q‑Learning Single SER Discrete [ResearchGate]
MPMOQL (outer‑loop MOQL) Multi SER Discrete [ResearchGate]
Optimistic Linear Support (OLS) Multi SER – Thesis chapter
Expected Utility Policy Gradient (EUPG) Single ESR Discrete [ResearchGate]
Iterated Pareto Referent Optimisation (IPRO / IPRO‑2D) Multi SER Continuous [arXiv:2402.07182]
Non‑Linear Multi‑Objective PPO (NLMOPPO) Single SER Continuous Derived from [VUB paper]

*PCN assumes deterministic transitions; IPRO‑2D works only for exactly two objectives.

Benchmarking & reproducibility

  • Integrated with the Open RL Benchmark suite, allowing side‑by‑side comparison with CleanRL, Stable‑Baselines3, etc.
  • Experiment results are publicly available on a Weights & Biases project: https://wandb.ai/openrlbenchmark/MORL-Baselines.
  • Detailed protocol and settings are documented in the repo’s issue tracker (#43) and on the project website.

Getting started (quick‑start)

# clone and install (requires Python ≥3.9)
git clone https://github.com/LucasAlegre/morl-baselines.git
cd morl-baselines
pip install -e .   # installs the package and its deps

Run an example (e.g., GPI‑LS on a continuous‑control MO‑Gymnasium env):

python -m examples.run_gpi_ls --env mo_gymnasium:HalfCheetah-v0 --weights-and-biases

A Colab notebook is linked in the README for an interactive demo.

Community & contribution

  • Discord for discussion: https://discord.gg/ygmkfnBvKA
  • Open to pull requests – new algorithms, bug fixes, or documentation improvements are welcomed.
  • Maintainers: Florian Felten (ffelten) and Lucas N. Alegre (LucasAlegre).

License & citation

  • MIT License (permissive, commercial‑friendly).
  • If you use the library in research, cite the NeurIPS‑2023 paper:
@inproceedings{felten_toolkit_2023,
  author = {Felten, Florian and Alegre, Lucas N. and Nowé, Ann and Bazzan, Ana L. C. and Talbi, El Ghazali and Danoy, Grégoire and Silva, Bruno Castro da},
  title = {A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning},
  booktitle = {NeurIPS 2023},
  year = {2023}
}

TL;DR

MORL‑Baselines is a well‑maintained, test‑covered PyTorch library that gives researchers ready‑to‑run implementations of dozens of multi‑objective RL algorithms, all compatible with the MO‑Gymnasium API and integrated with reproducible benchmarking tools.

Related

  • Project
    pytorch/rl

    TorchRL is a PyTorch‑native, modular reinforcement‑learning library that provides a unified TensorDict data model and interchangeable components (environments, collectors, replay buffers, loss modules, trainers, etc.) for building, scaling, and experimenting with RL, multi‑agent, model‑based, and LLM post‑training pipelines.

  • Project
    AgileRL/AgileRL

    AgileRL is a Python library for deep reinforcement learning that adds RLOps‑style tooling. It bundles many RL algorithms and provides evolutionary hyper‑parameter optimisation, population‑based distributed training, multi‑agent support, offline/bandit methods, and a cloud platform (Arena) for scaling experiments. Install via `pip install agilerl` (optional extras for Box2D, Arena, LLM fine‑tuning). Licensed under Apache‑2.0.

  • Project
    DLR-RM/stable-baselines3

    A set of reliable PyTorch implementations of reinforcement learning algorithms designed to provide stable baselines for research and industry.

  • Project
    Farama-Foundation/MO-Gymnasium

    A Python library providing a standardized API and benchmark environments for developing and comparing multi-objective reinforcement learning algorithms.

  • Project
    kengz/SLM-Lab

    A modular PyTorch framework for Deep Reinforcement Learning that allows users to configure and run RL experiments via JSON specs across various Gymnasium environments.