FilippoAiraldi/mpc-reinforcement-learning

Reinforcement Learning with Model Predictive Control

What it solves

This library provides a framework for training model-based Reinforcement Learning (RL) agents where Model Predictive Control (MPC) serves as the function approximation. It merges MPC's ability to predict future environment behavior and compute optimal actions with RL's adaptability to unknown and complex environments, allowing the agent to learn optimal control policies in systems where the underlying dynamics or costs are not fully known.

How it works

The framework uses an MPC controller that is parametrized in its objective, predictive model, and constraints. This controller acts as both the policy provider (generating actions) and the function approximator for state and state-action value functions. An RL algorithm—such as Q-learning or Deterministic Policy Gradient (DPG)—is then used to tune these MPC parameters to increase performance. The library leverages csnlp to compute the sensitivities of the MPC controller relative to its parameters, which are essential for calculating RL updates.

Who it’s for

Researchers and engineers working in optimal control and reinforcement learning, specifically those dealing with continuous state and action spaces in physical or simulated systems (e.g., Linear Time Invariant systems).

Highlights

  • Hybrid Approach: Combines MPC and RL into a single data-driven control technique.
  • Multiple RL Algorithms: Supports gradient-based methods like on-policy and off-policy Q-learning and DPG, as well as gradient-free methods like Bayesian Optimization.
  • Integration: Works with gymnasium environments and leverages csnlp for sensitivity analysis.
  • Parametrized Control: Allows the tuning of predictive models, objectives, and constraints through RL learning agents.

Related

  • Project
    AgentR1/Agent-R1

    A modular framework for Agentic Reinforcement Learning that trains multi-step LLM agents by treating each interaction turn as a step-level MDP transition.

  • Project
    nicklashansen/tdmpc2

    A scalable, robust model-based reinforcement learning algorithm for continuous control that can handle multiple tasks across different domains and embodiments using a single agent.

  • Project
    facebookresearch/Pearl

    A production-ready Reinforcement Learning library from Meta for building modular AI agents capable of handling complex, real-world environments with sparse feedback and dynamic action spaces.

  • Dispatch
    Hugging Face Deep Reinforcement Learning Introduction

    Hugging Face released an introductory tutorial for a free Deep Reinforcement Learning class, covering core RL concepts, the RL framework, exploration‑exploitation trade‑off, policy‑based vs value‑based methods, and how deep neural networks integrate into RL.

  • Project
    OpenManus/OpenManus-RL

    An open-source framework for optimizing LLM agents using reinforcement learning to enhance their reasoning, tool use, and decision-making capabilities.