ReinFlow/ReinFlow

What it solves

ReinFlow addresses the difficulty of fine-tuning flow matching policies (which are often trained via imitation learning) using online reinforcement learning (RL). It specifically solves the problem of making policy probabilities tractable for gradient-based optimization, even when the policy uses very few denoising steps (e.g., 1 to 4 steps), making it robust to discretization and Monte Carlo approximation errors.

How it works

ReinFlow is a policy gradient framework that fine-tunes pre-trained flow policies. The process typically begins with training a flow policy using imitation learning (behavior cloning). ReinFlow then applies online RL to further optimize the policy. The core innovation is the training of an end-to-end noise injection network that allows the model to calculate policy probabilities accurately regardless of the number of denoising steps used.

Who it’s for

It is designed for researchers and developers working on robotic control and embodied AI, specifically those using flow-based policies (such as 1-Rectified Flow or Shortcut Models) for tasks like legged locomotion, state-based manipulation, and visual manipulation.

Highlights

• Broad Compatibility: Supports 1-Rectified Flow, Shortcut Models, and other ODE-defined policies.
• VLA Model Support: Capable of fine-tuning large Vision-Language-Action (VLA) models, including NVIDIA's GR00T and $\pi_0/\pi_{0.5}$.
• Efficiency: Enables effective RL fine-tuning with very few denoising steps (1, 2, or 4).
• Proven Performance: Demonstrated success across diverse robotic benchmarks including OpenAI Gym, Franka Kitchen, and Robomimic.