YYHDBL/shopping-grpo-longhorizon

面向长程购物 Agent 的可复现后训练

What it solves

This project provides a reproducible pipeline for training and evaluating AI agents designed for long-horizon shopping tasks. It addresses the challenge of building agents that can reliably navigate complex e-commerce environments—performing tasks like searching, comparing product specifications, and making final purchases based on specific user constraints.

How it works

The project uses a multi-stage training pipeline:

  1. Data Collection: It uses a teacher model to collect successful interaction trajectories within the ShopSimulator environment.
  2. Supervised Fine-Tuning (SFT): It performs LoRA-based fine-tuning on these trajectories to teach the model valid tool-use strategies.
  3. GRPO Training: It employs the veRL library to further optimize the model using Group Relative Policy Optimization (GRPO), which allows the agent to learn from its own environment rollouts without needing a separate reward model.
  4. Evaluation: It uses a rigorous, multi-faceted evaluation pipeline that combines deterministic code-based checks with LLM-as-a-Judge scoring to assess performance across five dimensions, including search strategy and decision quality.

Who it’s for

This project is for researchers and developers working on agentic AI, specifically those focused on long-horizon task planning, tool-use, and reinforcement learning from environment feedback (RLHF/GRPO) in simulated real-world scenarios.

Highlights

  • End-to-End Pipeline: Includes everything from data collection and SFT to GRPO training and a comprehensive evaluation suite.
  • ShopSimulator Integration: Comes with an embedded version of the ShopSimulator v2.1 environment for consistent testing.
  • Deterministic Reward System: Uses a rule-based Reward v3 system to evaluate outcomes objectively, avoiding the subjectivity of pure LLM-based reward modeling.
  • Reproducible Benchmarking: Provides a "Final-200 Clean" benchmark set to ensure fair and consistent comparison between different model iterations.

Related

  • Project
    shibing624/MedicalGPT

    A comprehensive training pipeline for medical-domain LLMs that implements pre-training, supervised fine-tuning, and multiple preference alignment methods like RLHF, DPO, and GRPO.

  • Dispatch
    LFM2.5-350M GRPO Fine-tuning Boosts IFStruct Score to 29.7%

    Fine‑tuning the 350M‑parameter LFM2.5 model with Group Relative Policy Optimization (GRPO) for just 100 steps raises its IFStruct benchmark score from 22.6% to 29.7%, demonstrating that inexpensive task‑specific reward training can markedly improve structured‑output compliance.

  • Project
    opendilab/LightRFT

    LightRFT is an open‑source, distributed reinforcement‑learning framework for fine‑tuning large language and multimodal generative models. It unifies training back‑ends (DeepSpeed, FSDP) with rollout engines (SGLang, vLLM), supports a range of RL algorithms (GRPO, REINFORCE++, DAPO, etc.), and works across text, image, video, and audio modalities. The library provides ready‑to‑run examples (e.g., GSM8K with Qwen2.5), extensive docs, and Docker images, making it suitable for researchers and engineers building RL‑based alignment pipelines.

  • Project
    huggingface/open-r1

    An open-source reproduction of the DeepSeek-R1 pipeline, providing tools for SFT, GRPO reinforcement learning, and synthetic data generation to create reasoning models.

  • Project
    AgentR1/Agent-R1

    A modular framework for Agentic Reinforcement Learning that trains multi-step LLM agents by treating each interaction turn as a step-level MDP transition.