YYHDBL/shopping-grpo-longhorizon

What it solves

This project provides a reproducible post-training and evaluation pipeline for long-horizon shopping agents. It addresses the challenge of training LLMs to handle complex, multi-step shopping tasks that require instruction understanding, tool use, long-context management, and constraint satisfaction (such as budget, brand, and specific product specifications).

How it works

The project implements a continuous post-training pipeline consisting of three main stages:

1. SFT (Supervised Fine-Tuning): The model learns legal and complete shopping behaviors from high-quality trajectories collected by a teacher model (DeepSeek-V4-Flash) within the ShopSimulator environment.
2. GRPO (Group Relative Policy Optimization): Using the veRL framework, the model is further optimized through online rollouts in the ShopSimulator environment. It is guided by a deterministic "Reward v3" system that evaluates final purchase results and constraint satisfaction without needing an external LLM-as-a-Judge for training.
3. Evaluation: Models are tested on a "Final-200 Clean" benchmark. The evaluation uses a combination of hard-coded checks and two specialized LLM judges (DeepSeek V4 Flash for rubric curation and DeepSeek V4 Pro for trajectory judging) to provide a multi-dimensional score on search strategy, decision quality, and efficiency.

Who it’s for

Researchers and developers working on LLM agents, reinforcement learning from human/environmental feedback, and long-horizon task planning in e-commerce settings.

Highlights

• Integrated Environment: Includes the ShopSimulator v2.1 environment and product data directly in the repository.
• Deterministic Reward System: Uses a weighted reward system (Reward v3) for categories, budget, brand, and specifications to avoid subjective judge bias during training.
• Multi-Stage Pipeline: A clear path from teacher trajectory collection $\rightarrow$ LoRA SFT $\rightarrow$ online GRPO $\rightarrow$ audited evaluation.
• Comprehensive Evaluation: A rigorous evaluation pipeline that isolates the judge from the gold answers to prevent answer-leakage during scoring.