Ecom-RLVE: Adaptive Verifiable Environments for E-Commerce Conversational Agents

Hugging Face has announced Ecom-RLVE, a framework designed to improve the task completion rates of e-commerce conversational agents. By extending the RLVE (Reinforcement Learning with Verifiable Environments) framework to multi-turn, tool-augmented conversations, EcomRLVE-GYM provides a set of verifiable environments where agents are optimized for actual outcomes—such as correct product selection and cart accuracy—rather than just conversational fluency.

Verifiable Rewards vs. LLM-as-a-Judge

Ecom-RLVE addresses the gap where large language models (LLMs) can be fluent in conversation but fail to complete complex transactional tasks. To solve this is, the framework utilizes Reinforcement Learning with Verifiable Rewards (RLVR), which removes the subjectivity of "LLM-as-a-judge" by using algorithmic verification.

Every outcome is evaluated by a program with access to the ground-truth goal. The reward signal is composed of three primary components:

• Task Reward: Measures whether the agent successfully completed the goal (e.g., correct product recommended or cart accuracy).
• Efficiency Reward: Penalizes wasted turns caused by agent mistakes, while ignoring turns caused by user follow-ups.
• Hallucination Penalty: Penalizes the agent for recommending product IDs that were not actually retrieved during the session.

Invalid outputs, such as malformed JSON or illegal tool calls, result in immediate failure scores to incentivize well-formed responses.

The Eight EcomRLVE-GYM Environments

EcomRLVE-GYM consists of eight distinct environments that simulate real-world shopping scenarios. Each requires the agent to use tools—such as catalog search, cart operations, and order lookups—to satisfy a user request.

Environment	Agent Objective
Product Discovery	Find products satisfying all user constraints
Substitution	Find a compatible alternative for an out-of-stock item
Cart Building	Add exact products, variants, and quantities to the cart
Return + Replacement	Identify the correct order line, initiate return, and suggest replacement
Order Tracking	Resolve the intended order and report its status
Policy QA	Answer deterministic questions about store policies
Bundle Planning	Recommend a complete shopping list within a budget
Multi-Intent Journey	Chain 2–5 of the above tasks in a single sequence

Adaptive Difficulty Curriculum

To prevent training saturation or starvation, Ecom-RLVE employs an adaptive difficulty curriculum. A single difficulty value (d) controls 12 independent axes of task complexity.

Representative difficulty axes include:

• Constraint Volume: The number of user constraints increases from 2 (at d=0) to 8 (at d=12).
• Constraint Omission: The frequency with which users omit constraints increases from 5% to ~80%, forcing the agent to ask clarifying questions.
• Distractor Ratio: The fraction of search results that are distractors increases from 0% to 24%.
• Stock Volatility: The frequency of items going out of stock mid-conversation increases from 0% to 50%.

Each environment tracks the agent's success rate independently, advancing the difficulty only when the agent reliably passes the current level.

Deep Dive: Cart Building (E_CART)

The Cart Building environment highlights the necessity of variant selection. Because real catalogs are sparse, the framework synthesizes variants (e.g., connector types for electronics, sizes for clothing) at episode initialization.

To succeed in E_CART, the agent must master five skills: product discovery, variant selection, cart management, clarification dialogue, and handling multi-item orders. It utilizes six tools, including catalog_search, catalog_get_variants, cart_add, cart_view, user_get_visit_history, and ask_user.

User Simulation and Scaling

Ecom-RLVE uses Qwen3.5 (9.7B) as a user simulator to generate natural, varied messages, including typos and topic switches. The simulator ensures that user preferences match stated constraints and strategically omits information to force agent clarification.

Environment scaling follows a nested structure (C1 ⊂ C2 ⊂ C4 ⊂ C8), where agents trained on the full suite (C8) are hypothesized to outperform specialists trained on only one environment.

Early Training Results

In a viability study, a Qwen 3 8B model was trained using DAPO over 300 steps on the Cart Building (C1) environment. The results demonstrated progressive growth in the difficulty levels reached, confirming that adaptive scheduling provides a steady learning signal for agentic task completion.