OpenDriveLab/WorldEngine

WorldEngine – Post‑Training Platform for Autonomous Driving

What it is – An open‑source framework that lets you take a pre‑trained autonomous‑driving model, discover the rare, safety‑critical situations it fails on, turn those moments into photorealistic, fully controllable simulations, and then fine‑tune the model with reinforcement learning on the generated rollouts. The goal is to close the long‑tail data gap that plagues real‑world AD systems.

Key components

Module	Role	Core tech
SimEngine	Closed‑loop simulation where the ego vehicle and surrounding agents interact in real time.	3‑D Gaussian Splatting (MTGS) for photorealistic rendering, Hydra for config, Ray for distributed execution
AlgEngine	End‑to‑end training, evaluation, rare‑case extraction and RL‑based post‑training.	MMDetection3D, UniAD / VADv2 back‑ends, HydraMDP for RL
Scene Reconstruction (MTGS)	Converts raw nuPlan logs into multi‑traversal Gaussian‑splatting scenes that SimEngine can render.	Multi‑Traversal Gaussian Splatting (MTGS)

How it works

1. Run a base model on a large driving dataset (e.g., nuPlan).
2. Automatic long‑tail discovery – the pre‑trained agent flags failure‑prone logs without human labeling.
3. Reconstruct scenes with MTGS, producing a real‑time, photorealistic simulation environment for each discovered scenario.
4. Behavior World Model (BWM) (future integration) can synthesize variations of these scenarios, expanding the rare‑case distribution.
5. RL post‑training – the model is fine‑tuned on the generated rollouts, dramatically improving closed‑loop success on rare cases.

Results highlighted in the README

• Post‑training with WorldEngine raises rare‑case closed‑loop success from ~73 % (base) to 88.9 %, and the PDMS* score from 60.98 % to 70.12 %.
• In a 200 km night‑time on‑road test on a mass‑produced ADAS platform, the system achieved zero disengagements.
• The pipeline matches the benefit of a ~10× increase in pre‑training data while using far fewer real logs.

Getting started

1. Clone the repo and follow the two‑environment conda setup described in docs/installation.md.
2. Run the quick‑test script:

   export WORLDENGINE_ROOT=$(pwd)
   bash scripts/closed_loop_test.sh   # single‑GPU
   # or
   bash scripts/multigpu_closed_loop_test.sh   # default 8 GPUs
   

   This loads a pre‑trained VADv2 checkpoint, runs 288 rare‑case rollouts in SimEngine, and reports PDMS, success rate, and ego progress.
3. Dive deeper via the module‑specific guides (simengine_usage.md, algengine_usage.md).

Who should use it

• Researchers building end‑to‑end autonomous‑driving stacks who need a systematic way to improve rare‑case performance.
• Companies that already have large driving logs and want to extract the most valuable failure scenarios automatically.
• Anyone interested in photorealistic, distributed closed‑loop simulation for AD (the 3‑D Gaussian‑splatting scenes are reusable outside the RL pipeline).

Resources

• Dataset – released on Hugging Face and ModelScope (WorldEngine dataset).
• Paper – World Engine: Towards the Era of Post‑Training for Autonomous Driving (arXiv:2606.19836).
• Related code – MTGS (scene reconstruction), SimScale (large‑scale simulation), UniAD/VADv2 (model back‑ends).

License – Apache‑2.0 for code; the released data follows the CC‑BY‑NC‑SA 4.0 license of nuPlan.

---

All statements are taken directly from the repository’s README.