ZhiChengAIR/Chengling-PWM

What it solves

Chengling is a Physical World Model (PWM) designed for robots to learn how the physical world behaves. It allows a robot to predict the consequences of its actions and mentally simulate outcomes based on raw sensory inputs (vision and proprioception) and language prompts before executing them in the real world.

How it works

Chengling uses an Energy-Based Transformer (EBT) architecture. It processes multi-view RGB observations through a frozen vision encoder and fuses them with action history and language prompts via a Transformer decoder.

Instead of generating actions in a single pass, it uses an energy function to evaluate how physically consistent a candidate action trajectory is. At inference, it employs Langevin Dynamics MCMC—an iterative sampling process—to refine actions by following the energy gradient toward lower-energy, more physically plausible trajectories. A StateMapper component projects different robot embodiments into a unified 128-dimensional latent action space, allowing one model to control diverse robots.

Who it’s for

This project is for robotics researchers and developers building world models for embodied AI, specifically those looking for alternatives to diffusion-based action generation.

Highlights

• Energy-Based Modeling: Directly measures world consistency rather than modeling marginal distributions.
• Mental Simulation: Uses iterative gradient-based refinement (Langevin Dynamics) to imagine and improve trajectories.
• Flexible Inference: The number of refinement steps can be adjusted at runtime to trade off speed for quality without retraining.
• Multi-Embodiment Support: A unified latent action space enables the model to reason across different robot hardware configurations.