JEPA World Models: Optimizing Planning Without Retraining

Improving World Model Performance Through Planning

Recent experimentation with Joint-Embedding Predictive Architecture (JEPA) models, specifically the LeWorldModel (LeWM), reveals that the primary bottleneck for long-horizon tasks is not the quality of the model's predictions, but the strategy used for planning. By implementing hierarchical planning and memory-based goal selection, researchers can achieve substantial performance gains without retraining the model's weights.

The Planning Horizon Bottleneck

While vanilla LeWM models struggle with long-horizon tasks (e.g., 75 steps), they perform with high accuracy over shorter intervals (e.g., 25 steps). Experimental data shows that splitting a 75-step task into three 25-step segments increases success rates from 22% to 80%. This suggests that the model's utility is limited by its effective planning horizon, and chaining shorter, accurate searches is more effective than attempting to optimize a single, long-range prediction.

Leveraging Latent Memory Graphs

To eliminate reliance on expert demonstrations during deployment, a latent memory graph can be constructed from existing demonstration data. By embedding these frames and connecting them, the planner can navigate from a starting state to a goal state. To prevent the model from creating physically impossible paths, a cost penalty must be applied to jumps between different demonstrations. This method allows the model to achieve performance comparable to using expert-provided subgoals.

Data Augmentation via Symmetry

Because physical tasks like pushing a block are invariant under rotation, translation, and mirroring, memory gaps can be filled by transforming existing demonstration segments. This approach allows the model to handle goals that fall outside the direct path of original training data, raising success rates on difficult, out-of-distribution tasks.

The Impact of Goal Specification

Standard planning often fails because the cost function includes the robot's end-effector position (the pusher) in addition to the target object. Because the goal image contains a specific pusher position, the model attempts to match both, which is often unnecessary and overly restrictive. Removing the pusher from the cost function—or searching over multiple valid pusher positions—can significantly increase success rates, as it allows the model to focus on the task objective rather than matching irrelevant visual details.

Community Perspectives

Discussion surrounding these findings highlights the relationship between JEPA and classical control theory. Some observers note that JEPA functions similarly to Model Predictive Control (MPC), where the 'model' is a learned latent-space representation. Others have raised questions regarding the evaluation methodology, suggesting that relying on demonstration-derived goals may lead to overfitting on the training distribution.

"JEPA is very similar to Model Predictive Control. Here, you swap the model of the system with a black-box simulator (a neural net), apply some version of dynamic programming / Bellman equation to find the optimal action over a planning horizon. In the planning stage, by factoring out the optimization step from the rollout step, you can free the model to focus on learning the dynamics in latent space." — @hazrmard

"This is an interesting technique to recover incomplete states or lost memories after demonstration. I'm guessing that the original LeWM does not have the technique of transformative memories?" — @rvz

Sources

Related

  • Dispatch
  • Project
  • Dispatch
  • Dispatch