OpenDCAI/OpenWorldLib

Unified Codebase for Advanced World Models.

OpenWorldLib – A Unified Framework for World Models

What it is – OpenWorldLib is an open‑source codebase that brings together research on world models: multimodal systems that perceive, remember, reason about, and generate content for complex environments. It provides a standardized library of components (diffusion, LLM cores, perception, memory, 3D handling, etc.) and ready‑to‑run pipelines so researchers can plug‑in new methods or benchmark existing ones.

Key capabilities

  • Multimodal understanding – visual, audio and language perception modules.
  • Visual‑action prediction – pipelines for vision‑language‑action (VLA) tasks such as VQA, action planning, and interactive generation.
  • Visual generation – video, navigation‑driven video, long‑form video, egocentric interaction video, and 3D scene synthesis.
  • Memory & reasoning – long‑term memory structures and reasoning modules (spatial, audio, general).
  • Extensible pipelines – a high‑level Pipeline class that loads pretrained weights, handles operators (input validation), and orchestrates synthesis, reasoning, and representation modules. A stream() interface supports multi‑turn interactions with automatic context storage.

How to get started

# 1. Create a conda env
conda create -n openworldlib python=3.10 -y
conda activate openworldlib

# 2. Install all dependencies (scripts handle submodule installs, CUDA libs, etc.)
cd OpenWorldLib
bash scripts/setup/default_install.sh

After installation you can run a quick demo for the matrix‑game‑2 video‑generation model:

bash scripts/test_inference/test_nav_video_gen.sh matrix-game-2
bash scripts/test_stream/test_nav_video_gen.sh matrix-game-2

The repo includes many similar scripts under scripts/test_inference and scripts/test_stream for other models.

Project layout

OpenWorldLib/
├─ assets/                     # logos, demo media
├─ data/                       # benchmark and test data
├─ docs/                       # installation guide, planning, curated "awesome" list
├─ examples/                   # evaluation notebooks / scripts
├─ scripts/                    # setup, test, and streaming scripts
├─ src/openworldlib/           # core library
│   ├─ base_models/            # diffusion, LLM/Multi‑LLM cores, perception backbones
│   ├─ memories/               # context‑memory and reasoning memory
│   ├─ operators/              # input preprocessing / interaction handling
│   ├─ pipelines/              # high‑level Pipeline classes (inference & streaming)
│   ├─ reasoning/              # audio, general, spatial reasoning modules
│   ├─ representations/        # point‑cloud, simulation env utilities
│   └─ synthesis/              # audio, visual, VLA generation modules
├─ submodules/                # third‑party tools (e.g., diff‑gaussian‑raster)
├─ test/ & test_stream/        # unit‑tests and interactive demos
└─ tools/vibe_code/           # prompts for rapid contribution

Developers typically instantiate a pipeline, call pipeline(input) for a single turn, or pipeline.stream() for a dialogue‑style interaction where the memory component keeps track of prior turns.

Documentation & community

  • Installation guide – docs/installation.md details per‑method dependencies.
  • Planning & roadmap – docs/planning.md lists short‑term goals and upcoming training/optimization work.
  • Curated resources – docs/awesome_world_model.md aggregates papers, surveys, and related open‑source projects.
  • Contribution – The repo encourages pull‑requests that add new world‑model methods or update the curated lists. A “Vibe Coding” prompt library (tools/vibe_code/prompts) helps contributors generate boiler‑plate code quickly.

Related projects – OpenWorldLib builds on the OpenDCAI DataFlow and DataFlow‑MM frameworks and collaborates with RayOrch and Paper2Any.

License – The project is released under the repository’s LICENSE file (standard open‑source, see badge in the README).

Citation – If you use the library, cite the two arXiv papers listed in the README (2026 arXiv:2604.04707 and arXiv:2602.01630).

OpenWorldLib aims to be the go‑to toolbox for anyone building or evaluating next‑generation world models that combine perception, memory, reasoning, and generation across video, 3D, and language modalities.

Related

  • Project
    galilai-group/stable-worldmodel

    A platform for reproducible world model research and evaluation that provides a unified interface for data collection, training, and model-predictive control evaluation.

  • Project
    OpenMOSS/EasyWAM

    EasyWAM is a unified, high-performance research framework for training and evaluating World Action Models, designed to accelerate development through modular design and optimized training/inference pipelines.

  • Project
    huggingface/transformers

    A model-definition framework for state-of-the-art machine learning that provides a unified API for inference and training across text, vision, audio, and multimodal models.

  • Project
    OpenEnvision/WorldFoundry

    WorldFoundry is an open‑source Python framework that unifies inference, visualisation, and benchmark evaluation for world‑model tasks such as video generation, 3‑D/4‑D reconstruction, and embodied‑action. It provides a model/benchmark catalog, a CLI, and a browser‑based Studio UI, all running in a single CUDA‑enabled Conda environment.

  • Project
    Junchao-cs/SolarWM

    SolarWM is a unified, open-source framework for training and running long-horizon video world models, supporting multiple backbones like Wan2.2 and MiniMax-H3 with a shared data and training pipeline.