NoizAI/HelixWorld

🪐 HelixWorld: real-time interactive audio-visual world model.

HelixWorld – Real‑time Interactive Audio‑Visual World Model

What it is – HelixWorld 1.0, from Noiz AI, is a generative model that creates a synchronized video‑plus‑sound scene that reacts instantly to a user’s camera movements. You feed the system an initial image and a text prompt; as you “walk” forward or turn the camera, the model predicts the next frame and the corresponding spatial audio so that the sound field rotates with the viewpoint, rather than being a pre‑mixed soundtrack.

Why it matters – Most existing generative video or audio tools treat the two modalities separately or generate them offline. HelixWorld aims to close that gap by learning a joint audio‑visual representation that can be rolled out causally (i.e., without peeking into the future) and then distilled into a fast pipeline suitable for real‑time interaction.

How it works (high‑level pipeline)

  1. Spatial AV data collection – The training set mixes first‑person real‑world footage (with on‑location sound) and synthetic captures from game engines where geometry, camera pose, and listener position are known.
  2. Joint generation + action – A model takes the current visual frame, the current audio field, and the user’s action (e.g., move forward, rotate) and predicts the next frame and the next sound field.
  3. Causal rollout – The generator is trained to be causal: it must continue producing plausible AV content using only its own past outputs, enabling true interaction.
  4. Realtime distillation – The heavy joint generator is distilled into a lightweight version that can run the full predict‑decode‑output loop fast enough for interactive use.

Current status – The repository currently hosts a placeholder README. The authors promise to release:

  • Model weights
  • Inference code (real‑time pipeline)
  • A technical report with detailed methodology, figures, and ablations (expected in the coming weeks)

Licensing – The code will be released under the Apache 2.0 license. The model checkpoint will carry a separate weight license that will be published alongside the release.

Who might use it – Researchers exploring multimodal generative models, developers building immersive VR/AR experiences, and anyone interested in synchronized audio‑visual world simulation.

All information above is taken directly from the project's README; no additional features have been inferred.

Related

  • Project
    v-modal/vmodal_sdk_android

    V‑Modal AI Android SDK – a native Kotlin/Java library that lets Android apps upload videos/images, have them indexed by V‑Modal’s cloud AI, and perform semantic search across the media. The SDK handles authentication, signed streaming uploads, progress/cancellation, and returns typed results, leaving all UI design to the developer.

  • Project
    anymouschina/TapCanvas

    An Agent-native infinite canvas for AI film and multimodal content production that integrates scripts, storyboards, and assets into a traceable workflow.

  • Project
    OpenMOSS/MOSS-VL

    MOSS‑VL is an open‑weight 11 B‑parameter family of video‑language models that can understand and converse about live video streams in real time. It uses a cross‑attention architecture with timestamped frames and a 3‑D rotary embedding (XRoPE) to achieve low‑latency, interruptible dialogue, proactive silence, and dynamic correction. Three variants are released (Realtime, Instruct, Base), with quantized checkpoints for single‑GPU use, integration with FlashAttention‑3, SGLang, ms‑swift, and LoRA fine‑tuning. The project includes benchmarks showing state‑of‑the‑art streaming performance and provides scripts for both realtime and offline inference.

  • Project
    NomaDamas/CozyClay

    A browser-based 3D staging studio for blocking scenes and sequencing AI-generated motion, featuring direct AI control via the Model Context Protocol.