NoizAI/HelixWorld
🪐 HelixWorld: real-time interactive audio-visual world model.
HelixWorld – Real‑time Interactive Audio‑Visual World Model
What it is – HelixWorld 1.0, from Noiz AI, is a generative model that creates a synchronized video‑plus‑sound scene that reacts instantly to a user’s camera movements. You feed the system an initial image and a text prompt; as you “walk” forward or turn the camera, the model predicts the next frame and the corresponding spatial audio so that the sound field rotates with the viewpoint, rather than being a pre‑mixed soundtrack.
Why it matters – Most existing generative video or audio tools treat the two modalities separately or generate them offline. HelixWorld aims to close that gap by learning a joint audio‑visual representation that can be rolled out causally (i.e., without peeking into the future) and then distilled into a fast pipeline suitable for real‑time interaction.
How it works (high‑level pipeline)
- Spatial AV data collection – The training set mixes first‑person real‑world footage (with on‑location sound) and synthetic captures from game engines where geometry, camera pose, and listener position are known.
- Joint generation + action – A model takes the current visual frame, the current audio field, and the user’s action (e.g., move forward, rotate) and predicts the next frame and the next sound field.
- Causal rollout – The generator is trained to be causal: it must continue producing plausible AV content using only its own past outputs, enabling true interaction.
- Realtime distillation – The heavy joint generator is distilled into a lightweight version that can run the full predict‑decode‑output loop fast enough for interactive use.
Current status – The repository currently hosts a placeholder README. The authors promise to release:
- Model weights
- Inference code (real‑time pipeline)
- A technical report with detailed methodology, figures, and ablations (expected in the coming weeks)
Licensing – The code will be released under the Apache 2.0 license. The model checkpoint will carry a separate weight license that will be published alongside the release.
Who might use it – Researchers exploring multimodal generative models, developers building immersive VR/AR experiences, and anyone interested in synchronized audio‑visual world simulation.
All information above is taken directly from the project's README; no additional features have been inferred.
Related
- Project
- Project
- Project
- Project