jd-opensource/JoyAI-Echo

JoyAI-Echo-1.5: Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds

What it solves

JoyAI-Echo addresses the challenge of creating long-form, coherent audio-visual content. It provides tools to generate extended videos that maintain narrative continuity over time and interactive world models that respond to navigation with synchronized environmental sounds, music, and speech.

How it works

The project consists of two independent modules. The Echo-LongVideo module uses a memory bank to maintain consistency across multiple shots, enabling the generation of videos lasting 10 minutes or more. The Echo-WM (World Model) module functions as an omnimodal engine that generates real-time, interactive audio-visual experiences that evolve as a user navigates through a virtual environment.

Who it’s for

This project is intended for researchers and developers interested in long-horizon generative media, interactive world simulation, and advanced audio-visual synthesis. It is strictly for academic and non-commercial use.

Highlights

  • Long-Horizon Generation: Supports multi-shot video creation exceeding 10 minutes with persistent narrative memory.
  • Omnimodal World Modeling: Generates synchronized video, environmental audio, music, and speech that react to continuous navigation.
  • Simulation Pipeline: Includes a Unreal Engine-based pipeline for physics-based trajectory generation and high-quality rendering.
  • Consumer-GPU Support: Optimized inference profiles designed to run on consumer-grade hardware.
  • Modular Architecture: Two distinct, self-contained environments that prevent dependency conflicts.

Related

  • Project
    jd-opensource/JoyAI-VL-Interaction

    An 8B-scale vision-language interaction model and system that enables AI to proactively respond to real-time video streams in under a second without waiting for user prompts.

  • Project
    Yuliang-Liu/Monkey

    A series of Large Multi-modal Models (LMMs) that improve image understanding by prioritizing high image resolution and detailed text labels.

  • Project
    Tencent-Hunyuan/HunyuanOCR

    HunyuanOCR‑1.5 is a lightweight OCR‑focused vision‑language model that unifies document parsing, table extraction, and text‑image translation. It speeds up long‑output decoding with DFlash speculative decoding, offers a unified vLLM/Transformer inference environment, and can be run on laptops via llama.cpp. The repo provides scripts for data packing, supervised fine‑tuning, DFlash draft training, and ready‑to‑use inference servers.

  • Project
    fikrikarim/parlor

    A fully on-device, real-time multimodal AI assistant that enables low-latency voice and vision interactions using Gemma 4 and Kokoro TTS.