Hugging Face AI WebTV: Building an Automated Video and Music Synthesis Stream

Hugging Face has introduced AI WebTV, an experimental demonstration showcasing the integration of open-source text-to-video and music synthesis models to create an automated broadcast stream. The project serves as a tech demo and showreel for the capabilities of current generative AI in producing short, synthesized video sequences and accompanying audio.

System Architecture and Pipeline

AI WebTV utilizes a pipeline that transforms high-level human ideas into a continuous stream of video and audio. The process begins with a base theme and idea provided by a human, which is then processed by a Large Language Model (LLM), specifically ChatGPT, to generate a variety of individual prompts for specific video shots.

Video Generation Chain

The core video synthesis is powered by Zeroscope V2, a model based on ModelScope. The generation process typically follows a two-stage chain:

  1. Initial Generation: The zeroscope_v2_576 model creates a video clip at a resolution of 576x320.
  2. Upscaling: An optional second pass using zeroscope_v2_XL upscales the video to 1024x576. For this stage, the same prompt used in the initial generation must be applied.

Post-Processing and Audio

To refine the visual output and add auditory elements, the pipeline employs two additional tools:

  • Frame Interpolation: The FILM (Frame Interpolation for Large Motion) algorithm is used to smooth transitions and repair small rendering errors, particularly during camera pans or rotations.
  • Audio Synthesis: Music is generated using the musicgen-melody model from the MusicGen family.

Broadcasting Infrastructure

The system is implemented using NodeJS and TypeScript. It leverages Hugging Face Spaces to host the models, calling them via the @gradio/client NPM package. For the final broadcast, FFmpeg is used to read a playlist of .mp4 video and .m4a audio files, which are then sent as an FLV stream to an RTMP server (specifically node-media-server).

Capabilities and Observations

The AI WebTV demo highlights the ability of text-to-video models to emulate real-life physical phenomena and diverse artistic styles.

Dynamic Scene Simulation

Text-to-video models can synthesize motion for fluids, animals, and vehicles. Examples from the project include:

  • Bees buzzing around a flower with soft bokeh backgrounds.
  • A grizzly bear catching a salmon in a rushing river with splashing water.
  • Waves crashing against a rocky shore during a sunrise.

Styling and Composition

The system demonstrates versatility in visual styles, ranging from photorealistic cinematic shots (e.g., an astronaut and llama at dawn) to 3D rendered animations in the style of Pixar or Studio Ghibli.

Technical Limitations and Failure Cases

Despite the successes, the project identified several recurring failure modes in current text-to-video synthesis:

  • Directional Errors: Models may struggle with movement direction, sometimes producing clips that appear to be playing in reverse.
  • Rendering Artifacts: Realistic scenes occasionally exhibit vertical lines or wave-like artifacts.
  • Prompt Leakage: The model may literally insert words from the prompt into the image. For example, mentioning "IMAX" or "Canon EOS" can cause those words or objects to appear physically within the scene.
  • Attribute Neglect: Specific modifiers (e.g., the color "green") are occasionally ignored by the model.

Best Practices for Text-to-Video Prompting

Based on the AI WebTV experiment, Hugging Face recommends the following strategies for improving output quality:

  • Use Video-Specific Keywords: Be explicit about camera movement, character orientation, speed, and direction to avoid random or reversed animations.
  • Ensure Inter-Scene Consistency: When creating a sequence of videos, include exhaustive detail in every prompt to maintain consistency in elements like color across different takes.
  • Utilize Frame Interpolation: Use tools like FILM to smooth out animations and create slow-motion effects, which can turn rendering defects into acceptable stylistic features.

Sources