Gemini Omni Flash API Release

Google has released the Gemini Omni Flash API, enabling developers to programmatically access advanced video generation and editing capabilities. Unlike traditional video models, Gemini Omni Flash focuses on conversational editing and high-fidelity world simulation, allowing users to modify specific elements of a video while maintaining consistency across shots.

Core Capabilities of Gemini Omni Flash

Gemini Omni Flash distinguishes itself from other models like Veo through four primary technical strengths:

Conversational Video Editing

Conversational editing allows for the modification of specific elements within a video without altering the rest of the scene. This includes:

• Character Swapping: Changing a subject (e.g., changing a black cat to a ginger cat) while keeping the same blocking and background.
• Relighting: Altering the time of day or lighting conditions of a scene.
• Attribute Modification: Changing clothing or characters (e.g., swapping a man for a woman in a red dress) while preserving the environment.

Multimodal Reference Inputs

The model can condition video generation on multiple types of input simultaneously:

• Image-to-Video: Using a static image as a reference for the visual style or subject.
• Cross-Reference Integration: Combining a video with a new image for a location and another image for a specific subject (e.g., a specific pet) to create a composite scene.
• Audio Translation: While deep-fake lip-syncing is restricted for safety, the model can translate spoken audio into other languages.

World Model and Simulation

Gemini Omni Flash attempts to simulate real-world physical properties to create believable environments. A key example is the addition of environmental effects like rain and puddles, which the model renders with accurate reflections of characters and objects, demonstrating an understanding of light and surface interaction.

Integrated Text and Logo Rendering

The model can insert and track text or brand logos within a video. It can modify existing signs to display specific text in English or integrate specific brand assets (such as the Go Go Curry logo) into the scene, though the precision of tracking and font accuracy can vary.

Technical Implementation via Interactions API

Gemini Omni Flash utilizes a new Interactions API, designed for multi-turn tasks where the output is a video rather than a standard text chat.

Video Generation Modes

• Text-to-Video: Generates video and audio from a text prompt. Users can specify the aspect ratio (e.g., 16:9 or portrait for social media) and duration.
• Image-to-Video: Uses a reference image (generated via models like NanoBanana) and a text prompt to animate the scene.
• Multi-Reference Generation: Allows multiple images (e.g., a subject and an object) to be passed as references to guide the final video output.

Multi-Turn Editing Workflow

Developers can string together interactions to refine a video iteratively:

1. Initial Generation: Create a base video from text or images.
2. Edit Prompt: Pass the previous interaction as context and provide a text prompt to change a specific detail (e.g., "turn the cat into a puma kitten").
3. Stylization: Apply a style reference (e.g., a watercolor painting) to the existing video to change its visual aesthetic without altering the motion.

Editing Existing Footage

The API supports editing uploaded videos, provided they are 10 seconds or shorter. Users can provide a reference video and a text prompt to add special effects or alter the narrative (e.g., animating a cat crawling out of a computer screen in a real-world recording).

Current Limitations and Constraints

• Duration: Video generation is currently capped at a maximum of 10 seconds.
• Safety Restrictions: Google has implemented strict guards against deep-fake creation; the model will not lip-sync a provided audio file to a provided image of a face.
• Consistency: While powerful, the model can occasionally produce artifacts or become confused during complex multi-turn style transfers.