FLUX 3 and FLUX-mimic: Integrating Video Generation with Robot Action Models

FLUX-mimic leverages video generation to drive physical robotics

FLUX-mimic is a next-generation video-action model developed by Black Forest Labs (BFL) and mimic robotics. It utilizes the FLUX 3 multimodal foundation model as a backbone to enable robots to perform general-purpose manipulation tasks. The core thesis is that a model capable of generating realistic video must inherently learn a "world model"—an understanding of contact, motion, weight, and cause-and-effect—which can then be decoded into robotic actions.

The FLUX 3 Multimodal Foundation

FLUX 3 is a single backbone model jointly trained on images, video, and audio. While audio represents a small fraction of the total tokens, video prediction accounts for over 95% of the total compute costs. BFL asserts that learning to render the world accurately requires the model to learn how the world behaves physically.

Unified Modalities

Actions are treated as a low-dimensional representation of a robot's state, tightly coupled to visual observations. Because actions, audio, and video frames are all partial representations of the same physical reality, the model can predict actions as an extension of its existing world knowledge rather than as a separate capability.

Performance Stability

Adding action prediction to the training curriculum does not permanently degrade video generation quality. During large-scale training, human ratings for text-to-video and image-to-video tasks initially dropped by up to 10%, but recovered to full previous quality after 3,500 steps. This demonstrates that video generation and action prediction can share a single foundation without competing for capacity.

Architecture: From World Model to Action Decoder

FLUX-mimic functions by decoding actions from the learned world representation of the FLUX 3 backbone. This is achieved through a lightweight action decoder trained on top of intermediate features extracted from the video prediction path.

Improving Representation with Self-Flow

To ensure the world model is usable for downstream tasks, BFL utilized a framework called Self-Flow. While generative models often produce high-quality simulations, they can result in "entangled" representations that are difficult for action decoders to use. Self-Flow unifies generation and representation learning, resulting in reciprocal improvements: higher generation quality across modalities and higher success rates for robot control tasks in simulation.

Scaling for Real-World Application

FLUX 3 is a scaled-up version of Self-Flow, trained on:

• Tens of millions of hours of general video content to learn broad world dynamics.
• Hundreds of thousands of hours of video focused on human and robot manipulation tasks.

Deployment and Real-World Performance

FLUX-mimic has been deployed in real factory environments, specifically with Audi, to handle tasks that conventional automation struggles with, such as kitting parts, inserting electronic control units into tight fixtures, and handling flexible materials like cables and seals.

Sample Efficiency and Robustness

Because the physics of the world are already embedded in the backbone's representations, adapting the model to a new task requires significantly less demonstration data. In Self-Flow experiments, action prediction reached target success rates in half the training steps compared to models without Self-Flow. Furthermore, FLUX-mimic exhibits natural recovery from failure; for example, a robot that misses a grasp will automatically correct itself and retry, a behavior derived from the model's internal world knowledge rather than specific demonstrations.

Latency and Hardware Optimization

To meet the real-time requirements of a production line, the system is optimized for low latency:

• Backbone Efficiency: Better representations allow for a smaller, faster backbone to achieve the same performance.
• Hardware: The backbone can run in under 80ms on a single NVIDIA RTX 5090 GPU.
• System Latency: Through mimic's deployment stack—which includes real-time chunking and reduced inter-process latency—the total system reaction time is 101ms, comparable to human visual reaction time.

Industry Impact and Community Perspective

Audi Production Lab has validated FLUX-mimic in real-world production, noting that the model solves complex soft-body manipulation tasks that were previously impossible with conventional robotics.

Community discussion on Hacker News highlights the significance of this approach, with observers noting the potential for a new business model where scaling video generation leads directly to robotics capabilities. One user noted the impressive nature of the robot's ability to resolve errors autonomously:

"The video at around 3.30min, where the robot arm took 3 attempts to reseat the window trim, was quite unnerving - I have not seen such resolving before."