Hugging Face Modular Diffusers Release

Hugging Face has introduced Modular Diffusers, a framework that enables the construction of diffusion pipelines through the composition of reusable, self-contained blocks. This approach allows developers to mix and match components—such as text encoding, image encoding, denoising, and decoding—to create tailored workflows without needing to write entire pipelines from scratch.

Composable Pipeline Architecture

Modular Diffusers complements the existing DiffusionPipeline class by introducing the ModularPipeline class. This architecture separates the definition of the workflow from the loading of model weights, allowing for greater flexibility in memory management and pipeline modification.

Key Operational Workflow

  • Workflow Definition: Using ModularPipeline.from_pretrained(), users define the sequence of blocks without immediately loading weights.
  • Weight Loading: The .load_components() method is used to configure data types, quantization, and load the actual model weights.
  • Block Manipulation: Because pipelines are composed of flexible blocks, users can inspect, add, remove, or swap blocks. For instance, a text encoder block can be popped out of a pipeline and run as an independent pipeline using .init_pipeline().

Custom Block Development

Users can create custom blocks by defining a Python class that inherits from ModularPipelineBlocks. A custom block must specify three primary properties:

  1. expected_components: Defines the required models and their default Hugging Face Hub repository paths.
  2. inputs: Defines the required and optional input parameters.
  3. intermediate_outputs: Defines the data the block produces for downstream components.

Computation logic is handled within the __call__ method. Once defined, these blocks can be inserted into existing workflows. For example, a DepthProcessorBlock using Depth Anything V2 can be inserted at the beginning of a ControlNet workflow, where its output automatically flows to the subsequent blocks that require it.

Modular Repositories and Hub Integration

Modular Diffusers introduces "Modular Repositories," which allow a repository to reference components from their original model repos rather than hosting all weights locally. This is managed via a modular_model_index.json file.

Custom blocks can be published to the Hugging Face Hub and loaded by other users with trust_remote_code=True. This ecosystem enables the community to share not only model weights but also the specific Python logic required to execute a block within a modular pipeline.

Community Implementations

Several high-performance pipelines have already been implemented using the Modular Diffusers framework:

  • Krea Realtime Video: A 14B parameter model distilled from Wan 2.1 that achieves 11fps on a single B200 GPU, supporting text-to-video and streaming video-to-video.
  • Waypoint-1: A 2.3B parameter real-time diffusion world model from Overworld that generates interactive environments from text prompts and control inputs.

Integration with Mellon

Modular Diffusers integrates with Mellon, a node-based visual workflow interface. This integration is powered by the consistent API of modular blocks (inputs, intermediate_outputs, and expected_components), which allows Mellon to automatically generate user interfaces for blocks without requiring custom UI code.

Key features of the Mellon integration include:

  • Dynamic Nodes: Nodes that adapt their interface based on the selected model.
  • Single-Node Workflows: The ability to collapse an entire modular pipeline into a single node to reduce canvas clutter.
  • Hub Integration: Custom blocks published to the Hub can be loaded into Mellon via a repo_id and automatically configured.

Sources