Hugging Face AI Tools for Art Newsletter Issue 1

Hugging Face has introduced a monthly newsletter dedicated to creative AI, marking a transition from the foundational breakthroughs of 2024 to a more multimodal future in 2025. The first issue summarizes the shift in image generation architectures and the rapid growth of open-source video, audio, and 3D generation tools.

Image Generation: The Shift to Diffusion Transformers (DiT)

Open-source image generation has reached a point where it competes directly with closed-source models in text-to-image generation, editing, and controlled generation. The primary technical evolution in 2024 was the transition from traditional Unet-based architectures to Diffusion Transformers (DiT) and the adoption of flow matching objectives.

Technical Evolution and Key Models

Gaussian flow matching provides a vector field parametrization of network output that differs from previous diffusion model methods. This architectural shift was first announced by Stability AI with Stable Diffusion 3, followed by the first open-source DiT model, HunyuanDiT. Subsequent releases including AuraFlow, Flux.1, and Stable Diffusion 3.5 continued this trend.

Flux.1 [dev] is noted as a pivotal release, achieving state-of-the-art performance that surpasses closed-source models such as Midjourney v6.0 and DALL·E 3 (HD) on various benchmarks.

Personalization and Zero-Shot Techniques

While early personalization relied on fine-tuning methods like Textual Inversion, DreamBooth, and LoRA, 2024 saw a surge in "zero-shot" techniques. These allow for high-quality portrait generation from a single reference image without further optimization. Key tools in this category include:

  • IP adapter FaceID
  • InstantID
  • Photomaker

Currently, many personalization and controlled generation tools remain based on Stable Diffusion XL (SDXL) because the semantic roles of its components are better understood than those in newer DiT architectures. Identifying these roles within DiTs is expected to be a primary focus for 2025.

Video and Audio Generation Progress

While video generation lags behind image generation in terms of maturity, 2024 marked a significant leap in capabilities, influenced largely by the expectations set by OpenAI's Sora.

Open Video Models

Several open-source video models have emerged, including CogVideoX, Mochi, Allegro, LTX Video, and HunyuanVideo. However, these models face challenges regarding motion quality, coherence, and high computational requirements, which often necessitate memory optimizations and quantization for local hardware usage.

Audio and Music Breakthroughs

Audio generation has evolved from simple sound effects to full songs with lyrics. Notable 2024 releases include OuteTTS and IndicParlerTTS for text-to-speech, and OpenAI's Whisper large v3 turbo for speech recognition. January 2025 has already seen the release of several new models:

  • Text-to-Speech: Kokoro, LLasa TTS, and OuteTTS 0.3.
  • Music: JASCO and YuE.

YuE is highlighted as a high-performing open-source music foundation model for full song generation, achieving results competitive with closed-source models like Suno under an Apache 2.0 license.

Community-Driven Creative Tools

The open-source ecosystem has produced several specialized tools based on existing pipelines:

  • Flux Fine-tuning: Enabled largely by the AI-toolkit by ostris.
  • Face to All: Combines Instant ID with ControlNet depth constraints and SDXL LoRAs for training-free stylized portraits.
  • Flux Style Shaping: Utilizes Flux [dev] Redux and Flux [dev] Depth for style transfer and optical illusions.
  • 3D Generation: TRELLIS established a high standard for 3D asset creation, followed by January 2025 releases including Hunyuan 3D-2, SPAR3D, and DiffSplat.
  • Other Utilities: IC-Light for consistent relighting, Live Portrait and Face Poke for animating static portraits, and Diffusers Image Outpaint for expanding images using SDXL Fill Pipeline.

Outlook for 2025

Hugging Face anticipates 2025 to be the year open-source catches up in video, movement, and audio modalities. As image generation reaches a natural plateau, the focus is expected to shift toward more efficient computing, quantization, and the integration of multiple modalities.

Sources