JoyAI-Image: a unified multimodal foundation model for spatial reasoning, image generation, and instruction-guided editing

What it solves

JoyAI-Image addresses the gap between image understanding and generation by creating a unified model that can perceive, generate, and edit images within a single framework. It specifically focuses on "spatial intelligence," enabling the model to handle complex spatial reasoning, precise object manipulation, and viewpoint changes that typically challenge standard image models.

How it works

The project uses a hybrid architecture combining an 8B Multimodal Large Language Model (MLLM) for understanding and a 16B Multimodal Diffusion Transformer (MMDiT) for generation. These two components work in a closed-loop collaboration: the MLLM provides scene parsing and relational grounding to guide generation, while the generative capabilities (like changing viewpoints) provide new visual evidence to help the MLLM reason about spatial relationships more accurately.

Who it’s for

This tool is designed for researchers and developers working in multimodal AI, specifically those needing high-fidelity image editing, 3D-aware image synthesis, or advanced spatial reasoning for tasks like 3D reconstruction and video generation.

Highlights

• Unified Framework: A single model family that handles image understanding, text-to-image generation, and instruction-guided editing.
• Spatial Intelligence: Supports precise spatial editing patterns including moving objects to specific regions, rotating objects to canonical views, and controlling camera yaw, pitch, and zoom.
• Advanced Typography: Optimized for complex text rendering, including multi-line text, multilingual typography, and handwritten styles.
• Multi-Image Editing: The "Plus" version supports cross-image composition and joint manipulation across multiple images.
• Integration: Compatible with the Hugging Face Diffusers library and ComfyUI.