OpenMOSS/MOSS-Image

What is MOSS‑Image?

MOSS‑Image is an open‑source text‑to‑image diffusion model released by the OpenMOSS team. It focuses on layout‑controlled generation, meaning you can tell the model where to place text and objects inside the picture via a simple JSON description. Two checkpoints are provided:

• MOSS‑Image‑Base – the original 5 B‑parameter diffusion model.
• MOSS‑Image‑SFT – a fine‑tuned version that produces clearer text rendering and more organized compositions.

The repo includes the model code, a custom Diffusers pipeline, a Gradio web UI, a ComfyUI plugin, and a “prompt‑enhancer” skill that helps agents turn rough sketches into the required JSON layout.

---

Key Highlights (as listed in the README)

Feature	What it means
Layout control	You supply a JSON with bounding boxes for text and objects; the model respects those positions.
Strong text rendering	Generates legible, accurate characters (including non‑Latin scripts) inside images.
Long‑prompt support	Accepts up to 4 K tokens, allowing detailed scene descriptions.
Versatile visual styles	Can produce anime‑style or photorealistic images.
Easy fine‑tuning	The base model can be adapted to custom domains via LoRA or full‑model SFT.
Compact & efficient	Only a 5 B backbone, making it more affordable to run than larger diffusion models.
Heterogeneous GPU training	Trained on a mix of GPU architectures, so the code works on many hardware setups.

---

Getting Started

1. Install the code

git clone https://github.com/OpenMOSS/MOSS-Image.git
cd MOSS-Image
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

2. Run the model via Python

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "OpenMOSS-Team/MOSS-Image-Base",
    custom_pipeline="pipeline_moss_image",
    trust_remote_code=True,
    torch_dtype={"default": torch.bfloat16, "vae": torch.float32},
)
pipe.scheduler = type(pipe.scheduler).from_config(pipe.scheduler.config)
pipe.to("cuda")

prompt = "A wide view of a refrigerator … handwritten in Chinese…"
image = pipe(
    prompt,
    height=1456,
    width=1456,
    num_inference_steps=50,
    guidance_scale=3.0,
    uncond_skip_layers=[22],
    generator=torch.Generator("cpu").manual_seed(42),
).images[0]
image.save("refrigerator_note.png")

3. Launch the Gradio demo

pip install -e '.[ui]'
python gradio_app.py --model OpenMOSS-Team/MOSS-Image-Base

The UI lets you switch memory modes (GPU‑resident, component, sequential) to fit your hardware.

4. Use with ComfyUI

Search for MOSS‑Image in the ComfyUI Manager and install the plugin. Example workflows are provided under comfyui/moss-image/example_workflows/.

---

Layout‑to‑Image Generation

MOSS‑Image accepts a JSON layout that describes:

• a high‑level summary (overall style, canvas type),
• an items list where each entry has a label (text or object), a human‑readable caption, optional text content, and a bbox specifying position.
• coordinate system details (bbox_coord_space, bbox_format).

Paste that JSON into the Gradio prompt field (or pass it as a string in the Python API) and the model will place the elements exactly where you indicated.

---

Resources

• Model weights: Hugging Face and ModelScope links for both Base and SFT checkpoints.
• Demo: Hosted on Hugging Face Spaces.
• Documentation: English and Chinese user guides (USERGUIDE.md, USERGUIDE_ZH.md).
• Prompt‑enhancer skill: A helper for agents (e.g., GPT‑6‑family) to turn sketches into the required JSON layout.
• Community: Discord, QQ group, Feishu community, and X (Twitter) account for support.
• Citation: BibTeX entry provided for academic use.

---

License

The pipeline code is released under the Apache License 2.0.