Japanese Stable Diffusion release by rinna

TL;DR

rinna announced Japanese Stable Diffusion, a fine‑tuned text‑to‑image model that accepts Japanese prompts and produces images reflecting Japanese culture, addressing the limitations of the English‑centric original Stable Diffusion.

Background: Stable Diffusion

Stable Diffusion, created by CompVis, Stability AI, and LAION, uses the CLIP text encoder and a latent diffusion model to generate images from text. It was trained primarily on the English subset of LAION‑5B, which makes it most effective with English prompts and results that align with Western cultural norms. The model runs inference on a ~10 GB VRAM GPU, offering high quality at modest computational cost.

Why a Japanese‑Specific Model?

The original model requires translating non‑English prompts, and it often misinterprets Japanese‑specific terms, slang, onomatopoeia, and proper nouns. For instance, the Japanese word "サラリーマン" (salary man) conveys a cultural image distinct from the generic English "businessman". Japanese Stable Diffusion was built to:

  • Generate images in a Japanese visual style.
  • Correctly understand Japanese‑adapted English words.
  • Recognize Japanese onomatopoeia.
  • Handle Japanese proper nouns.

Training Data

  • Approximately 100 million images paired with Japanese captions, including the Japanese subset of LAION‑5B.
  • Low‑quality samples were filtered out using the rinna‑published model japanese-cloob-vit-b-16, discarding items below a quality threshold.

Training Procedure

Because the Japanese dataset is roughly 1/20th the size of the English dataset used for the original model, rinna fine‑tuned Stable Diffusion instead of training from scratch. The process followed two stages inspired by the PITI approach (arXiv:2205.12952).

Stage 1 – Japanese‑Specific Text Encoder

  • The latent diffusion model was frozen.
  • The English CLIP text encoder was replaced with a Japanese encoder trained on a sentence‑piece tokenizer designed for Japanese.
  • Using the CLIP tokenizer on Japanese text yields unintelligible byte‑level tokens (e.g., ['ãĤ', 'µ', …]), whereas the custom tokenizer produces meaningful tokens like ['▁', 'サラリーマン', '▁', '油', '絵'].
  • After this stage, the model could interpret Japanese prompts but still generated Western‑styled images (e.g., a salary man with a Western face).

Stage 2 – Joint Fine‑Tuning of Text Encoder and Latent Diffusion Model

  • Both the text encoder and the latent diffusion model were trained together.
  • This joint fine‑tuning enabled the model to produce Japanese‑style imagery, such as a salary man with a Japanese facial appearance.

Open Release Strategy

rinna follows an open‑source philosophy to democratize AI beyond English. The company has already released Japanese versions of GPT‑1B, BERT‑base, and CLIP‑ViT‑B‑16, and now adds Japanese Stable Diffusion to its portfolio. All assets are publicly available:

Limitations and Future Work

Japanese Stable Diffusion is less versatile than the original model and still exhibits accuracy gaps. rinna plans to continue improving language‑specific models and is exploring self‑supervised models for Japanese speech, extending the multilingual AI ecosystem.

Sources