OpenMOSS/AnyGPT

AnyGPT – a unified multimodal language model

What it is – AnyGPT is an open‑source large language model that can read and generate text, speech, images, and music. It works by converting every modality into a shared discrete token stream, then training a standard next‑token predictor (a LLaMA‑2‑based transformer). The repo ships:

• a base model that aligns the four modalities and can be used for single‑turn tasks (e.g., text‑to‑image, ASR, TTS, text‑to‑music, etc.)
• a chat model fine‑tuned on the AnyInstruct dataset for multi‑turn multimodal conversations
• inference scripts for both models
• the AnyInstruct instruction dataset (hosted on HuggingFace) that contains prompts for arbitrary modal conversions.

Key components

• SpeechTokenizer + SoundStorm for speech tokenisation/reconstruction
• SEED‑tokenizer for image tokenisation
• unCLIP SD‑UNet (downloaded automatically) for image decoding
• Encodec‑32k for music tokenisation/reconstruction

Getting started

# clone and set up the environment
git clone https://github.com/OpenMOSS/AnyGPT.git && cd AnyGPT
conda create -n AnyGPT python=3.9 && conda activate AnyGPT
pip install -r requirements.txt

Download the model checkpoints from HuggingFace (base, chat, speech modules, SEED tokenizer) and place them where the CLI scripts expect them.

Running inference

• Base model – use cli_infer_base_model.py and supply the paths to the model and the various tokenizers. Prompts follow a simple modality|modality|payload syntax, e.g.:
  • text|image|A futuristic city at sunset
  • image|text|path/to/pic.jpg
  • text|speech|Hello world
  • speech|text|path/to/audio.wav
• Chat model – use cli_infer_chat_model.py with an interleaved instruction format that can mix text, image, speech, and music in a single turn.

Dataset – The AnyInstruct dataset (🤗 fnlp/AnyInstruct) contains multimodal instruction‑response pairs covering all conversion directions. It is used for the SFT (instruction‑tuning) stage.

License – The code and models are released under the same license as Meta’s LLaMA‑2 (see the linked license page).

Citation

@article{zhan2024anygpt,
  title={AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling},
  author={Zhan, Jun and others},
  journal={arXiv preprint arXiv:2402.12226},
  year={2024}
}

Why it matters – By turning images, audio, and music into discrete token streams, AnyGPT demonstrates that a single LLM can be trained on all media with the same objective, enabling zero‑shot cross‑modal generation and conversation without separate specialist models.