jingyaogong/minimind-o

🎙️ A 0.1B Omni model trained from scratch, capable of listening, speaking, and seeing!

MiniMind‑O – A Tiny End‑to‑End Omni Model (≈0.1 B parameters)

What it is – MiniMind‑O is an open‑source, multimodal (text + speech + image) model that can understand any of those inputs and respond with both text and streamed speech. The whole “Thinker → Talker” pipeline is implemented from scratch in PyTorch (no high‑level framework magic) and ships with training scripts, data, pretrained weights and demo interfaces.

Key Features (as described in the README)

  • Omni capability – a single weight file handles three input modalities (text, audio, vision) and two output modalities (text, real‑time speech).
  • Ultra‑small footprint – the base “minimind‑3o” model is ~113 M trainable parameters (≈0.1 B). A MoE variant (≈315 M) is also provided. This makes full‑chain training possible on a single RTX‑3090 in ~2 hours with the mini dataset.
  • Thinker–Talker dual‑path architecture
    • Thinker: a MiniMind transformer that fuses multimodal embeddings and generates the textual reply.
    • Talker: an independent MiniMind block that, conditioned on Thinker’s hidden state, predicts multiple layers of Mimi audio codes via MTP (Multi‑Token Prediction). The codes are decoded into 24 kHz speech on‑the‑fly, enabling barge‑in and near‑duplex interaction.
  • Modality projectors – frozen external encoders (SenseVoice‑Small for audio, SigLIP‑2 for vision) feed 2‑layer MLP projectors that map their features into MiniMind’s hidden space.
  • Voice cloning & prompts – in‑context voice cloning using speaker embeddings (CAM++) and reference audio codes. Five built‑in voice prompts and seven unseen prompts are provided for experiments.
  • Training data – two bundles:
    • mini: a small English‑only subset that runs the whole pipeline in ~2 h.
    • full: the complete multilingual (Chinese + English) dataset used for the released weights.
  • Full‑stack code – from data loading (train_sft_omni.py) to evaluation (eval_omni.py) and demo UI (Gradio web UI and a real‑time phone‑mode UI).
  • No third‑party abstractions – all core layers are written directly with PyTorch tensors; the code still works with the HuggingFace tokenizer for convenience.

Typical Use‑Cases

Scenario How MiniMind‑O helps
Research & education – learn how an omni model is built and trained from scratch. The repo provides a minimal 0.1 B model, full training scripts, and a tiny dataset that can be run on a single GPU.
Prototype multimodal assistants – chat with text, speak to the model, or show it an image and get spoken answers. Use the provided CLI (eval_omni.py) or the Gradio/WebUI to interact in real time, with barge‑in support.
Voice‑style experiments – test in‑context voice cloning or compare different speaker embeddings. Built‑in voice prompts + ability to supply arbitrary reference audio codes.
Benchmarking tiny omni models – compare parameter‑efficiency, CER/WER, or voice‑cloning similarity against larger open models. Published evaluation curves, CER/WER numbers, and speaker‑similarity tables are included in the technical report.

Quick‑Start (from the README)

  1. Clone & install
    git clone --depth 1 https://github.com/jingyaogong/minimind-o
cd minimind-o
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
  2. Download required models & data (ModelScope or HuggingFace LFS)
    modelscope download --model gongjy/SenseVoiceSmall      --local_dir ./model/SenseVoiceSmall
modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve
modelscope download --model gongjy/mimi               --local_dir ./model/mimi
modelscope download --model gongjy/campplus           --local_dir ./model/campplus
modelscope download --model gongjy/minimind-3o-pytorch llm_768.pth --local_dir ./out   # base LLM weights
  3. Run inference (CLI example)
    python eval_omni.py --load_from model --weight sft_omni
    Or start the Gradio demo:
    cp -r minimind-3o ./scripts/minimind-3o   # copy transformer‑format weights
cd scripts && python web_demo_omni.py
  4. Train on the mini dataset (single‑GPU example)
    cd trainer
CUDA_VISIBLE_DEVICES=0 torchrun --master_port 29560 --nproc_per_node 1 \
    train_sft_omni.py --learning_rate 5e-4 \
    --data_path ../dataset/sft_t2a_mini.parquet --epochs 1 \
    --batch_size 40 --use_compile 1 --from_weight llm --save_weight sft_zero \
    --max_seq_len 512 --use_wandb 0 --use_moe 0
    (Subsequent audio‑proj and full‑pipeline steps follow the same pattern.)

Where to Find More

  • Technical Report – arXiv: 2605.03937 (linked at the top of the README).
  • Model weights & collections – ModelScope and HuggingFace under the MiniMind‑O collection.
  • Demo videos & live Gradio demo – links provided in the README.
  • Related projects – MiniMind (LLM) and MiniMind‑V (vision‑language) for the underlying transformer backbone.

TL;DR

MiniMind‑O gives anyone with a modest GPU a complete, end‑to‑end multimodal model (text + speech + image) that can be trained from scratch or used out‑of‑the‑box for interactive demos. Its tiny 0.1 B footprint, open‑source code, and bundled data make it a practical entry point for learning or prototyping omni‑modal AI.

Related

  • Project
    OpenBMB/MiniCPM-o-Demo

    A demo system for MiniCPM-o 4.5 that enables real-time, full-duplex omnimodal interaction, allowing the AI to see, listen, and speak simultaneously.

  • Project
    k2-fsa/OmniVoice

    OmniVoice is a diffusion‑based, zero‑shot TTS model covering 600+ languages. It offers voice cloning, attribute‑driven voice design, and auto‑voice generation, with fast inference (RTF ≈ 0.025). Install via pip, load the model with `OmniVoice.from_pretrained`, and call `model.generate` (or use the provided CLI tools). Optional FlashInfer kernels give 2‑3× speedup. The repo includes training scripts, a Gradio demo, and a Colab notebook.

  • Dispatch
    Qwen 3.5‑397B‑A17B release: hybrid linear‑attention MoE model with 1 M token context and state‑of‑the‑art multimodal performance

    Qwen 3.5‑397B‑A17B is a 397 billion‑parameter multimodal model that activates only 17 billion parameters per token, delivering state‑of‑the‑art performance on language, coding, reasoning and vision tasks while being up to 19× faster than its predecessor.

  • Project
    openai/whisper

    Whisper is OpenAI’s open‑source, transformer‑based speech‑to‑text system that handles transcription, language identification, and translation across many languages. It ships as a pip‑installable package with six model sizes (tiny → large, plus a fast “turbo” variant). The CLI (`whisper …`) and a simple Python API (`whisper.load_model(...).transcribe(...)`) let you turn audio files into text with just a few commands. The project runs on Python 3.8‑3.11, PyTorch, and requires ffmpeg (and optionally Rust for the tokenizer). All code and model weights are MIT‑licensed.

  • Project
    Ma-Zhuang/OmniNWM

    OmniNWM is a research codebase that builds a panoramic, multi‑modal world model for autonomous‑driving simulation. It jointly generates RGB, semantic, depth, and 3‑D occupancy maps from a vehicle trajectory, uses normalized Plücker ray‑maps for precise control, and provides occupancy‑based dense rewards for closed‑loop policy evaluation. The repo includes installation steps (including a manual patch to `transformers`), data preparation instructions for nuScenes, pretrained checkpoint download links, and scripts for inference, out‑of‑distribution testing, and staged training.