logitcai/tinybrain

A Chinese-first small language model built from scratch: open 60B-token dataset, tokenizer, mHC + Engram + MoE model, pretraining through RL, all in PyTorch. 从零做中文小模型,数据和代码全部开源

What it solves

Training a usable Chinese language model usually requires assembling many separate pieces: a tokenizer, a pretraining corpus, a model architecture, supervised fine-tuning, reinforcement learning, and an inference runtime. tinybrain addresses this by packaging the entire pipeline into one codebase, so a single developer can go from raw public data to a chat-ready model on a single consumer GPU, without relying on opaque black boxes or closed-source components.

How it works

The project is built around a fully transparent, modular pipeline. Raw text is downloaded from 26 public sources, cleaned, deduplicated, and mixed into four tiers of pretraining data (nano, mini, base, full) using a custom tokenizer trained on ~35 million sampled tokens. The model architecture itself uses novel components: multi-stream residual connections (mHC) with doubly-stochastic mixing, an Engram lookup-table memory, GQA attention with a 3:1 sliding window, and an optional mixture-of-experts feedforward layer. Training proceeds through standard stages: pretraining with cosine annealing, annealing with a mix of pretraining and conversational data, supervised fine-tuning (SFT) on answer-only targets, multi-teacher online distillation (MOPD), and reinforcement learning via GRPO (covering RLVR, agent RL, and RLAIF). The project also includes a web chat interface, an OpenAI-compatible server, and a suite of evaluation tools. Everything is controlled by a single tinybrain command-line tool that chains stages automatically, saving checkpoints between steps.

Who it’s for

This project is aimed at developers and researchers who want to understand or reproduce a complete Chinese LLM training pipeline from scratch. It is also suitable for those who want to experiment with a small (167M or 499M parameter) model on a single GPU, or who need a fully transparent data and training workflow for educational or research purposes.

Highlights

  • Complete from-scratch pipeline: data collection, cleaning, deduplication, tokenizer training, pretraining, SFT, distillation, RL, inference, and evaluation—all code is readable and self-contained.
  • Novel architecture: multi-stream residual connections (mHC) with doubly-stochastic constraint, Engram lookup-table memory, GQA with sliding window, and optional MoE.
  • Chinese-focused design: tokenizer, data ratios, and evaluation rubrics are tailored for Chinese, including classical Chinese and poetry.
  • Practical for single-GPU: the small model can be pretrained on ~20B tokens in about 20 hours on an RTX 5070 Ti 16GB; a demo run on a 23M-parameter model takes about 3.3 hours on an M4 Max.
  • Highly instrumented: logs show loss, gradient norm, speed, and ETA; TensorBoard sync and a benchmark tool are included.
  • Publically released weights: Small and Medium chat research weights are available on ModelScope, along with a matching runtime.

Related

  • Project
  • Project
  • Project
  • Project