jingyaogong/minimind

🧠 Train a 64M-parameter LLM from scratch in just 2h!

What it solves

MiniMind aims to lower the barrier to learning Large Language Model (LLM) development by providing a fully open-source, reproducible path to train a tiny language model from scratch. It addresses the problem where most AI learners only perform minor fine-tuning (like LoRA) on massive, "black-box" models, preventing them from understanding the underlying physics and core implementation of LLMs.

How it works

The project implements a lightweight LLM (around 64M parameters) using native PyTorch, avoiding high-level abstractions from third-party libraries to ensure transparency. It provides a complete training pipeline including:

  • Pre-training: Training on general text corpora.
  • Supervised Fine-Tuning (SFT): Instruction tuning for dialogue and tool use.
  • Reinforcement Learning (RL): Implementing RLHF (DPO) and RLAIF (PPO, GRPO, CISPO) from scratch.
  • Architecture: A Transformer Decoder-Only structure aligned with the Qwen3 ecosystem, featuring Pre-Norm, RMSNorm, SwiGLU, and RoPE (with YaRN for long-text extrapolation).
  • MoE Support: An optional Mixture-of-Experts version to increase capacity while maintaining low active parameters.

Who it’s for

  • LLM Beginners: Individuals wanting to understand every line of code in the LLM lifecycle.
  • Independent Developers: Those with limited hardware (e.g., a single NVIDIA 3090) who want to train and experiment with models quickly and cheaply.
  • Researchers: People looking for a minimal, extensible baseline for experimenting with new training algorithms or architectures.

Highlights

  • Extreme Efficiency: Can be trained from scratch in approximately 2 hours for under 3 RMB on a single GPU.
  • Full-Stack Implementation: Covers the entire chain from tokenizer training and data cleaning to pre-training, SFT, and advanced RL (Agentic RL).
  • Native PyTorch: Core algorithms are implemented from zero without relying on high-level framework wrappers.
  • Broad Compatibility: Compatible with transformers, llama.cpp, vllm, and ollama for inference.
  • Integrated Tooling: Includes a Streamlit-based WebUI and an OpenAI-compatible API server supporting reasoning content and tool calls.

Related

  • Project
    jingyaogong/minimind-v

    A minimal implementation of a Vision Language Model (VLM) that allows users to train a 65M parameter multimodal model on a single consumer GPU in just 2 hours.

  • Project
    qibin0506/Cortex

    An open-source project that provides a full-lifecycle training pipeline for building a lightweight 0.1B MoE large language model from scratch, including pretraining, SFT, and RLHF.

  • Project
    FareedKhan-dev/train-llm-from-scratch

    A complete from-scratch PyTorch implementation of an LLM pipeline, covering everything from raw text tokenization and Transformer architecture to SFT and advanced RLHF alignment techniques like PPO and GRPO.

  • Project
    karpathy/nanochat

    A minimal, hackable harness for training LLMs end-to-end on a single GPU node, designed to produce compute-optimal micro-models for under $100.

  • Dispatch
    nanoVLM: A Minimalist PyTorch Toolkit for Training Vision Language Models

    Hugging Face has released nanoVLM, a lightweight, pure PyTorch toolkit designed to simplify the training and understanding of Vision Language Models (VLMs) for beginners and researchers.