jingyaogong/minimind

🧠 Train a 64M-parameter LLM from scratch in just 2h!

What it solves

MiniMind aims to lower the barrier to entry for learning Large Language Model (LLM) development. It provides a way for individuals to train a functional, ultra-small language model (around 64M parameters) from scratch using consumer-grade GPUs, avoiding the "black box" nature of massive models and the high-level abstractions of third-party frameworks.

How it works

The project implements a complete LLM training pipeline using native PyTorch, avoiding high-level library abstractions to ensure transparency. It follows a Transformer Decoder-Only architecture (aligned with the Qwen3 ecosystem) and supports both Dense and Mixture-of-Experts (MoE) configurations. The pipeline covers the entire lifecycle: tokenizer training, pre-training, supervised fine-tuning (SFT), LoRA, RLHF (DPO), RLAIF (PPO/GRPO/CISPO), tool use, and model distillation.

Who it’s for

  • AI beginners and students who want to understand the internal mechanics of LLMs by writing and training code from zero.
  • Developers looking for a lightweight, reproducible starting point for LLM experimentation.
  • Users with limited hardware (e.g., a single NVIDIA 3090) who want to experience the full training process.

Highlights

  • Ultra-Lightweight: The smallest version is roughly 1/2700th the size of GPT-3, enabling fast training (SFT can be completed in ~2 hours on a single 3090).
  • Full-Stack Implementation: Native PyTorch implementation of pre-training, SFT, RLHF, and RLAIF without relying on high-level wrappers.
  • Broad Compatibility: Compatible with transformers, llama.cpp, vllm, and ollama for inference.
  • Advanced Features: Supports MoE, YaRN for long-text extrapolation, Agentic RL for tool-use scenarios, and adaptive thinking chains.

Related

  • Project
    jingyaogong/minimind-v

    A minimal implementation of a Vision Language Model (VLM) that allows users to train a 65M parameter multimodal model on a single consumer GPU in just 2 hours.

  • Project
    qibin0506/Cortex

    An open-source project that provides a full-lifecycle training pipeline for building a lightweight 0.1B MoE large language model from scratch, including pretraining, SFT, and RLHF.

  • Project
    FareedKhan-dev/train-llm-from-scratch

    A complete from-scratch PyTorch implementation of an LLM pipeline, covering everything from raw text tokenization and Transformer architecture to SFT and advanced RLHF alignment techniques like PPO and GRPO.

  • Project
    karpathy/nanochat

    A minimal, hackable harness for training LLMs end-to-end on a single GPU node, designed to produce compute-optimal micro-models for under $100.

  • Dispatch
    nanoVLM: A Minimalist PyTorch Toolkit for Training Vision Language Models

    Hugging Face has released nanoVLM, a lightweight, pure PyTorch toolkit designed to simplify the training and understanding of Vision Language Models (VLMs) for beginners and researchers.