DLLXW/baby-llama2-chinese

What it solves

This project provides a complete, small-scale pipeline for building a Chinese Llama2-style language model. It aims to lower the barrier for beginners to enter the LLM field by providing a manageable codebase that covers the entire lifecycle of a model, from data collection and pre-training to supervised fine-tuning (SFT).

How it works

The project implements a small-parameter model (ranging from 92M to 218M parameters) and provides scripts for each stage of development:

1. Data Pre-processing: Includes tools for cleaning high-quality Chinese corpora (filtering short texts, Minhash/Simhash deduplication) and tokenizing data using the ChatGLM2-6B tokenizer to save storage space.
2. Pre-training: A script to train the base model on large-scale Chinese datasets (up to 63.4 billion tokens) to develop basic text completion capabilities.
3. Supervised Fine-Tuning (SFT): A full fine-tuning process to transform the base model into a chat-capable assistant, with specific support for general conversation and medical vertical domains.
4. Inference: A unified entry point (infer.py) that supports both pre-trained and SFT weights across different hardware (CUDA, MPS, CPU).

Who it’s for

• LLM beginners who want to learn the full training pipeline by running a real project on consumer-grade hardware (e.g., NVIDIA RTX 3090).
• Developers interested in training small, domain-specific (e.g., medical) Chinese language models.

Highlights

• End-to-End Pipeline: Covers pre-training, SFT, and inference in one repository.
• Optimized Tokenization: Uses a 64k vocabulary size that fits within uint16, reducing data storage by half compared to int32.
• Domain Adaptation: Demonstrates the transition from a general base model to a specialized MedicalChat model.
• Data Cleaning Suite: Includes built-in tools for Minhash and Simhash deduplication to improve training data quality.