esbatmop/MNBVC

MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、wiki、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。

What it solves

MNBVC is a massive, ongoing effort to create the largest possible open-source Chinese language corpus for the internet. It addresses the lack of high-quality, diverse, and large-scale Chinese text data available for training AI models, providing a wide range of content from mainstream culture to niche subcultures, including news, novels, academic papers, and chat logs.

How it works

The project collects raw text data from across the internet, which is then processed into formats like JSONL and Parquet. The team uses a variety of specialized tools for encoding detection, deduplication, PDF parsing, and web crawling (including GitHub and Arxiv), { "//": "The README mentions specific tools for these tasks, such as charset_mnbvc and deduplication_mnbvc" } which are also open-sourced. Data is desensitized by removing numeric strings of 8 digits or longer.

Who it’s for

AI researchers, developers, and NLP practitioners who need massive amounts of Chinese language data for training or fine-tuning Large Language Models (LLMs) and multimodal models.

Highlights

  • Massive Scale: Target of 253TB of data, with over 60TB already collected.
  • Diverse Content: Includes everything from academic papers and legal documents to forum posts and lyrics.
  • Comprehensive Tooling: Provides a dedicated suite of tools for cleaning, crawling, and parsing diverse formats (PDF, HTML, XML, and code).
  • Open Access: Data is available via P2P (VerySync), Baidu Netdisk, ModelScope, and Hugging Face.

Related

  • Project
    datawhalechina/diy-llm

    A Chinese‑language, open‑source curriculum that teaches you to build, train, and evaluate large language models from scratch, complete with lecture notes, six hands‑on assignments, and runnable PyTorch code.

  • Project
    OFA-Sys/Chinese-CLIP

    A Chinese-language version of the CLIP model trained on 200 million image-text pairs for cross-modal retrieval and zero-shot image classification.

  • Project
    dongrixinyu/JioNLP

    A Python library for Chinese NLP preprocessing and parsing, providing tools for data cleaning, entity extraction, and automatic LLM evaluation.

  • Project
    Jesseovo/last30days-skill-cn

    An AI Agent skill that automates deep research across eight major Chinese internet platforms, aggregating and analyzing content from the last 30 days to generate verifiable reports.

  • Project
    OpenDCAI/DataFlow

    A data-centric AI system for generating, cleaning, and preparing high-quality LLM training datasets from noisy sources using visual, low-code pipelines.