esbatmop/MNBVC
MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、wiki、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。
What it solves
It addresses the scarcity of massive, diverse, and high-quality Chinese language datasets required for training large-scale AI models. The project aims to build the largest possible Chinese internet corpus by collecting data from a wide range of sources, including mainstream and niche cultures, news, literature, and specialized documents.
How it works
The project collects raw text data from across the internet and processes it into usable formats like JSONL and Parquet. It employs a distributed effort involving various specialized teams for OCR conversion, Q&A alignment, text augmentation, and code extraction. To manage the scale, the project provides a suite of optimized tools for encoding detection, deduplication, PDF parsing, and web crawling.
Who it’s for
AI researchers and developers focusing on Chinese Natural Language Processing (NLP), Large Language Model (LLM) training, and multimodal AI development who need vast amounts of diverse Chinese training data.
Highlights
- Massive Scale: Target size of 253TB of data, with over 61TB already collected.
- Diverse Content: Includes everything from news, novels, and academic papers to chat logs, lyrics, and "Mars language" (internet slang).
- Comprehensive Tooling: Provides specialized tools for cleaning, deduplication, and crawling code repositories from GitHub and Bitbucket.
- Multimodal Support: Includes tools for PDF parsing and Arxiv document processing to create image-text pairs for multimodal models.
Related
- Project
- Project
- Project
- Project
- Project