voidful/awesome-chatgpt-dataset

What it solves

Training a capable ChatGPT-style model needs large amounts of high-quality conversational and instruction data. Finding, comparing, and mixing the right datasets from the many public options is time-consuming and confusing. This project solves that by collecting dozens of datasets into a single, size-sorted list, with information about language, source, and license for each.

How it works

The project is a curated table (an "awesome list") of datasets hosted mostly on Hugging Face and GitHub. Each entry shows the dataset name, approximate size (from 1K to 79M samples), the language(s) it covers, a short description of its origin and content, and its license. The list is sorted by size from small to large, with unknown-size entries at the end. Additionally, the repository provides a small script (preprocess.py) that lets you clone the repo, pick any combination of the listed datasets from the mixed/dataset folder, then merge and upload the resulting mixed dataset directly to the Hugging Face Hub.

Who it’s for

This is for anyone training or fine-tuning a large language model, especially those who want a ChatGPT-like assistant. It is useful for researchers, developers, or hobbyists who need a quick overview of available instruction-tuning, preference, reasoning, code, and multilingual datasets, and who want to assemble a custom training mix without manually hunting for each dataset.

Highlights

• Lists over 100 datasets, from small (1K samples) to massive (79M samples)
• Covers many languages including English, Chinese, Traditional Chinese, Japanese, Portuguese, and multilingual sets
• Includes datasets for instruction tuning, preference learning, reasoning, code generation, safety, medical Q&A, math, and more
• Each entry has a direct link, size, language, and license
• Provides a ready-to-use script to merge chosen datasets and upload to Hugging Face Hub
• Sorted by size to help you quickly find a dataset that fits your compute and data budget