sybil-solutions/ai-data-extraction

AI Coding Assistant Training Data Extraction Toolkit

What it is – A collection of Python scripts that scan the local storage of popular AI‑coding assistants (Claude, Cursor, Codex, Gemini, OpenCode, etc.) and pull out the full conversation history, code snippets, diffs, tool‑call results and metadata. The extracted material is written as line‑delimited JSON (JSONL) so it can be fed directly into dataset‑building pipelines for fine‑tuning or other ML research.

Key features

• Auto‑detects the operating system and looks in the default config / data folders for each supported tool.
• Handles a variety of storage formats: JSONL files, SQLite databases, Tauri .dat files, and content‑addressed blob stores.
• Preserves rich context: file paths, line ranges, code selections, suggested diffs, tool outputs, timestamps, model identifiers, and session hierarchy.
• One‑click scripts for each assistant (extract_cursor.py, extract_claude_code.py, …) plus an extract_all.sh wrapper.
• Optional privacy filter that runs the OpenAI privacy‑filter model locally to mask emails, secrets, etc.
• Helper utilities to merge, filter by date, or select only conversations that contain diffs.
• A corpus_to_skills.py script that can turn a filtered corpus into reusable Agent Skills via a local LLM.

Typical workflow

1. Install Python 3.6+ (no extra dependencies for the core extractors).
2. Run the desired script, e.g. python3 extract_cursor.py.
3. The tool creates extracted_data/…jsonl files containing one JSON object per conversation.
4. Load the JSONL files with libraries like 🤗 Datasets, optionally filter, and use them for fine‑tuning a model.

Why it matters – Training high‑quality code‑assistant models requires large, well‑structured corpora that include not just the textual exchange but also the surrounding code context and edit history. This toolkit automates the otherwise tedious task of pulling that data from the many proprietary storage formats used by existing assistants, letting researchers build richer datasets while keeping the extraction process local‑first and privacy‑aware.

License & safety – MIT‑licensed. The README stresses that users must own the data they extract, respect each tool’s terms of service, and run secret‑scanning/privacy checks before sharing any extracted material.