open-thoughts/OpenThoughts-Agent

OpenThoughts‑Agent (OT‑Agent)

What it is – A research‑grade codebase for building and evaluating small agentic language models. It focuses on the whole data‑to‑evaluation pipeline: generating training data ("datagen"), fine‑tuning models (SFT), running inference traces, and measuring performance on benchmarks via the Harbor evaluation framework.

Key components

• Datagen stack – utilities for large‑scale trace generation using Ray and vLLM, with optional HPC‑ready extras (.[datagen]). Includes helpers for CUDA/C++ extensions (flash‑infer, flash‑attn, triton) and scripts to launch jobs on clusters or cloud VMs.
• SFT stack – wraps the open‑source LLaMA‑Factory repo (as a submodule) to fine‑tune models. Install with the [sft] extra; training configs live under sft/lf_configs/.
• Evaluation stack – a local runner (eval/local/run_eval.py) and a unified listener (eval/unified_eval_listener.py) that drive Harbor/Daytona sandbox jobs, collect trace artifacts, and upload results to Supabase and HuggingFace. Supports both single‑node testing and large‑scale SLURM/HPC runs.
• Launchers – a modular python -m hpc.launch entry point that abstracts over job type (datagen, sft, eval_listener, etc.) and target platform (SLURM clusters, SkyPilot cloud VMs, or local Docker/Modal). Generates sbatch scripts, handles environment dotfiles, and can auto‑retry scarce GPU resources.
• Cloud helpers – SkyPilot‑based wrappers (data/cloud/launch_tracegen_cloud.py, eval/cloud/launch_eval_cloud.py) let you run the same datagen/eval pipelines on GCP, AWS, Lambda, or Kubernetes without writing SLURM scripts.

Typical workflow

1. Set up a Python 3.12 env and install the base stack with uv pip install -e .[datagen,sft,cloud].
2. Generate data – run a datagen script via python -m hpc.launch --job_type datagen … or the cloud launcher. The script produces datasets and optional trace logs.
3. Fine‑tune – launch an SFT job (--job_type sft) that uses a LLaMA‑Factory config to train a small agentic model on the generated data.
4. Evaluate – either run eval/local/run_eval.py for quick sanity checks or submit a full evaluation listener on a cluster (--job_type eval_listener). Results are automatically uploaded and appear on the public leaderboard.

Infrastructure notes

• Designed for HPC clusters (SLURM, module‑based toolchains) but also works on containers (Singularity) and cloud VMs.
• Heavy GPU‑oriented dependencies (flash‑attn, vLLM, deepspeed, bitsandbytes) are optional extras; the README provides detailed compiler/CUDA version guidance.
• Secrets (HF token, W&B, Supabase, Daytona) are expected in a private env file referenced by DC_AGENT_SECRET_ENV.

Who it’s for – Researchers or engineers who need an end‑to‑end pipeline for creating agentic LLMs at a modest scale, especially those comfortable with HPC or cloud‑based GPU clusters.

Where to learn more – Project website, leaderboard, and trace‑viewer links are listed at the top of the README; the docs/EVAL_GUIDE.md and hpc/README.md provide deeper usage details.