MakazhanAlpamys/Soup

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

Soup – One‑command LLM fine‑tuning and post‑training

What it is – Soup is a Python‑based CLI (and optional web UI) that wraps the whole LLM fine‑tuning pipeline into a single, reproducible workflow. With a tiny YAML config you can:

  • download a base model from HuggingFace,
  • apply LoRA / QLoRA adapters,
  • run quantized training on a modest GPU (as low as 4 GB VRAM) using layer‑streaming to keep the frozen base off the GPU,
  • evaluate, merge, export (GGUF, ONNX, TensorRT, etc.), and even serve the model via an OpenAI‑compatible API.

All of this is orchestrated by the soup command; no manual SSH, no custom scripts, and no fiddly environment variables.

Key features (as described in the README)

Feature What you get
One‑command workflow soup init … && soup train does everything from data loading to checkpointing.
Layer streaming (beta) Streams decoder layers from RAM to GPU one at a time, letting an 8B model train on a 4 GB laptop GPU (≈ 120 tokens / s, 3.3 GB peak).
QLoRA & 4‑bit quantization Memory‑efficient fine‑tuning with automatic selection of batch size, gradient scaling, etc.
Broad model support Any model that loads with AutoModelForCausalLM – Llama‑3.x/4, Qwen, Gemma, Mistral, Mixtral, Phi‑4, etc.
Multiple training objectives SFT, DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, BCO, tool‑calling, pre‑training, distillation, vision/audio, etc.
Web UI & dashboard soup ui launches a local Gradio‑style interface for dataset inspection, live metrics, and chat.
Export & serving Merge LoRA, export to GGUF (llama.cpp/Ollama), ONNX, TensorRT, AWQ, GPTQ, BitNet, and run an OpenAI‑compatible server (soup serve).
Extensive docs & recipes Over 100 ready‑made model recipes, detailed guides for data formats, compliance, and performance tuning.
Cross‑platform Works on CUDA GPUs, Apple Silicon (MPS), and even CPU (slow, for testing). Docker image available.
Community‑driven Recent releases contain >95 % external PRs, active Discord/Telegram, and a DOI‑linked paper.

Quick start (from the README)

# Install the light CLI (no PyTorch)
pipx install soup-cli
# Add the training stack (torch, transformers, peft, …)
pipx install "soup-cli[train]"

# Initialise a config (interactive wizard or template)
soup init --template chat

# Train with the generated soup.yaml
soup train

The same commands work with pip, uv tool, or directly from a Git checkout.

Strengths

  • Low‑resource fine‑tuning – layer streaming makes 8B models feasible on cheap laptops.
  • Zero‑config ergonomics – auto‑detects batch size, GPU type, quantization level, and data format.
  • All‑in‑one toolchain – training, evaluation, merging, exporting, and serving are covered.
  • Extensible – optional extras ([fast], [mlx], [ui], [serve], etc.) let you add speed‑ups, Apple‑silicon support, or a UI without pulling unnecessary heavy dependencies.
  • Good documentation – a full docs folder, command reference, and many tutorials (Colab notebook, videos).

Limitations / caveats (as called out by the project)

  • Layer streaming is still beta – may fail on some GPU models (e.g., free Colab/T4 tier) and requires stream_layers: true in the config.
  • Python version bound – officially supports 3.10‑3.12; 3.13+ may hit wheel‑resolution issues.
  • Torch 2.5.x incompatibility – torch>=2.5.0 conflicts with trl>=0.29; a fresh install with a newer torch is needed.
  • CPU training is only for testing – extremely slow, not meant for production runs.
  • Certain advanced backends (DeepSpeed, FSDP) are optional extras and need extra setup.
  • Security – soup serve now exits with an error if bound to a non‑loopback host without a tool‑auth token; the /v1/tools/bash endpoint is re‑enabled only behind OS‑level isolation.

Who might use it

  • Researchers or hobbyists who want to fine‑tune LLMs on a personal workstation without wrestling with CUDA configs.
  • Small teams looking for a reproducible, single‑command pipeline that also handles export to deployment formats.
  • Anyone needing a quick UI for dataset inspection or live chat while training.

Bottom line

Soup is a genuine, actively maintained open‑source project that abstracts away the plumbing of LLM fine‑tuning. Its standout feature is layer‑streaming, which pushes the limits of what can be trained on low‑VRAM hardware, while the rest of the toolchain (config‑driven CLI, web UI, export options) makes it a practical choice for both experimentation and lightweight production.

Related

  • Dispatch
    Soup Enables Fine‑Tuning an 8B Model on a 4 GB Laptop GPU via Layer Streaming

    Soup lets you fine‑tune an 8B parameter LLM on a 4 GB laptop GPU using layer streaming and QLoRA, achieving bit‑exact results with minimal overhead.

  • Project
    Lightning-AI/litgpt

    A high-performance framework for pretraining, finetuning, and deploying 20+ LLMs with from-scratch implementations and no abstraction layers.

  • Project
    lyogavin/airllm

    AirLLM enables the execution of massive LLMs on low-VRAM consumer GPUs by streaming model layers one at a time instead of loading the entire model into memory.

  • Project
    DaoyuanLi2816/can-i-finetune-this

    A CLI tool to estimate VRAM usage, benchmark performance, and generate training recipes for fine-tuning LLMs on consumer GPUs using LoRA and QLoRA.

  • Project
    axolotl-ai-cloud/axolotl

    A free and open-source framework for streamlining the post-training and fine-tuning of large language models and multimodal models using a configuration-driven approach.