Soup Enables Fine‑Tuning an 8B Model on a 4 GB Laptop GPU via Layer Streaming

Overview

Soup turns LLM fine‑tuning into a simple workflow: one YAML config, one command, and no SSH or infrastructure hassle. The project targets training on modest hardware, specifically enabling an 8B model to run on a laptop GPU with only 4 GB of VRAM.

How It Works

Layer streaming keeps the frozen base model out of VRAM and feeds it to the GPU one decoder layer at a time. During LoRA the base is read‑only, so it can reside in host RAM and be streamed into a small pre‑allocated VRAM buffer, prefetched one layer ahead on a dedicated CUDA stream. This reduces peak VRAM usage to roughly the size of a single layer instead of the whole model.

Quantization is applied to the base (default 4‑bit NF4) to further shrink its memory footprint. The adapter (LoRA) trains normally in VRAM.

For preference‑based losses such as DPO, the reference model is not a second copy; Soup re‑uses the same streamed base with its adapters switched off, so only one set of weights is streamed. This makes the reference "free" in memory, though it incurs a time cost: DPO reads the layer stack ~1.52× as often per step as standard supervised fine‑tuning.

The system guarantees bit‑exactness against a non‑streamed resident run: the maximum absolute logit difference is 0.0 across multiple architectures and precisions, verified as a CI test.

Key Features

• Zero SSH: No need to log into remote GPU machines.
• One configuration: A single soup.yaml file controls all aspects of training.
• Automatic handling: Batch size, GPU detection, and quantization are managed automatically.
• Local execution: Training runs on the user’s own GPU with QLoRA; no cloud required.
• Preference loss support: v0.72.4 added DPO, ORPO, SimPO and KTO over layer streaming, with the reference model handled as described above.
• Correctness focus: The project emphasizes bit‑exact reproducibility rather than raw speed.

Recent Updates (v0.72.4)

• Layer streaming now works for DPO, ORPO, SimPO and KTO, not just supervised fine‑tuning.
• Measured on an RTX 3050 Laptop (4 GB, Windows): streamed DPO peaked at 0.914× the supervised‑fine‑tuning peak VRAM usage.
• Forcing a second full model for DPO’s reference would add ~730 MB, essentially one extra copy of the weights.
• ORPO and SimPO are genuinely reference‑free; KTO re‑uses the same reference mechanism as DPO.
• The VRAM pre‑flight accounts for paired losses (chosen + rejected) as twice the rows.
• grpo and ppo remain excluded because their generation step would re‑read every layer per token, which streaming cannot amortize.
• The release is still marked BETA.

Usage Guide

Installation

# Light core (CLI, config, data tools)
pip install soup-cli
# Add training dependencies (torch, transformers, peft, trl, datasets, …)
pip install "soup-cli[train]"

Create a Configuration

# Interactive wizard
soup init
# Or start from a template (e.g., chat)
soup init --template chat

A typical soup.yaml for an 8B model might look like:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
data:
  train: ./data/train.jsonl
  format: alpaca
  val_split: 0.1
training:
  epochs: 3
  lr: 2e-5
  batch_size: auto
  lora:
    r: 64
    alpha: 16
  quantization: 4bit
  stream_layers: true
  stream_source: auto
output: ./output

Train, Test, and Ship

soup train --config soup.yaml          # LoRA, quantization, batching handled
soup chat  --model ./output            # talk to your model
soup push  --model ./output --repo you/my-model

Additional commands include soup merge, soup export (to GGUF, ONNX, TensorRT, etc.), soup eval benchmark, soup data inspect, soup recipes list, soup autopilot, and soup doctor for environment checks.

Performance Measurements (Author‑Reported)

• On an RTX 3050 Laptop (4 GB, Windows): Llama‑3.1‑8B in NF4 with layer streaming achieved 119.6 tok/s, a peak VRAM usage of 3.32 GB, and 100 % SM occupancy.
• The same card runs Qwen2.5‑3B with an un‑quantized bf16 base at 143 tok/s in 2.15 GB VRAM, which would CUDA‑OOM if the base were kept resident.
• Overhead of streaming versus a resident baseline (measured at 0.5 B model size) is 1.43×; the author provides the baseline for verification.
• These numbers are Windows‑specific; Linux would likely be slightly better.
• All measurement records, including discarded runs, are available in the repository’s benchmarks/ directory.

Community Insights

• Local model ROI: Commenters noted that small, open‑weight models avoid the cost of large hosted models and address the ROI crisis many businesses see with LLMs.
• Practical use: One user runs a fine‑tuned 4B model for AML compliance at community banks, citing the same ROI rationale.
• VRAM question: A comment asked why a hard VRAM requirement still exists; the author explained that the frozen base must be streamed because it still needs to arrive before each matrix multiplication, and streaming reduces the requirement to a single layer.
• Hyper‑parameter tuning: Another comment inquired how Soup auto‑tunes hyper‑parameters and makes complex training decisions. The source does not describe automatic hyper‑parameter search; the CLI handles batch size, GPU detection, and quantization automatically, while learning rate, LoRA rank, etc., are set in the configuration file.
• Data amount: Questions about how much data is needed for fine‑tuning were raised. The repository provides only short example datasets; no specific guideline on required data size is given in the source.
• Hardware recommendation: A request for a specific 4 GB GPU laptop recommendation appeared; the source does not endorse any particular model.
• Website cost and readability: A user questioned whether the "Get Started for Free" label might change and commented on the site’s gray‑on‑black readability. The project’s license is Apache‑2.0 and it remains free.
• Comment quality: An observer remarked that nearly half of the comments in the thread were dead or appeared LLM‑generated, noting a high ratio of low‑signal comments relative to the post’s score.

Limitations and Frequently Asked Questions

• VRAM ceiling: The author does not claim support for models larger than 8B on a 4 GB card; 14B NF4 would require ~7.5 GB of page‑locked host memory, which exceeds the measured ~7.12 GB ceiling on the test laptop.
• Preference loss overhead: While the reference model is free in memory, DPO incurs a 1.52× increase in layer‑stack reads per step.
• Excluded algorithms: GRPO and PPO are intentionally omitted because their per‑token generation would re‑read every layer, negating the benefits of streaming.
• Automatic hyper‑parameter tuning: The source does not provide details on auto‑tuning of learning rate, LoRA rank, or other hyper‑parameters; these must be specified in soup.yaml.
• Data size guidance: No universal rule** for required fine‑tuning dataset size is given; users must experiment with their own data.

Conclusion

Soup demonstrates that, with layer streaming and 4‑bit quantization, an 8B parameter LLM can be fine‑tuned on a consumer laptop GPU containing only 4 GB of VRAM. The approach preserves numerical exactness (bit‑exact matches to resident training) while keeping infrastructure demands minimal—no SSH, a single config file, and automatic handling of batch size and quantization. The project remains open source (Apache‑2.0) and encourages community contributions, especially for hardware‑intensive validation beyond what a single 4 GB laptop can provide.