Soup: Fine-Tuning 8B LLMs on 4 GB Laptop GPUs via Layer Streaming

Soup enables 8B model fine-tuning on consumer laptop GPUs

Soup is a technical framework and CLI designed to eliminate the infrastructure overhead of LLM fine-tuning. Its primary breakthrough is layer streaming, which allows users to fine-tune 8B parameter models on GPUs with as little as 4 GB of VRAM. By treating the frozen base model as a read-only stream rather than a resident VRAM object, Soup reduces the peak memory requirement from the entire model size to the size of a single decoder layer.

Layer Streaming: Technical Implementation and Memory Efficiency

Layer streaming solves the VRAM bottleneck by moving the frozen base model out of VRAM and into host RAM (or NVMe disk if RAM is insufficient). The model is fed to the GPU one decoder layer at a time, with the next layer prefetched on a dedicated CUDA stream to maintain throughput.

Memory Performance on 4 GB Hardware

Measured on an RTX 3050 Laptop (4 GB VRAM, Windows), Soup achieves the following results for Llama-3.1-8B (NF4 quantization):

  • Peak VRAM Usage: 3.32 GB
  • Throughput: 119.6 tok/s
  • SM Occupancy: 100%

For smaller models like Qwen2.5-3B (un-quantized bf16), Soup enables training in 2.15 GB of VRAM, a scenario that would typically result in a CUDA Out-of-Memory (OOM) error if the model were resident in VRAM.

Correctness and Bit-Exactness

Because streaming can fail silently by cutting the autograd path while the loss still decreases, Soup implements a strict correctness protocol. Every release is verified to be bit-exact against a non-streamed resident run, maintaining a maximum absolute logit difference of 0.0 across nine architecture families.

Support for Preference Optimization (DPO, ORPO, SimPO, KTO)

In version 0.72.4, Soup expanded layer streaming to support preference losses beyond supervised fine-tuning (SFT).

Memory-Free Reference Models

Direct Preference Optimization (DPO) typically requires a reference model to compare against the tuned model, which would normally double the VRAM requirement. Soup optimizes this by using the same streamed base model with its adapters switched off as the reference.

On an RTX 3050 4 GB, streamed DPO peaked at 0.914× the peak of SFT. In contrast, forcing a second physical model into VRAM added 730 MB of overhead.

Computational Trade-offs

While memory is saved, there is a time cost. DPO requires reading the layer stack 1.52× as often per step as SFT does. Other methods like ORPO and SimPO are genuinely reference-free and do not incur this specific overhead.

The Soup Workflow: From Configuration to Deployment

Soup simplifies the post-training stack into a single CLI. The workflow is designed to be "one config, one command."

Installation and Setup

Soup is distributed via PyPI as soup-cli. Users can install specific stacks based on their needs:

  • pip install soup-cli (Light core)
  • pip install "soup-cli[train]" (Training stack including PyTorch, Transformers, and PEFT)
  • pip install "soup-cli[all]" (Complete suite)

Configuration and Training

Training is managed via a soup.yaml file. A typical configuration for a 4 GB card includes:

  • stream_layers: true: Enables the base model to stream out of VRAM.
  • quantization: 4bit: Uses NF4 to reduce the store size.
  • stream_source: auto: Automatically chooses between RAM and NVMe disk.

Post-Training Tools

Beyond training, Soup provides a suite of deployment and evaluation tools:

  • soup ship: A regression gate that uses extraction-based scorers to ensure a tune does not break core capabilities (e.g., tool-calling or JSON validity) before shipping.
  • soup reward synth: Generates a deterministic reward verifier from reference outputs.
  • soup draft: Measures speculative decoding acceptance rates and distills target models into dense draft models.
  • soup export: Supports multiple formats including GGUF for Ollama and llama.cpp, as well as ONNX and TensorRT.

Community Insights on Local LLM ROI

Discussion surrounding the project highlights a shift toward small, open-weight local models for business applications. Community members note that while massive hosted models dominate headlines, many enterprise use cases (such as AML compliance for community banks) do not require that level of power and can be solved more cost-effectively with fine-tuned small models, potentially solving the "ROI crisis" currently facing LLM-based AI implementations.

Sources