noonghunna/club-3090

Community recipes for serving LLMs on RTX 3090/4090/5090 CUDA gpus. Multi-engine (vLLM, llama.cpp, ik_llama) and model-agnostic. Currently shipping Qwen3.6-27B Qwen3.6 35B Gemma 4 26B Gemma 4 31B configs for 1× and 2× cards.

What it solves

club-3090 provides a curated set of "recipes" (Docker Compose configurations) and tooling to help users serve Large Language Models (LLMs) locally on NVIDIA RTX 3090 GPUs. It removes the guesswork from hardware-aware model serving by providing validated configurations for different GPU counts (single or dual 3090s) and various inference engines, ensuring users can maximize throughput or context window size without crashing due to VRAM limits.

How it works

The project uses a model-agnostic framework of scripts and Docker Compose files to deploy LLMs. It supports multiple inference engines including vLLM, llama.cpp, and ik_llama.

  • Deployment Flow: Users can use a CLI wizard (launch.sh) or a terminal UI cockpit (c3) to select a model and a configuration variant (e.g., a high-throughput dual-card setup or a robust single-card setup).
  • Hardware Awareness: The tooling projects VRAM budgets and filters runnable variants based on the user's available GPU count.
  • Validation: It includes a suite of benchmarks (bench.sh) and quality tests (quality-test.sh) to verify that the model is serving correctly and maintaining performance across different prompts and context lengths.
  • Universal Pull: A pull utility evaluates HuggingFace safetensors repositories to determine if a model will fit on the available hardware.

Who it’s for

  • Homelab enthusiasts and developers who own one or two RTX 3090s (or 4090/5090s) and want to run modern LLMs locally.
  • Users seeking stability who want to avoid "prefill cliffs" (OOM crashes during long prompts) and need validated, OpenAI-compatible API endpoints.

Highlights

  • Multi-Engine Support: Validated recipes for vLLM (max throughput), llama.cpp (max robustness/context), and ik_llama (optimized GGUF quants).
  • Llama-style TUI: A keyboard-driven terminal UI (c3) for discovering, serving, and monitoring models.
  • Detailed Benchmarking: Built-in tools for throughput (TPS) and behavioral quality testing.
  • Hardware-Specific Recipes: Specific configurations for single-card (up to 200K context) and dual-card (up to 262K context) setups.
  • Cross-Rig Diagnostics: A reporting tool (report.sh) that generates redacted, paste-ready hardware and stack reports for troubleshooting.

Related

  • Project
    ggml-org/llama.cpp

    A high-performance C/C++ implementation for local LLM and VLM inference, supporting a wide range of hardware through quantization and hardware acceleration.

  • Project
    syv-ai/qwen38-27b-rtx3090

    An optimized serving setup for Qwen3.8-27B on a single RTX 3090, using speculative decoding and quantization to achieve high throughput and low latency.

  • Project
    weicj/vLLM-2080Ti-Definitive

    A specialized vLLM fork optimized for dual RTX 2080 Ti 22GB GPUs, enabling high-performance inference for 27B-35B class models on Turing-architecture hardware.

  • Project
    turboderp-org/exllamav3

    An inference library for running local LLMs on consumer GPUs, featuring the efficient EXL3 quantization format and support for a wide range of model architectures.

  • Dispatch
    llama.cpp and llama.app: Local LLM Inference and Ecosystem

    llama.cpp and its official home llama.app provide high-performance, hardware-agnostic local LLM inference, now featuring a simplified 'llama serve' command and integration with local coding agents like Pi.