noonghunna/club-3090

Community recipes for serving LLMs on RTX 3090/4090/5090 CUDA gpus. Multi-engine (vLLM, sglang, llama.cpp and other custom engines) and model-agnostic. Currently shipping Qwen3.8-27B Qwen3.6 35B Gemma 4 26B Gemma 4 31B configs for 1×/2×/4x/8x cards.

What it solves

Running large language models (LLMs) on consumer-grade hardware like the NVIDIA RTX 3090 can be complex due to memory constraints, configuration overhead, and engine-specific optimizations. This project provides pre-configured, tested recipes that simplify deploying these models locally while ensuring they run efficiently on specific hardware setups.

How it works

It uses Docker and NVIDIA Container Toolkit to manage the deployment environment. The project organizes model configurations (or "slugs") by hardware topology (e.g., single card, dual card, multi-card). Users select a configuration, and the provided scripts handle the downloading, verification, and launching of the model via high-performance inference engines like vLLM, SGLang, or llama.cpp. It exposes an OpenAI-compatible API, allowing it to integrate with existing LLM tools and interfaces.

Who it’s for

This is for developers, researchers, and enthusiasts who own NVIDIA RTX 3090s (or similar cards) and want to self-host LLMs without spending excessive time troubleshooting environment setups, quantization, or engine-specific performance bottlenecks.

Highlights

  • Hardware-Aware Recipes: Pre-tested configurations for 1, 2, or more GPUs to maximize VRAM usage and inference speed.
  • Unified API: Provides an OpenAI-compatible endpoint, making local models drop-in replacements for cloud-based LLM services.
  • Extensive Benchmarking: Includes a shared database of performance metrics (TPS, latency) and quality checks contributed by the community.
  • Modular Design: Supports multiple inference engines (vLLM, llama.cpp, etc.) and includes a terminal-based cockpit (c3) for managing deployments.
  • Multimodal Support: Optional "AI Studio" extension for running image, video, and audio generation models alongside chat models.

Related

  • Project
    n8n-io/self-hosted-ai-starter-kit

    A Docker Compose template that bundles n8n, Ollama, and Qdrant to quickly set up a self-hosted, local AI development environment for building AI agents and workflows.

  • Project
    autonomous-ai/Personal-AI-Computer

    The repository provides open‑source hardware designs and assembly instructions for building high‑performance, locally‑run AI workstations (from a 2‑GPU desktop to an 8‑GPU rack). It pairs the hardware with open‑source inference software (Grid, vLLM, Ollama, llama.cpp) so users can run large language models on‑prem, avoiding cloud costs and preserving data privacy.

  • Project
    docker/model-runner

    A tool for managing and deploying AI models via Docker, streamlining the process of pulling and serving LLMs using backends like llama.cpp and vLLM.

  • Project
    varunvasudeva1/llm-server-docs

    A comprehensive guide to building a private, local LLM server on Debian, integrating inference engines, chat interfaces, and multimodal tools.

  • Dispatch
    Running SOTA LLMs Locally: Hardware and Configuration Guide

    A detailed technical guide on building high-VRAM local LLM workstations, ranging from $2k budget builds to $40k+ professional setups using RTX 6000 Pros and PCIe switches.