Budgeting VRAM: Adding a Datacenter GPU to a Gaming PC for Local LLMs

For developers and AI enthusiasts running Large Language Models (LLMs) locally, the primary bottleneck is almost always VRAM. While high-end consumer cards like the RTX 4080 provide excellent gaming performance, their 16GB of VRAM is often insufficient for the larger, more capable models that can truly compete with proprietary cloud APIs.

When the cost of upgrading to a 32GB or 48GB consumer card becomes prohibitive, there is an alternative: the secondhand datacenter market. By integrating a decommissioned Tesla V100 SXM2 into a standard gaming PC, it is possible to double available VRAM for a fraction of the cost of new hardware.

The Hardware: Tesla V100 SXM2

The Tesla V100 SXM2 is a Volta-architecture GPU designed for NVIDIA's DGX/HGX server racks. Unlike consumer cards, it lacks a PCIe connector, display outputs, and standard power connectors, as it was designed to sit on a proprietary board and communicate via NVLink.

Despite its age (launching in 2017), the V100 remains a powerhouse for inference due to its HBM2 memory. With a 4096-bit memory bus, it delivers 900 GB/s of bandwidth. For comparison, this outperforms the memory bandwidth of the Apple M3 Max (400 GB/s) and M4 Max (546 GB/s), and even edges out the RTX 4080's 736 GB/s. In LLM inference, where memory bandwidth determines tokens per second, this makes the V100 a highly efficient choice for the price.

Overcoming the Connector Gap

To make an SXM2 card work in a consumer motherboard, a third-party SXM2-to-PCIe adapter is required. This bare PCB converts the proprietary socket into a standard PCIe edge connector, allowing the GPU to slot into a motherboard alongside existing hardware.

Taming the "Fan from Hell"

Datacenter GPUs are designed for industrial 2U server chassis with high-static-pressure cooling. The fans provided with these adapters are typically non-PWM, running at 100% speed constantly. In a home environment, this results in noise levels around 82dB—comparable to a lawnmower.

To solve this, the fan pinout can be modified. By identifying the JST PH2.0 connector on the adapter and using a 2.54mm male to PH2.0 female jumper cable, the fan can be wired directly to a motherboard fan header. This allows for PWM control, enabling the fan to run at 10% speed while keeping the GPU under 50°C under full load, effectively silencing the system.

Software Orchestration with NixOS

Combining two different GPU architectures (Ada Lovelace for the RTX 4080 and Volta for the V100) creates a driver compatibility challenge. NVIDIA dropped Volta support in driver branch 560, meaning a legacy driver is required to support both cards simultaneously.

Using NixOS simplifies this dependency management. To get the system running, the following configuration is necessary:

  • Kernel: Linux 6.6 (newer kernels may not be supported by the legacy driver).
  • Driver: nvidiaPackages.legacy_535 (branch 550.x).
  • CUDA: Version 12.2 (pulled from an older nixpkgs release, as current versions ship 12.6+).
  • X Server: Must be enabled (services.xserver.enable = true) for the NVIDIA kernel modules to load, even on a headless server.

Performance and Model Execution

With a combined 32GB of VRAM, models like Qwen3.6-27B-MTP (quantized at Q5_K_M) can be fully offloaded to the GPUs. Using llama.cpp with tensor splitting (-ts 1.0,1.0), the model is distributed across both the 4080 and the V100.

Key Performance Metrics:

  • Inference Speed: ‹32 tokens/second
  • Prompt Processing: 133–160 tokens/second
  • Context Window: 128k tokens

One significant advantage of this specific model is Multi-Token Prediction (MTP), which allows the model to predict several future tokens at once. When MTP hits its stride, generation speeds can jump to 50–60 tokens/second, particularly for structured output like code.

Vision Capabilities

By adding a multimodal projector file (mmproj), the setup supports image input. The vision encoder compresses image pixels into vectors that live in the same mathematical space as text tokens, allowing the LLM to reason about images with only an additional 1GB of VRAM overhead.

Critical Considerations and Trade-offs

While the "capability-per-pound" is high, this approach is not without drawbacks:

  1. Prefill Latency: As noted by community members, while 32 tok/s is great for chat, the prefill speed (prompt processing) can be a bottleneck for agentic workloads. Processing 100,000 tokens at 150 tok/s still results in a wait of over 11 minutes.
  2. Hardware Limitations: The V100 does not support bfloat16, which is a standard in many modern models. While not a dealbreaker for local experimentation, it is a hardware limitation to keep in mind.
  3. Power and Heat: Datacenter GPUs have high idle power draws (25–50W). Without active cooling or a custom water block, they will overheat almost instantly.
  4. Stability: Some users report ACPI enumeration issues where the GPU may disappear after a warm reboot, requiring a full cold power cycle to restore.

Conclusion

For those willing to tinker with drivers, kernels, and hardware jumpers, the secondhand datacenter market offers an incredible value proposition. For roughly £200, you can transform a mid-range gaming PC into a legitimate AI workstation capable of running models that rival proprietary cloud services, all while maintaining complete data privacy and zero per-token costs.

Sources