Running Vicuna 13B on AMD GPUs with ROCm

TL;DR

Hugging Face has demonstrated that the Vicuna 13B open-source chatbot can be run on a single AMD GPU by leveraging ROCm (Radeon Open Compute) and GPTQ 4-bit quantization. This approach reduces the GPU memory requirement from approximately 28GB (for fp16) to 7.52GB, enabling the model to run on consumer-grade hardware like the Radeon RX6900XT.

The Vicuna 13B Model

Vicuna is an open-source chatbot with 13 billion parameters developed by a collaboration between UC Berkeley, CMU, Stanford, and UC San Diego. It was created by fine-tuning a LLaMA base model using approximately 70,000 user-shared conversations from ShareGPT.com. Initial assessments using GPT-4 as a reference indicate that Vicuna-13B achieves over 90% of the quality of OpenAI's ChatGPT, with a total training cost of approximately $300.

Memory Optimization via GPTQ Quantization

Running Vicuna-13B in fp16 precision requires roughly 28GB of GPU RAM, which exceeds the capacity of many single GPUs. To address this, Hugging Face utilizes GPTQ (accurate post-training quantization), which allows models with more than 10B parameters to maintain comparable accuracy to fp16 while using 3-bit or 4-bit precision.

Quantization is critical not only for memory footprint but also for latency. Because token generation in LLMs is typically limited by memory bandwidth rather than raw computation (TFLOPs), quantized models do not suffer from increased token generation latency when the GPU is in a memory-bound state.

Technical Implementation on AMD Hardware

System Requirements

To deploy Vicuna 13B on AMD hardware, the following environment is required:

  • Hardware: An AMD GPU supporting ROCm (tested on Instinct MI210 and Radeon RX6900XT).
  • OS: Linux-based, preferably Ubuntu 18.04, 20.04, or 22.04.
  • Software: ROCm 5.4.3, PyTorch 2.0, and Python 3.6+.
  • Environment: Conda or Docker.

Deployment Workflow

  1. ROCm Installation: Install the amdgpu-install package and configure the system for hiplibsdk, rocm, and dkms.
  2. Containerization: Use a ROCm-compatible PyTorch Docker image (e.g., rocm/pytorch:rocm5.4.2_ubuntu20.04_py3.8_pytorch_2.0.0_preview).
  3. Model Acquisition: Download 4-bit quantized Vicuna-13b weights from Hugging Face or quantize a floating-point model using the GPTQ-for-LLaMa repository.
  4. Inference: Use the llama_inference.py script with the --wbits 4 and --groupsize 128 flags to generate text.
  5. API Integration: The model can be exposed via a Web API server using FastChat, requiring a controller, a model worker, and a Gradio web server.

Performance and Accuracy Metrics

Memory Consumption

Quantization significantly reduces the VRAM footprint. For a 13B model, the memory requirement drops from over 28GB in fp16 to 7.52GB in 4-bit quantization, occupying only 46% of the 16GB DDR available on a Radeon RX6900XT.

Accuracy (Perplexity)

Accuracy was measured using Perplexity (PPL) on 2,048 examples from the C4 dataset. The results show that 4-bit quantized models (using either fp32 or fp16 Matmul) maintain high accuracy relative to the fp16 baseline.

Latency

Token generation latency was measured across different configurations. The data indicates that the 4-bit quantized versions do not introduce significant latency penalties compared to the fp16 baseline.

Appendix: Manual Model Quantization

For users wishing to create their own quantized models, the process involves:

  1. Applying Delta Weights: Since Vicuna is released as delta weights, they must be applied to the original LLaMA base model using the fastchat.model.apply_delta command.
  2. Quantization Process: Using the GPTQ-for-LLaMa repository, the model is quantized to 4-bit precision using the C4 dataset for calibration, saving the result as a .safetensors file.

Sources