Llama 2 on Amazon SageMaker Benchmark
Hugging Face has released a comprehensive benchmark analyzing over 60 deployment configurations for Llama 2 on Amazon SageMaker. This analysis provides data-driven recommendations for organizations to optimize their generative AI deployments based on whether they prioritize cost-efficiency, maximum throughput, or minimum latency.
Deployment Infrastructure and Methodology
The benchmark utilizes the Hugging Face LLM Inference Container, which is powered by Text Generation Inference (TGI). TGI is an open-source solution that enables high-performance text generation through dynamic batching and Tensor Parallelism.
Benchmark Parameters
To evaluate real-world performance, Hugging Face tested three Llama 2 model sizes (7B, 13B, and 70B parameters) across the following variables:
- Instance Types: NVIDIA A10G GPUs (g5.2xlarge, g5.12xlarge, g5.48xlarge) and NVIDIA A100 40GB GPUs (p4d.24xlarge).
- Load Levels: Concurrent request volumes of 1, 5, 10, and 20.
- Quantization: Performance was compared using standard weights versus GPTQ 4-bit quantization. GPTQ reduces model weights from 32 bits to 3-4 bits, significantly lowering memory requirements and allowing larger models (such as Llama 2 13B) to run on a single GPU.
Performance Metrics
Performance was measured using two primary metrics:
- Throughput: The number of tokens generated per second.
- Latency: The time required to generate a single token (measured in milliseconds per token).
Optimal Deployment Recommendations
Based on the benchmark results, Hugging Face identifies three primary deployment strategies depending on the business objective.
Most Cost-Effective Deployment
For users prioritizing the lowest cost per token, GPTQ quantization is the most effective strategy. This allows for the deployment of Llama 2 13B on a single GPU instance.
| Model | Quantization | Instance | Concurrent Requests | Latency (ms/token) | Throughput (tokens/sec) | Cost ($/h) | Cost per 1M Tokens |
|---|---|---|---|---|---|---|---|
| Llama 2 7B | GPTQ | g5.2xlarge | 5 | 34.25 | 120.09 | $1.52 | $3.50 |
| Llama 2 13B | GPTQ | g5.2xlarge | 5 | 56.24 | 71.71 | $1.52 | $5.87 |
| Llama 2 70B | GPTQ | ml.g5.12xlarge | 5 | 138.35 | 33.33 | $7.09 | $59.08 |
Best Throughput Deployment
To maximize the volume of tokens processed per second, higher-end GPU instances and higher concurrency are required. The highest overall throughput recorded was 688 tokens/sec for Llama 2 13B on the ml.p4d.12xlarge instance.
| Model | Quantization | Instance | Concurrent Requests | Latency (ms/token) | Throughput (tokens/sec) | Cost ($/h) | Cost per 1M Tokens |
|---|---|---|---|---|---|---|---|
| Llama 2 7B | None | ml.g5.12xlarge | 20 | 44.00 | 449.94 | $7.09 | $3.97 |
| Llama 2 13B | None | ml.p4d.12xlarge | 20 | 67.40 | 668.02 | $37.69 | $15.67 |
| Llama 2 70B | None | ml.p4d.24xlarge | 20 | 59.80 | 321.54 | $37.69 | $32.56 |
Best Latency Deployment
For real-time applications such as chat interfaces, minimizing the time to generate a single token is critical. The lowest latency was achieved by Llama 2 7B on the ml.g5.12xlarge instance.
| Model | Quantization | Instance | Concurrent Requests | Latency (ms/token) | Throughput (tokens/sec) | Cost ($/h) | Cost per 1M Tokens | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | Llama 2 7B | None | ml.g5.12xlarge | 1 | 16.81 | 61.46 | $7.09 | $32.05 | | Llama 2 13B | None | ml.g5.12xlarge | 1 | 21.00 | 47.16 | $7.09 | $41.76 | | Llama 2 70B | None | ml.p4d.24xlarge | 1 | 41.35 | 24.51 | $37.69 | $427.05 |
Conclusions
The benchmark demonstrates that deployment efficiency for Llama 2 on Amazon SageMaker depends heavily on the choice of quantization and instance type. GPTQ 4-bit quantization significantly lowers the barrier to entry for cost-effective deployment, while high-performance NVIDIA A100 instances are necessary for maximizing throughput.