Hugging Face: Optimizing Cost and Latency for 1 Billion Classifications

Processing over one billion classifications or embeddings per day is primarily a financial challenge. Hugging Face has developed a methodology to calculate and optimize cost and latency for large-scale inference, demonstrating that the right combination of hardware, batch size, and parallelism can significantly reduce cloud expenditures.

Cost Benchmarks for 1 Billion Inputs

Using NVIDIA L4 GPUs ($0.8/hr), the cost to process 1 billion inputs varies drastically based on the model architecture and task:

Use Case Model Hardware Cost of 1B Inputs
Classification lxyuan/distilbert-base-multilingual-cased-sentiments-student nvidia-L4 $253.82
Embedding Alibaba-NLP/gte-modernbert-base nvidia-L4 $409.44
Vision-Embedding vidore/colqwen2-v1.0-merged nvidia-L4 $44,496.51

Technical Optimization Framework

To achieve these costs, Hugging Face utilized a specific stack for deployment and load testing:

  • Inference Server: Infinity, chosen for its ability to serve multimodal embeddings and support for various hardware (AMD, Nvidia, CPU, Inferentia).
  • Deployment: Hugging Face Inference Endpoints for flexible hardware selection and the Hugging Face Hub Library for programmatic deployment.
  • Load Testing: k6 from Grafana, using the shared-iterations executor to simulate parallel client requests and measure throughput and P95 latency.

Key Optimization Parameters

Efficiency is driven by three primary variables:

  1. INFINITY_BATCH_SIZE: Determines how many documents undergo a forward pass. Too low leads to GPU underutilization; too high exceeds GPU capacity.
  2. Virtual Users (VUs): Simulates parallel client requests to ensure the batch size is fully utilized.
  3. Hardware Choice: The NVIDIA L4 was found to provide the best performance-to-cost ratio for modern workloads, outperforming the NVIDIA T4 and CPUs.

Use Case Analysis

Text Classification

For lightweight classification tasks, DistilBERT was evaluated. The most cost-effective configuration for 1 billion inputs ($253.82) utilized an NVIDIA L4 GPU with a batch size of 64 and 448 VUs using the default Infinity image.

Text Embeddings

ModernBERT was used for embedding tasks due to its support for Flash-Attention-2 and an 8k context window. The optimal configuration for 1 billion inputs ($409.44) used an NVIDIA L4 GPU with a batch size of 32 and 256 VUs.

Vision Embeddings

Vision-embedding tasks using ColQwen2 are significantly more expensive due to the model's 2.21B parameter size and ColBERT-style multi-vector representations, which return a vector for each token rather than one vector per input. The cheapest configuration for 1 billion inputs was $44,496.51, utilizing an NVIDIA L4 with a batch size of 4 and 4 VUs.

Core Findings and Trade-offs

  • Hardware Efficiency: The NVIDIA L4 is the preferred choice for modern encoder workloads over the T4.
  • The Latency-Cost Trade-off: Increasing allowed latency typically allows for higher throughput, which reduces overall cost. This relationship can be visualized via a Pareto curve.
  • Vision Cost Penalty: Image-based retrieval is approximately two orders of magnitude more expensive than text-based tasks due to larger model sizes, complex architectures (including decoders), and higher API/egress costs for image data.
  • Scaling Strategy: To scale horizontally, once a single-GPU baseline is established, throughput can be increased by adding replica GPUs.

Sources