SetFit Inference Acceleration with πŸ€— Optimum Intel on Xeon

TL;DR

Hugging Face has demonstrated that SetFit models can achieve up to a 7.8x increase in inference throughput on Intel Xeon CPUs by utilizing post-training static quantization through the πŸ€— Optimum Intel library. This optimization allows for production-grade deployment of few-shot learning models with minimal impact on accuracy.

SetFit: Efficient Few-Shot Learning

SetFit is a framework designed for few-shot fine-tuning of Sentence Transformers, specifically addressing the challenge of limited labeled training data. It provides two primary advantages over Large Language Model (LLM) based methods:

  • Elimination of Prompts: Unlike in-context learning with LLMs, which requires brittle, handcrafted prompts, SetFit generates embeddings directly from a small number of labeled examples.
  • Training Efficiency: SetFit is typically an order of magnitude faster to train and run inference with than LLMs like GPT-3.5 or Llama 2.

In terms of performance, SetFit has been shown to outperform GPT-3.5 in 3-shot prompting and outperform 3-shot GPT-4 on the Banking 77 financial intent dataset when using 5 shots.

Accelerating Inference with πŸ€— Optimum Intel

To accelerate SetFit inference on Intel CPUs, Hugging Face utilizes Optimum Intel, an open-source library that leverages Intelˆ Advanced Vector Extensions 512 (AVX-512), Vector Neural Network Instructions (VNNI), and Advanced Matrix Extensions (AMX). These hardware accelerations include built-in BFloat16 (bf16) and int8 GEMM accelerators in every core.

Post-Training Static Quantization (PTQ)

The core optimization technique used is Post-Training Static Quantization (PTQ) via the Intel Neural Compressor (INC). Quantization reduces the number of bits used to represent weights and activations (e.g., converting high-precision numbers to INT8), which reduces the memory footprint and enables faster computation.

PTQ is particularly effective because it reduces latency and memory usage without requiring additional training, needing only a small unlabeled calibration set (typically around 100 samples) to represent the distribution of unseen data.

Benchmarking and Performance Results

Performance was evaluated using a bge-small model on the sst2 dataset, comparing the original PyTorch/Transformers fp32 implementation against an IPEX-bfloat16 version and the quantized Optimum-int8 version.

Latency and Model Size

At a batch size of 1, the optimized Optimum-int8 model demonstrated significant improvements:

  • Latency Reduction: A 3.45x reduction in latency (4.55 ms vs 15.69 ms for the fp32 model).
  • Model Size Reduction: The model size shrunk by 2.85x (44.65 MB vs 127.32 MB).
  • Accuracy Preservation: Accuracy remained virtually unchanged, moving from 88.4% (fp32) to 88.1% (int8).

Throughput Gains

While latency improvements are substantial, the most significant gains appear in throughput. When comparing the highest achievable throughput across various batch sizes, the Optimum-int8 model is 7.8x faster than the original transformers fp32 model.

Metric bge-small (transformers) bge-small (ipex-bfloat16) bge-small (optimum-int8)
Model Size 127.32 MB 63.74 MB 44.65 MB
Accuracy (test set) 88.4% 88.4% 88.1%
Latency (bs=1) 15.69 +/- 0.57 ms 5.67 +/- 0.66 ms 4.55 +/- 0.25 ms

Implementation Workflow

To implement these optimizations, the process follows two primary steps:

  1. Quantization: Use INCQuantizer from optimum.intel with a PostTrainingQuantConfig set to "static" approach and "ipex" backend.
  2. Deployment: Wrap the quantized model body using an IPEXModel and integrate it into the SetFit model structure for inference.

Sources