Optimum-NVIDIA Release Notes

Hugging Face has introduced Optimum-NVIDIA, an inference library designed to dramatically accelerate Large Language Model (LLM) inference on NVIDIA hardware. By integrating NVIDIA TensorRT-LLM software and supporting the float8 (FP8) format, the library allows users to achieve up to 28x faster throughput and 1,200 tokens per second with minimal code changes.

Technical Foundation: FP8 and TensorRT-LLM

Optimum-NVIDIA is the first Hugging Face inference library to leverage the float8 format, which is supported on NVIDIA Ada Lovelace and Hopper architectures. This format, combined with the advanced compilation capabilities of NVIDIA TensorRT-LLM, enables significant reductions in computational overhead and increases inference speed.

Users can enable FP8 quantization using a single flag (use_fp8=True). This allows for the deployment of larger models on a single GPU at higher speeds without sacrificing accuracy. The library utilizes a predefined calibration strategy by default, though users can provide custom calibration datasets and tokenization to optimize quantization for specific use cases.

Performance Benchmarks

Performance is measured across two primary metrics: First Token Latency and Throughput.

First Token Latency

First Token Latency (or Time to First Token/prefill latency) determines the responsiveness of a model. Optimum-NVIDIA provides up to 3.3x faster First Token Latency compared to stock Hugging Face transformers.

Throughput

Throughput measures the speed of token generation, particularly during batched generations. Calculated as end-to-end latency divided by the total sequence length (input and output tokens across all batches), Optimum-NVIDIA delivers up to 28x better throughput than stock transformers.

Initial evaluations on the NVIDIA H200 Tensor Core GPU indicate up to an additional 2x boost in throughput for LLaMA models compared to the NVIDIA H100 Tensor Core GPU.

Implementation and API

Optimum-NVIDIA is designed for drop-in compatibility with the Hugging Face ecosystem. Users can transition from the standard transformers library to optimum-nvidia by modifying a single line of code:

  • Pipeline API: Users can replace from transformers.pipelines import pipeline with from optimum.nvidia.pipelines import pipeline and set use_fp8=True to enable acceleration.
  • Model API: For fine-grained control over sampling parameters, users can replace from transformers import AutoModelForCausalLM with from optimum.nvidia import AutoModelForCausalLM.

Model Support and Roadmap

Currently, Optimum-NVIDIA provides peak performance for the LLaMAForCausalLM architecture and task. This means all LLaMA-based models, including fine-tuned versions, are supported out of the box.

Future updates to the library will include:

  • Expansion of support to other text generation model architectures and tasks.
  • Integration of In-Flight Batching to improve throughput during prompt streaming.
  • Support for INT4 quantization to enable the execution of even larger models on a single GPU.

Sources