Hugging Face NVIDIA NIM API (serverless) launch and deprecation

TL;DR

Hugging Face launched the NVIDIA NIM API (serverless) for Enterprise Hub users, providing pay‑as‑you‑go, serverless inference of open‑source LLMs on NVIDIA DGX Cloud H100 GPUs via a standard OpenAI‑compatible API. The service simplifies access to high‑performance generative AI while charging per‑second GPU usage.

What is the NVIDIA NIM API (serverless)?

The NVIDIA NIM API (serverless) is a new Hugging Face Hub service that lets Enterprise Hub organizations run inference for popular open‑source generative models—such as Llama 3 and Mistral—on NVIDIA’s DGX Cloud H100 Tensor Core GPUs without managing any infrastructure. The service is built on the existing collaboration between Hugging Face and NVIDIA and complements the previously announced Train on DGX Cloud offering.

How the service works

Prerequisites

  • Membership in a Hugging Face Enterprise Hub organization.
  • A fine‑grained access token scoped to the organization (created via the Hugging Face Access Tokens page).

Step‑by‑step workflow

  1. Create a fine‑grained token – Generate a token with “org permissions” only; no additional scopes are required.
  2. Locate a supported NIM model – Browse the NVIDIA NIM Collection or the model’s Hub page. For example, the meta-llama/Meta-Llama-3-8B-Instruct card shows a Deploy menu with a NVIDIA NIM API (serverless) option that provides ready‑made code snippets.
  3. Send inference requests – Use the OpenAI‑compatible SDK (openai Python package) with the base URL https://huggingface.co/api/integrations/dgx/v1 and the fine‑grained token as the API key. The API currently supports chat.completions.create and models.list endpoints.
from openai import OpenAI

client = OpenAI(
    base_url="https://huggingface.co/api/integrations/dgx/v1",
    api_key="YOUR_FINE_GRAINED_TOKEN_HERE"
)

chat = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Count to 500"}
    ],
    stream=True,
    max_tokens=1024
)

for chunk in chat:
    print(chunk.choices[0].delta.content, end="")

The models.list endpoint can be used to enumerate all currently available NIM models.

Supported models and pricing

The service runs exclusively on NVIDIA H100 GPUs, priced at $8.25 per hour (≈ $0.0023 per second). Costs are calculated per request based on the number of GPUs required and the request latency.

Model ID H100 GPUs Typical latency* (500‑in, 100‑out) Approx. cost per request
meta-llama/Meta-Llama-3-8B-Instruct 1 1 s $0.0023
meta-llama/Meta-Llama-3-70B-Instruct 4 2 s $0.0184
meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 8 5 s $0.0917

*Latency measured on DGX Cloud H100 hardware.

Additional supported models include various Mixtral and Mistral variants, each mapped to a specific GPU count (e.g., Mixtral‑8x22B‑Instruct‑v0.1 uses 8 H100s). Usage fees are billed to the Enterprise Hub organization’s monthly billing cycle and can be inspected in the Hub’s billing settings.

Technical significance

  • Serverless abstraction – Developers no longer need to provision, scale, or maintain GPU clusters; inference is invoked via a simple HTTP call.
  • Standardized API – By adopting the OpenAI API schema, existing tooling and libraries can be reused without code changes.
  • Cost transparency – Per‑second GPU pricing makes budgeting predictable, especially for bursty workloads.
  • Performance edge – Leveraging NVIDIA DGX Cloud H100 GPUs and upcoming TensorRT‑LLM integration promises lower latency and higher throughput than generic cloud GPU offerings.

Future roadmap

Hugging Face announced plans to extend the API beyond chat.completions.create and models.list, adding more endpoint capabilities and expanding the catalog of supported models. A forthcoming integration of NVIDIA TensorRT‑LLM into Hugging Face’s Text Generation Inference (TGI) framework is expected to further boost inference speed, with benchmarks and best‑practice guides slated for release.

Deprecation notice

Update: This service was deprecated and became unavailable on April 10 2025. Users are directed to adopt the newer Inference Providers framework for continued serverless inference capabilities.

The information above reflects the state of the service as announced on July 29 2024. Subsequent deprecation means the service is no longer operational, but the architectural concepts remain relevant for understanding Hugging Face’s approach to serverless AI inference.

Sources