Deploying LLMs with Hugging Face Inference Endpoints

Hugging Face Inference Endpoints provides a managed SaaS solution for deploying open-source Large Language Models (LLMs) as production-ready APIs. This service eliminates the need for manual infrastructure management and MLOps, enabling developers to deploy models like Falcon, (Open-)LLaMA, X-Gen, StarCoder, and RedPajama efficiently.

Core Capabilities of Inference Endpoints

Inference Endpoints simplify the transition from model selection to production deployment through several key technical features:

  • Simplified Deployment: Models can be deployed as APIs with a few clicks, removing the complexity of managing underlying infrastructure.
  • Cost Optimization: The service includes an automatic "scale-to-zero" capability, which reduces costs by scaling down infrastructure when the endpoint is not in use. Users pay based on the uptime of the endpoint.
  • Enterprise-Grade Security: Endpoints can be deployed as secure offline instances accessible only via direct VPC connections. The service is SOC2 Type 2 certified and offers BAA and GDPR data processing agreements.
  • LLM-Specific Optimizations: To ensure high throughput and low latency, the service utilizes Text Generation Inference (TGI), which incorporates Paged Attention and Flash Attention, along with custom transformers code.
  • Broad Task Support: The platform provides out-of-the-box support for 🤗 Transformers, Sentence-Transformers, and Diffusers, while allowing customization for advanced tasks such as speaker diarization.

Deploying and Testing LLMs

Deploying a model, such as tiiuae/falcon-40b-instruct, involves selecting the model repository, cloud provider, and region. While the system suggests instance types based on model size (e.g., 4x NVIDIA T4 GPUs), users can manually select higher-performance hardware, such as a single NVIDIA A100 GPU, for optimal performance.

Once deployed, endpoints can be tested using the built-in Inference Widget for manual requests or via cURL commands. To control the behavior of the text generation, the API supports a wide range of parameters in the payload, including:

  • Temperature: Controls randomness (default 1.0).
  • max_new_tokens: Sets the maximum number of tokens to generate (default 20, max 512).
  • repetition_penalty: Controls the likelihood of token repetition.
  • top_k and top_p: Manage vocabulary filtering and nucleus sampling.
  • do_sample: Toggles between sampling and greedy decoding (default false).
  • stop: A list of tokens that trigger the end of generation.

Implementing Response Streaming

To improve user experience by reducing perceived latency, Inference Endpoints support streaming tokens as they are generated. This can be implemented using specific libraries for different languages:

Python Implementation

Using the huggingface_hub library and the InferenceClient, developers can set stream=True in the text_generation method. This allows the application to iterate over the generated tokens and yield them to the user in real-time, while filtering out special tokens or stopping at defined stop sequences.

JavaScript Implementation

Using the @huggingface/inference library and the HfInferenceEndpoint class, developers can utilize the textGenerationStream method. This approach uses an asynchronous iterator (for await...of) to process and display tokens as they are received from the endpoint.

Sources