Hugging Face Inference for PROs

Hugging Face has launched Inference for PRO, a community offering that provides PRO users with access to curated, ultra-fast API endpoints for high-performance models. This service is designed to facilitate rapid experimentation and prototyping without requiring users to manage their own infrastructure.

Accelerated Access to State-of-the-Art Models

Inference for PROs provides exclusive API endpoints powered by text-generation-inference (TGI) for a curated list of powerful models. While the free Inference API remains available to all users for testing over 200,000+ models, PRO users receive higher rate limits and access to specific high-performance models.

Supported Models and Capabilities

Model	Size	Context Length	Primary Use
Meta Llama 3 Instruct	8B, 70B	8k tokens	Chat
Mixtral 8x7B Instruct	45B MOE	32k tokens	High-performance chat
Nous Hermes 2 Mixtral 8x7B DPO	45B MOE	32k tokens	Advanced Mixtral tuning
Zephyr 7B β	7B	4k tokens	Chat (7B weight)
Llama 2 Chat	7B, 13B	4k tokens	Conversational AI
Mistral 7B Instruct v0.2	7B	4k tokens	Chat (7B weight)
Code Llama Base	7B, 13B	4k tokens	Code autocomplete/infill
Code Llama Instruct	34B	16k tokens	Conversational code assistant
Stable Diffusion XL	3B UNet	-	Image generation
Bark	0.9B	-	Text-to-audio generation

Technical Implementation and Integration

Integration is handled via HTTP POST requests to the model's API endpoint using a PRO account authentication token. Developers can use curl or the InferenceClient utility from the huggingface_hub Python library for streamlined access.

Messages API and OpenAI Compatibility

All text generation models now support the Messages API, making them compatible with OpenAI client libraries, as well as frameworks like LangChain and LlamaIndex. This allows developers to use the openai Python client by pointing the base_url to https://api-inference.huggingface.co/v1/.

Advanced Generation Control

Users can fine-tune model outputs using several generation parameters:

• Text Generation:
  • do_sample: Toggles between deterministic greedy search (False) and probabilistic sampling (True).
  • temperature: Controls variation; lower values are recommended for code, higher for open-ended text.
  • top_k and top_p: Limit the sampling pool to the most probable tokens.
  • repetition_penalty: Reduces the likelihood of repeated words.
  • max_new_tokens: Sets the maximum length of the generated sequence.
• Image Generation (SDXL):
  • negative_prompt: Defines content to exclude from the image.
  • guidance_scale: Controls how strictly the model adheres to the prompt.
  • num_inference_steps: Determines the number of denoising steps (typically 20-50).

Specialized Inference Tasks

Code Infilling

Using Code Llama, users can perform "infilling" by providing prefix and suffix sequences. The model predicts the content that should fit between them using the format: <PRE> {prefix} <SUF>{suffix} <MID>.

Token Streaming

To reduce perceived latency and improve user experience, the API supports token streaming. By passing stream=True in InferenceClient or using the -N flag with curl, tokens are returned one by one as they are generated.

Caching

Recent results are cached by default to improve performance. To force a fresh generation regardless of sampling settings, users can add the HTTP header x-use-cache: 0 to their requests.

Usage Recommendations

Inference for PROs is intended for experimentation and prototyping. For heavy production applications, Hugging Face recommends using Inference Endpoints, which provide dedicated infrastructure.