Hugging Face TGI Messages API Release

Hugging Face has introduced the Messages API for Text Generation Inference (TGI) starting with version 1.4.0. This update provides compatibility with the OpenAI Chat Completion API, allowing developers to transition from OpenAI models to open LLMs with minimal code changes.

OpenAI Compatibility for Open LLMs

TGI version 1.4.0 and Inference Endpoints now support a Messages API that is compatible with the OpenAI Chat Completion API. This compatibility allows users to swap OpenAI models for open LLMs by using existing OpenAI client libraries or third-party orchestration tools like LangChain and LlamaIndex.

According to Johnny Crupi, CTO at Ryght:

"The new Messages API with OpenAI compatibility makes it easy for Ryght's real-time GenAI orchestration platform to switch LLM use cases from OpenAI to open models. Our migration from GPT4 to Mixtral/Llama2 on Inference Endpoints is effortless, and now we have a simplified workflow with more control over our AI solutions."

Technical Implementation and Requirements

Deployment via Inference Endpoints

Users can deploy models—such as Nous-Hermes-2-Mixtral-8x7B-DPO—to Inference Endpoints using TGI. Deployment can be managed via the Hugging Face UI or programmatically using the huggingface_hub Python library. When using the Hub library, endpoints scale to zero after 15 minutes of idle time by default to optimize costs.

Integration with Client Libraries

To integrate an Inference Endpoint with OpenAI client libraries, users must append /v1/ to the endpoint URL and provide a valid Hugging Face user token as the API key.

  • Python Client: The OpenAI client is initialized with the base_url pointing to the TGI endpoint and the api_key set to the HF token. The model parameter is set to "tgi".
  • JavaScript Client: The OpenAI node library is used similarly, configuring the baseURL and apiKey to point to the TGI infrastructure.

Framework Integration (LangChain and LlamaIndex)

  • LangChain: Users can utilize the ChatOpenAI class by passing the TGI endpoint URL and HF token, allowing existing RAG pipelines to function with open models by changing a single line of code.
  • LlamaIndex: Integration is achieved via the OpenAILike class. This requires configuring parameters such as is_chat_model=True, is_local=False, and is_function_calling_model=False, while ensuring the context_window matches the MAX_TOTAL_TOKENS setting of the endpoint.

Capabilities and Limitations

Supported Parameters

The Messages API currently supports the following chat completion parameters:

  • stream
  • max_tokens
  • frequency_penalty
  • logprobs
  • seed
  • temperature
  • top_p

Current Constraints

  • Function Calling: The Messages API does not currently support function calling.
  • Model Requirements: The API only works for LLMs that have a chat_template defined in their tokenizer configuration (e.g., Mixtral 8x7B Instruct).

Strategic Implications

By providing OpenAI-compatible endpoints, Hugging Face enables users to gain complete control and transparency over their models and data, eliminate reliance on third-party rate limits, and fully customize their AI systems to specific needs.

Sources