Hugging Face Chat Templates

Hugging Face introduces chat templates to prevent silent performance degradation

Hugging Face has introduced a chat_template attribute for tokenizers to solve the problem of "silent performance killers"—severe model degradation that occurs when a chat model is prompted using a format different from the one used during its training. By storing the formatting logic as a Jinja template within the tokenizer itself, developers can ensure that conversation histories are converted into the exact string format the model expects, eliminating distribution shifts caused by incorrect formatting.

The problem of distribution shift in chat formatting

In standard language models, loading a tokenizer and model from the same checkpoint typically prevents distribution shifts. However, chat models require a sequence of messages (containing roles like "user", "assistant", and "system") to be converted into a single tokenizable string. Because there is no industry standard for this conversion, different models use wildly different formats, such as:

  • Simple labels: User: [text] \n Bot: [text]
  • Special tokens: [USER] [text] [/USER] \n [ASST] [text] [/ASST]
  • Boundary tokens: <|im_start|>user \n [text]<|im_end|>

Using the wrong format is a "silent error" because it does not trigger a Python exception or a loud failure; instead, the model simply performs significantly worse, making the issue difficult to debug.

Technical implementation via Jinja templates

Chat templates are implemented as Jinja template strings saved and loaded with the tokenizer. This approach was chosen over simpler systems (like per-role prefixes and suffixes) because templating is flexible enough to support all known message formats and allows for the logic and checks to be encoded directly into the template.

Example Template Logic

For a format using boundary tokens, a Jinja template might look like this:

{% for message in messages %}
    {{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}
{% endfor %}

Integration with the Transformers library

Chat templates are integrated directly into the tokenizer to maintain the principle that preprocessing information should stay with the model's tokenization logic.

Transition from class-level formatting

Previously, chat formatting was handled at the class level (e.g., all LLaMA checkpoints used the same hardcoded logic in the transformers library). To maintain backward compatibility, model classes now have default chat templates. However, Hugging Face strongly recommends explicitly setting a chat_template on every chat model to avoid fragility and ensure that future changes to default templates do not break the model's performance.

Usage and deployment

Developers can apply these templates using the tokenizer.apply_chat_template() method. If a tokenizer lacks a chat_template attribute, it will fall back to the class default, which may lead to the silent bugs mentioned above. Users are encouraged to identify the correct format from model cards and submit pull requests to add the chat_template attribute to checkpoints on the Hugging Face Hub.

Recommendations for new models

While Hugging Face acknowledges that a single standard format would be ideal, the existing variety of trained models makes a hardcoded standard impossible. For those training new chat models, Hugging Face recommends the ChatML format created by OpenAI, which uses <|im_start|> and <|im_end|> tokens. This format is flexible with roles and can be implemented with a single-line template assignment:

tokenizer.chat_template = "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}"

Sources