Llama 2 Release and Hugging Face Integration

Meta has released Llama 2, a family of state-of-the-art open-access large language models (LLMs) available for commercial use. This release provides the community with pretrained and fine-tuned models that offer a high-performance open alternative to closed-source chatbots.

Llama 2 Model Specifications and Improvements

Llama 2 consists of a family of models ranging from 7B to 70B parameters (specifically 7B, 13B, and 70B). These models introduce several technical improvements over the original Llama 1 series:

  • Increased Training Data: Pretrained models were trained on 40% more tokens than Llama 1.
  • Expanded Context Window: The context length has been increased to 4k tokens.
  • Optimized Inference: The 70B model utilizes grouped-query attention to enable faster inference.

Llama 2-Chat: Dialogue Optimization

Llama 2-Chat models are fine-tuned versions of the base models, specifically optimized for dialogue applications using Reinforcement Learning from Human Feedback (RLHF). According to human evaluations, Llama 2-Chat models achieve performance comparable to ChatGPT and outperform most other open models across a variety of safety and helpfulness benchmarks.

Hugging Face Integration and Deployment

Hugging Face has integrated Llama 2 into its ecosystem to facilitate deployment and fine-tuning. Available integrations include:

  • Transformers Library: Full support starting with release 4.31, allowing users to leverage safetensors, 4-bit quantization via bitsandbytes, and Parameter-Efficient Fine-Tuning (PEFT).
  • Text Generation Inference (TGI): A production-ready container supporting continuous batching, token streaming, and tensor parallelism for multi-GPU setups.
  • Inference Endpoints: Managed deployment options with recommended hardware configurations:
    • 7B models: 1x Nvidia A10G (GPU medium).
    • 13B models: 1x Nvidia A100 (GPU xlarge).
    • 70B models: 2x Nvidia A100 with bitsandbytes quantization (GPU 2xlarge) or 4x Nvidia A100 (GPU 4xlarge).

Efficient Fine-Tuning with PEFT and QLoRA

Llama 2 can be fine-tuned on consumer-grade hardware using QLoRA and the SFTTrainer from the trl library. For example, the 7B variant can be instruction-tuned on a single NVIDIA T4 GPU (16GB) available on Google Colab. This process allows users to merge LoRA weights into the model weights and save them as safetensor weights for production deployment via TGI or Inference Endpoints.

Prompting Framework for Llama 2-Chat

Llama 2-Chat requires a specific prompt template to maintain consistency with its training procedure. The template utilizes special tokens to separate system instructions from user messages:

First Turn Template: `[INST] <> {{ system_prompt }} <>

{{ user_message }} [/INST]`

Multi-turn Conversation Template: `[INST] <> {{ system_prompt }} <>

{{ user_msg_1 }} [/INST] {{ model_answer_1 }} [INST] {{ user_msg_2 }} [/INST]`

Because Llama 2 is an open-access model, users have full control over the system_prompt, allowing them to define the assistant's personality and behavior without the restrictions found in closed-source APIs. This capability is particularly useful for researchers studying the impact of different prompts on model behavior and safety.

Sources