Hugging Face Accelerate: Running Large Models with PyTorch

Hugging Face Accelerate allows users to load and run very large language models (LLMs) that exceed the available RAM or GPU memory of consumer hardware. By utilizing specific PyTorch features and a modified loading pipeline, Accelerate distributes model weights across available hardware resources, including multiple GPUs, CPU RAM, and disk storage.

Overcoming Memory Constraints in Model Loading

Traditional PyTorch model loading follows a linear process: creating the model, loading weights into a state_dict in memory, loading those weights into the model, and moving the model to a device. For very large models, this process is prohibitively expensive in terms of RAM. For example, a model with 6.7 billion parameters in float32 precision requires approximately 26.8GB of RAM just for the initial model creation, and another 26.8GB to load the state_dict, totaling over 53GB before the model even reaches a GPU.

To solve this, Accelerate implements a more memory-efficient pipeline:

  1. Create an empty model (without weights).
  2. Determine a device map to decide where each layer will reside.
  3. Load weights in small parts (shards).
  4. Load those weights into the empty model.
  5. Move weights to the designated device for inference.
  6. Repeat the process for all remaining weights.

Efficient Model Initialization via the Meta Device

Accelerate utilizes the PyTorch "meta" device (introduced in PyTorch 1.9) to instantiate models without allocating actual memory for data. Tensors on the meta device only store shape and data type information, allowing for the creation of arbitrarily large models without consuming CPU or GPU RAM.

Because rewriting every model in the Transformers library to support the device keyword would be impractical, Hugging Face developed the init_empty_weights() context manager. This allows any model to be instantiated as a "shell" on the meta device, providing the necessary structural information to calculate memory requirements without loading actual weights.

Automated Device Mapping and Resource Allocation

Accelerate uses the infer_auto_device_map function to automatically distribute model weights across available hardware. The system prioritizes resources in the following order: GPUs, then CPU RAM, and finally disk offload.

Device Map Configurations

Depending on the use case, users can choose different mapping strategies:

  • "auto" or "balanced": Distributes weights equally across all available GPUs.
  • "balanced_low_0": Distributes weights equally across GPUs but minimizes the load on the first GPU (GPU 0), which is useful when the first GPU is needed for model outputs (e.g., during text generation).
  • "sequential": Fills GPUs in order, potentially leaving later GPUs unused.

To prevent the system from splitting a single layer across multiple devices—which would break residual connections—Accelerate allows users to specify no_split_module_classes (e.g., ["OPTDecoderLayer"]).

Sharded Checkpoints for Reduced RAM Overhead

Loading a single massive state_dict file is unrealistic for most hardware; for instance, the BLOOM model (176B parameters) would require 352GB of RAM just to load the weights in bfloat16. To mitigate this, Hugging Face uses sharded checkpoints.

In a sharded checkpoint:

  • A pytorch_model.bin.index.json file maps each parameter name to a specific shard file.
  • The weights are split across multiple standard PyTorch state dict files (e.g., 72 files for BLOOM).

This approach ensures that the system only needs enough RAM to hold the largest single shard (e.g., 7.19GB for BLOOM) rather than the entire model. If GPU and CPU RAM are insufficient, Accelerate can offload weights to a specified offload_folder on the disk. The offload_state_dict=True option can further reduce RAM usage by temporarily offloading CPU-resident parts of the model while other shards are being processed.

Execution via Dynamic Hooks

To run a model split across multiple devices, Accelerate uses a mechanism inspired by PyTorch hooks. The dispatch_model function attaches hooks to every module and submodule that execute before and after each forward pass. These hooks perform the following operations:

  • Ensure all module inputs are on the same device as the weights.
  • Move weights from CPU to GPU 0 immediately before the forward pass and return them to the CPU immediately after.
  • Load weights from disk to RAM and then to GPU 0 before the forward pass, freeing the memory immediately after.

While this approach uses GPUs sequentially rather than employing complex pipeline parallelism, it enables the execution of massive models on significantly smaller hardware setups.

Sources