Retrieval Augmented Generation with Hugging Face Transformers and Ray
Hugging Face has integrated Ray into the Retrieval Augmented Generation (RAG) model's document retrieval mechanism. This integration reduces retrieval call latency by up to 2x and improves the scalability of distributed fine-tuning for knowledge-intensive NLP tasks.
Understanding Retrieval Augmented Generation (RAG)
Retrieval Augmented Generation (RAG) is a sequence-to-sequence (seq2seq) architecture that enhances its knowledge by retrieving contextual documents from an external knowledge base, such as a Wikipedia text corpus, during execution. Unlike standard models that rely solely on parameters baked into the model, RAG combines these internal parameters with information retrieved from external passages to produce an output. This dual-source approach allows RAG to outperform other state-of-the-art models in knowledge-intensive tasks like question answering.
Scaling Distributed Fine-Tuning with Ray
While the retrieval step is essential for RAG's performance, it introduces significant complexity during distributed fine-tuning. In data-parallel training routines, the document index is often too large for each training worker to load a replicated copy, creating a potential bottleneck.
Previously, RAG fine-tuning utilized the torch.distributed communication package for document retrieval. However, this implementation had two primary limitations:
- Synchronization Bottlenecks: The rank 0 worker was responsible for receiving inputs from all workers, performing the index query, and distributing the results back, which limited performance as the number of training workers increased.
- Framework Dependency: The retrieval process group was tied to the training process group, requiring PyTorch to be used for the training process.
By replacing torch.distributed with Ray—a Python library for general-purpose distributed and parallel programming—Hugging Face has created a framework-agnostic implementation. Using Ray's stateful actor abstractions, multiple processes separate from the training processes can load the index and handle retrieval queries concurrently, removing the rank 0 bottleneck.
Performance Benchmarks
Integrating Ray leads to superior retrieval performance during multi-GPU fine-tuning compared to the torch.distributed implementation. As the number of GPUs increases, the performance gap widens, and increasing the number of Ray retrieval processes further optimizes speed.
| Implementation | 2 GPU | 3 GPU | 4 GPU |
|---|---|---|---|
torch.distributed |
2.12 sec/retrieval | 2.62 sec/retrieve | 3.438 sec/retrieve |
| Ray (2 retrieval processes) | 1.49 sec/retrieve | 1.539 sec/retrieve | 2.029 sec/retrieve |
| Ray (4 retrieval processes) | 1.145 sec/retrieve | 1.484 sec/retrieve | 1.66 sec/retrieve |
Note: Benchmarks were conducted over 500 training steps with a per-GPU batch size of 8, measuring the time to retrieve contextual documents on the rank 0 worker.
Implementation and Usage
Users can implement Ray-based retrieval using the PyTorch Lightning-based fine-tuning script provided by Hugging Face. The process involves installing ray and transformers, and executing the following configuration in the fine-tuning script:
- Distributed Retriever: Set to
ray. - Retrieval Workers: Specified via the
--num_retrieval_workersflag.
For those seeking further optimization, the integration with Ray Tune can be used for scalable hyperparameter tuning to improve model accuracy.