ray-project/llm-applications

What it solves

This project provides a comprehensive guide and implementation for building and scaling production-ready Retrieval Augmented Generation (RAG) applications. It addresses the challenges of moving from a simple prototype to a scalable, high-performance system that can handle production workloads.

How it works

The project uses the Ray framework to scale the core components of a RAG pipeline, including data loading, chunking, embedding, indexing, and serving. It demonstrates how to implement a hybrid routing approach to balance the use of open-source LLMs (like Llama-2) and closed-source models (like GPT-4) and provides tools to evaluate the quality and performance of different configurations.

Who it’s for

Developers and AI engineers who want to learn how to build, scale, and productionize RAG-based LLM applications using Ray and Anyscale.

Highlights

• own a scalable RAG pipeline from scratch
• Scale major components like embedding and indexing
• Implement hybrid routing between OSS and closed LLMs
• Evaluation tools for optimizing retrieval and overall quality scores
• Support for serving applications in a highly available manner