Digital Green Farmer.chat: Bolstering RAG with LLM-as-a-Judge
Digital Green has developed Farmer.chat, a Retrieval-Augmented Generation (RAG) chatbot designed to provide personalized agricultural advice to smallholder farmers and extension workers. To ensure the reliability and accuracy of the information delivered from a knowledge base of 46,000 research papers, the team implemented an "LLM-as-a-judge" evaluation system to quantify performance at scale.
System Architecture for Agricultural Support
Farmer.chat utilizes a multi-component architecture to ensure responses are grounded in curated, trustworthy agricultural data.
Knowledge Base Construction
- Preprocessing: PDF documents are ingested via Scio APIs, with topics auto-categorized by geographic area and semantically grouped.
- Semantic Chunking: Text is processed into chunks where sentences with similar meanings are grouped together using small-text embeddings for cosine similarity.
- Vector Storage: Chunks are converted into vector representations and stored in QdrantDB.
RAG Pipeline and User Agent
- Information Retrieval: The system searches the vector database for text chunks matching the user's query.
- Generation: An LLM uses the retrieved chunks and the user query to generate a human-like response.
- User-facing Agent: Powered by GPT-4o, this planning agent uses a ReAct-based prompt to understand user intent, request missing information, and call execution tools. Available tools include a RAG QA endpoint, video retrieval, weather endpoints, and crop tables.
Implementing LLM-as-a-Judge for Evaluation
Because the quality of agricultural advice lacks a deterministic metric, Digital Green adopted the LLM-as-a-judge technique, using an LLM to rate outputs based on specific, predefined criteria.
Key Evaluation Metrics
- Prompt Clarity: Scores user inputs on a scale of 1-3 based on intent clarity, topic specificity, and entity-attribute identification.
- Question Type: Classifies queries into six cognitive complexity categories: remember, understand, apply, analyze, evaluate, and create.
- Answered Queries: Tracks the percentage of questions the chatbot successfully addresses to identify knowledge base gaps.
- RAG Accuracy: A binary (0 or 1) metric assessing whether the response is factually derived from the provided context. This method showed high correlation with human evaluations across approximately 360 test questions.
RAG Accuracy Prompting
To evaluate faithfulness, the system uses a natural language inference prompt inspired by the RAGAS library. The judge LLM analyzes atomic factual statements generated from the response and compares them against the retrieved context, assigning a 1 if the statement is derivable from the context and 0 otherwise.
Benchmarking LLMs for RAG Performance
Digital Green tested a dataset of over 700 randomized user queries across various crops and dates to compare four leading LLMs: GPT-4-Turbo, Llama-3-70B-Instruct, Gemini-1.5-Pro, and Gemini-1.5-Flash.
| LLM | Faithful | Relevant | Answered * Relevant | Answered * Faithful | Unanswered |
|---|---|---|---|---|---|
| GPT-4-turbo | 88% | 75% | 59% | 69% | 21.9% |
| Llama-3-70B | 78% | 76% | 76% | 78% | 0.3% |
| Gemini-1.5-Pro | 91% | 88% | 71% | 73% | 19.4% |
| Gemini-1.5-Flash | 89% | 78% | 74% | 85% | 4.5% |
Key Findings:
- Highest Faithfulness: Gemini-1.5-Pro achieved the highest factual correctness (91%).
- Best Trade-off: Gemini-1.5-Flash was selected for production due to its superior balance of high faithfulness (89%) and a low percentage of unanswered questions (4.5%).
Impact and Outcomes
Over one year of iterative development guided by LLM-as-a-judge metrics, Farmer.chat has achieved the following milestones:
- Reach: Serviced more than 20,000 farmers.
- Volume: Handled over 340,000 queries.
- Scale: Supported over 6 languages and 50 value chain crops.
- Safety: Maintained near-zero biases or toxic responses.