NVIDIA-NeMo/Guardrails
NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
What it solves
NeMo Guardrails is a toolkit designed to make LLM-based conversational applications more trustworthy, safe, and secure. It prevents LLMs from discussing unwanted topics (like politics), ensures they follow predefined conversational paths, and protects against common vulnerabilities such as jailbreaks and prompt injections.
How it works
The library acts as a programmable layer between the application code and the LLM. It uses a specialized modeling language called Colang to define dialogue flows and guardrails. These guardrails are categorized into five types:
- Input rails: Filter or modify user input before it reaches the LLM.
- Dialog rails: Steer the conversation flow and determine if the LLM should generate a response or use a predefined one.
- Retrieval rails: Filter or modify retrieved documents in RAG scenarios.
- Execution rails: Manage the input and output of custom tools/actions called by the LLM.
- Output rails: Filter or modify the LLM's response before it reaches the user.
Who it’s for
Developers building LLM-powered chatbots, RAG-based question-answering systems, or custom LLM endpoints who need fine-grained control over model behavior and safety moderation.
Highlights
- Colang Modeling Language: A Python-like language for designing flexible yet controllable dialogue flows.
- Comprehensive Protection: Built-in support for jailbreak detection, hallucination detection, and fact-checking.
- Multi-LLM Support: Compatible with models from OpenAI, LLaMa-2, Falcon, Vicuna, and Mosaic.
- LangChain Integration: Optional ability to wrap guardrails around LangChain chains.
- Evaluation Tools: Includes a CLI tool (
nemoguardrails evaluate) for testing topical rails and moderation effectiveness.