HowieHwong/TrustLLM

What it solves

TrustLLM provides a standardized way to measure how trustworthy large language models (LLMs) are. Instead of relying on a single metric, it evaluates models across six critical dimensions: truthfulness, safety, fairness, robustness, privacy, and ethics. This helps researchers and developers identify specific weaknesses in a model's behavior, such as its tendency to hallucinate, leak private information, or exhibit bias.

How it works

The toolkit operates in a three-step pipeline:

1. Data Acquisition: Users download a comprehensive benchmark dataset containing prompts designed to test the six dimensions of trustworthiness.
2. Generation: The toolkit runs these prompts through a target model—either via an API (like OpenAI) or by loading local weights (via Hugging Face)—and saves the responses.
3. Evaluation: A scoring engine processes the generated responses using a mix of rule-based checks, classifiers, embeddings, or an AI-based judge to produce final trustworthiness scores.

Who it’s for

It is designed for AI researchers and developers who need to rigorously benchmark the reliability and safety of LLMs before deployment or as part of academic research.

Highlights

• Six-Dimensional Analysis: Covers truthfulness, safety, fairness, robustness, privacy, and ethics.
• Flexible Backend: Supports both local model weights and API-based generation.
• ** uma Python API**: Offers both a command-line interface (CLI) and a Python library for orchestration.
• Automated Reporting: Generates HTML reports for both generation progress and final evaluation scores.