PacificAI/langtest

Deliver safe & effective language models

What it solves

LangTest is designed to help data scientists ensure that language models are safe, robust, and fair. It addresses the lack of-the-box tools for evaluating model quality beyond simple accuracy, specifically targeting issues like demographic bias, toxicity, and inconsistency in real-world conditions.

How it works

The library provides a Harness object that allows users to generate, execute, and report on over 60 distinct types of tests. It supports a wide range of NLP tasks (NER, Translation, Text Classification) and integrates with popular frameworks like Hugging Face, Spark NLP, and various LLM providers (OpenAI, Cohere, AI21, Azure-OpenAI).

Who it’s for

It is built for NLP practitioners and data scientists who need to rigorously test their models for robustness, fairness, and safety before deploying them into production systems.

Highlights

  • Comprehensive Testing: Covers robustness, bias, representation, fairness, and accuracy.
  • Automated Data Augmentation: Can automatically augment training data based on test results for certain models.
  • LLM Support: Specialized tests for LLMs including factuality, sycophancy, and toxicity.
  • Broad Integration: Works with multiple NLP frameworks and LLM APIs.

Related

  • Project
    EleutherAI/lm-evaluation-harness

    A unified framework to evaluate generative language models on a large number of standard academic benchmarks, serving as the backend for the Open LLM Leaderboard.

  • Project
    evidentlyai/evidently

    An open-source Python framework to evaluate, test, and monitor ML and LLM-powered systems, helping developers detect data drift and ensure output quality.

  • Project
    huggingface/lighteval

    A comprehensive toolkit from Hugging Face for evaluating LLMs across multiple backends with support for over 1,000 benchmarks and custom metrics.

  • Project
    modelscope/evalscope

    A comprehensive LLM evaluation framework that combines capability benchmarking, inference performance stress testing, and interactive result visualization in one tool.

  • Project
    confident-ai/deepeval

    DeepEval is an open‑source Python framework that lets you write pytest‑style tests for LLM‑powered apps (agents, chatbots, RAG pipelines). It ships >30 built‑in metrics—including LLM‑as‑a‑judge scores like G‑Eval and JevEval—as well as agent‑, RAG‑, multimodal‑, and bias‑related checks. Metrics run locally or with any LLM you choose. The library integrates with LangChain, CrewAI, OpenAI, Anthropic, AWS AgentCore, Google ADK, LlamaIndex, and even the TypeScript AI SDK, providing decorators and callbacks to capture full execution traces. A simple CLI (`deepeval test run …`) works in CI/CD, and the optional Confident AI platform adds cloud reporting and governance. Install with `pip install -U deepeval`, write `LLMTestCase` objects, pick a metric, and run—making LLM quality testing as systematic as traditional software testing.