MigoXLab/dingo

Dingo – AI‑Data‑Quality Evaluation Platform

What it is – Dingo is an open‑source Python library (and SaaS‑enabled product) that lets you automatically assess the quality of AI training data, fine‑tuning datasets, and even the outputs of production LLM or Retrieval‑Augmented Generation (RAG) systems. It combines fast, rule‑based checks with optional LLM‑driven deep assessments and can be run locally, via a Spark cluster, or through a built‑in Model Context Protocol (MCP) server for IDE‑style integration.

Core capabilities

Capability	How Dingo does it
Multi‑source ingestion	Reads local files (JSONL, CSV, Parquet, TXT), Hugging Face datasets, S3 buckets, and SQL databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) using streaming cursors so even billions of rows fit in memory.
Field‑level pipelines	You can map different quality checks to different columns (e.g., ISBN rule on `isbn`, text‑quality LLM on `title`). The framework isolates each field’s results to avoid context bleeding.
Rule‑based checks	30+ built‑in heuristics (regex, character‑set, PII detection, format validation, etc.) that run on every record at near‑zero cost. Custom rules can be registered via a simple decorator.
LLM‑based evaluation	Optional deep checks (e.g., `LLMTextQualityV4/V5`) that call OpenAI, DeepSeek, Kimi, or local models (Llama 3, Qwen). Prompts live under `dingo/model/llm/` and are easy to replace for domain‑specific needs.
Agent‑driven reasoning	Built‑in agents (FactCheck, Hallucination) can invoke external tools (web search, Tavily) and perform multi‑step verification. Two patterns are supported – a LangChain ReAct agent and a hand‑crafted workflow for fine‑grained control.
RAG evaluation	Five academically‑backed metrics (Faithfulness, Answer Relevancy, Context Precision/Recall, Context Relevancy) let you benchmark retrieval‑augmented pipelines.
Scalable execution	CLI, Python SDK, or Spark executor. Concurrency and batch size are configurable; the Spark executor distributes work across a cluster for massive datasets.
Reporting	JSON summary, field‑level breakdown, per‑rule violation details, and statistical aggregates (mean, min, max, std). The SaaS version adds a web UI with interactive charts, trend analysis, and export options.
MCP server	Starts a lightweight SSE or stdio server (`dingo serve`) so IDEs like Cursor or Claude Desktop can query the evaluator as a tool.

Quick start (local)

# Install core package
pip install dingo-python
# Optional extras – hallucination detection, retrieval benchmarks, or everything
pip install "dingo-python[hhem]"          # adds HHEM model
pip install "dingo-python[retrieval]"    # adds MTEB + pytrec‑eval
pip install "dingo-python[all]"          # all optional features

Example – rule + LLM on a single record

from dingo.io.input import Data
from dingo.model.rule.rule_common import RuleSpecialCharacter
from dingo.model.llm.text_quality.llm_text_quality_v4 import LLMTextQualityV4
from dingo.config.input_args import EvaluatorLLMArgs

sample = Data(data_id='123', prompt='hello', content='I am 8 years old. ^I love apple because:')

# Fast rule check
print(RuleSpecialCharacter().eval(sample))

# LLM check (requires an OpenAI key)
LLMTextQualityV4.dynamic_config = EvaluatorLLMArgs(
    key='YOUR_API_KEY',
    api_url='https://api.openai.com/v1/chat/completions',
    model='gpt-4o',
)
print(LLMTextQualityV4.eval(sample))

Example – evaluate a whole Hugging Face dataset

from dingo.config import InputArgs
from dingo.exec import Executor

cfg = {
    "input_path": "tatsu-lab/alpaca",
    "dataset": {"source": "hugging_face", "format": "plaintext"},
    "executor": {"result_save": {"bad": True}},
    "evaluator": [{"evals": [{"name": "RuleColonEnd"}, {"name": "RuleSpecialCharacter"}]}]
}
args = InputArgs(**cfg)
executor = Executor.exec_map["local"](args)
result = executor.execute()
print(result)

CLI usage

# Run rule‑only evaluation
 dingo eval --input .github/env/local_plaintext.json

# Run LLM‑backed evaluation (e.g., GPT‑4o)
 dingo eval --input .github/env/local_json.json

Extending Dingo

Custom rule – subclass BaseRule and register with @Model.rule_register('QUALITY_BAD_CUSTOM', ['default']).
Custom LLM evaluator – subclass BaseOpenAI (or any BaseLLM) and register with @Model.llm_register('custom_evaluator').
Custom agent – either use the LangChain‑based AgentFactCheck pattern or write a manual workflow (see examples/register/).

When to use the SaaS edition

If you need a no‑code UI, role‑based access (JWT + Google OAuth), visual dashboards, or RESTful APIs for downstream systems, the enterprise SaaS version (https://dingo.openxlab.org.cn) provides those on top of the same evaluation engine.

Who should consider Dingo?

Data engineers who must validate massive pre‑training corpora before feeding them to LLMs.
ML researchers building instruction‑tuned datasets and wanting automated 3H (Honest‑Helpful‑Harmless) checks.
Product teams running RAG pipelines that need systematic faithfulness and relevance metrics.
Compliance officers looking for PII, toxicity, or hallucination detection across production outputs.

License & Community

License: Apache‑2.0 (per the README metadata).
Community: Discord, WeChat, and a public GitHub repo with issue tracking, pre‑commit hooks, and CI badges. Contributions are encouraged via the plugin registration system.

Bottom line: Dingo gives you a production‑grade, extensible toolbox for measuring and improving the quality of any AI‑related data—whether it lives in files, databases, or the wild outputs of large language models.