cvs-health/uqlm

📚 What is UQLM?

UQLM (Uncertainty Quantification for Language Models) is a Python library that helps you detect when a large language model (LLM) is likely to hallucinate – i.e., produce statements that are factually wrong or unsupported. It does this by attaching a confidence score (0‑1) to every LLM response, where a higher score means the model is more trustworthy.

---

🎯 Core Idea

LLMs are great at generating fluent text, but they can confidently say things that are false. UQLM treats this as an uncertainty problem: if the model is uncertain, the confidence score should be low. The library implements several state‑of‑the‑art uncertainty‑quantification (UQ) techniques and wraps them in easy‑to‑use classes.

---

🧩 Main Components (Classes)

Class	What it does	Typical use‑case
BlackBoxUQ	Generates multiple responses to the same prompt and measures how consistent they are. Works with any LLM because it only needs the generated texts.	When you have cheap access to an LLM and can afford extra API calls.
WhiteBoxUQ	Scores a single generation using the model’s token‑level probabilities (e.g., minimum token probability, sequence probability). Requires an LLM that returns token probabilities.	Fast, low‑cost scoring when the provider exposes token‑probability data (e.g., Gemini, OpenAI “logprobs”).
LLMPanel	Uses one or more LLMs as judges that read the original response and output a confidence rating. Fully customizable via prompts.	When you want a “model‑as‑judge” approach or to combine opinions from several LLMs.
UQEnsemble	Combines several scorers (black‑box, white‑box, judge) via weighted averaging. Can be used off‑the‑shelf or tuned on a small labeled set to learn optimal weights.	Building a robust, production‑ready uncertainty estimator.
LongTextUQ	Breaks a long answer into individual claims, scores each claim, and can refine the answer by dropping low‑confidence claims.	Long‑form generation (articles, reports) where you need claim‑level reliability.
CodeGenUQ	Specialized for code‑generation; predicts functional correctness without running the code, using both token‑probability and functional‑equivalence scorers.	Evaluating generated code snippets for correctness.

---

⚙️ How you would use it (quick example)

from langchain_openai import ChatOpenAI
from uqlm import BlackBoxUQ

# 1️⃣ Pick any LangChain‑compatible LLM
llm = ChatOpenAI(model="gpt-4o-mini")

# 2️⃣ Create a scorer – here we ask for the "semantic_negentropy" black‑box scorer
bbuq = BlackBoxUQ(llm=llm, scorers=["semantic_negentropy"], use_best=True)

# 3️⃣ Run on a list of prompts (5 generations per prompt) and get a pandas‑like table
results = await bbuq.generate_and_score(prompts=["What is the capital of France?"],
                                         num_responses=5)
print(results.to_df())

The returned table contains the generated text, the confidence score, and (if use_best=True) the lowest‑uncertainty response.

---

📊 Types of Scorers (what’s under the hood?)

Category	Example scorers	What they measure
Black‑Box (Consistency‑based)	semantic_negentropy, non_contradiction, entailment	How much the multiple generations agree with each other.
White‑Box (Token‑probability‑based)	min_probability, sequence_probability, top‑k_negentropy	Direct statistical confidence from the model’s own probability distribution.
LLM‑as‑Judge	Categorical, Continuous, Likert, Panel	A separate LLM reads the answer and judges its truthfulness.
Ensemble	Weighted average of any of the above	Combines strengths; can be tuned on a validation set.
Long‑Text (Claim‑level)	entailment, graph‑based, LUQ	Scores each claim/sentence; can drop low‑confidence claims.
Code‑Gen	functional_equivalence_rate	Estimates if generated code matches the intended functionality.

---

📦 Installation & Getting Started

pip install uqlm   # pulls the latest release from PyPI

The library requires Python 3.10+ and works with any LLM that follows the LangChain chat‑model interface.

---

📚 Documentation & Learning Resources

• Full docs: https://cvs-health.github.io/uqlm/latest/index.html – API reference, tutorials, and troubleshooting.
• Example notebooks: examples/ folder on GitHub – covers every scorer type, ensemble tuning, and long‑text refinement.
• Discord community: a public Discord server for questions and collaborations.

---

🏆 When to Reach for UQLM?

• You need automatic, per‑response confidence for downstream pipelines (e.g., filtering LLM answers before showing them to users).
• You are working with high‑stakes domains (healthcare, finance, legal) where hallucinations are unacceptable.
• You want a plug‑and‑play solution that works with any LangChain‑compatible LLM, without writing your own uncertainty logic.

---

📖 Citation

If you use UQLM in research, cite the JMLR software paper:

@article{JMLR:v27:25-1557,
  author  = {Dylan Bouchard and Mohit Singh Chauhan and David Skarbrev ...},
  title   = {UQLM: Uncertainty Quantification for Language Models},
  journal = {Journal of Machine Learning Research},
  year    = {2024},
  volume  = {27},
  pages   = {1--...},
  url     = {https://www.jmlr.org/papers/v27/25-1557.html}
}

---

TL;DR

UQLM = a ready‑made toolbox that lets you measure and act on LLM uncertainty using a variety of proven techniques, from cheap consistency checks to sophisticated ensembles and claim‑level refinement. Install with pip, pick the scorer class that matches your latency/cost constraints, and you’ll get a 0‑1 confidence score for every generated answer.