Bespoke-Minicheck: Reducing LLM Hallucinations via Grounded Factuality Checking

Bespoke-Minicheck is a grounded factuality checking model developed by Bespoke Labs and integrated into Ollama. It reduces LLM hallucinations by verifying whether a generated claim is supported by a specific source document, outputting a binary "Yes" or "No" response.

Grounded Factuality Verification Mechanism

Bespoke-Minicheck operates as a verification layer that compares a generated output (the Claim) against a piece of factual information (the Document). The model determines if the source document supports the claim; if it does, the model outputs "Yes", and if it does not, it outputs "No".

Application in Retrieval Augmented Generation (RAG)

Bespoke-Minicheck is designed for use in Retrieval Augmented Generation (RAG) pipelines to ensure that model responses are grounded in the retrieved context. By implementing the model as a post-processing step, developers can detect hallucinations by verifying the final response against the context provided to the LLM.

Implementation and Usage

To use Bespoke-Minicheck via Ollama, users can run the model using the following command:

ollama run bespoke-minicheck

Verification requires a prompt structured with both a source document and a claim. For example:

• Supported Claim:

  • Document: A group of students gather in the school library to study for their upcoming final exams.
  • Claim: The students are preparing for an examination.
  • Result: Yes

• Unsupported Claim:

  • Document: A group of students gather in the school library to study for their upcoming final exams.
  • Claim: The students are out on vacation
  • Result: No

Technical Resources

Bespoke Labs and Ollama provide several implementation examples on GitHub for integrating this model:

• Fact Checking: A simple check of a claim against source information.
• RAG Integration: An example of using the model to verify grounded factuality in a RAG application.