petergpt/bullshit-benchmark

BullshitBench measures whether AI models challenge nonsensical prompts instead of confidently answering them, created by Peter Gostev.

What it solves

BullshitBench provides a way to measure if AI models can detect nonsense premises in questions and call them out clearly rather than confidently continuing with invalid assumptions.

How it works

The benchmark uses a set of questions containing invalid premises across 13 nonsense techniques in domains like software, finance, legal, medical, and physics. A three-judge panel of LLMs (Claude Sonnet 4.6, GPT-5.2, and Gemini 3.1 Pro Preview) evaluates the model responses. Responses are categorized as "Clear pushback" (rejecting the broken premise), "Partial challenge" (flagging problems but still engaging), or "Accepted nonsense" (treating the premise as valid).

Who it’s for

It is designed for researchers and developers who want to evaluate the robustness of LLMs against hallucinated or invalid assumptions in prompts.

Highlights

  • Multi-domain coverage: Tests nonsense techniques across software, finance, legal, medical, and physics.
  • Automated grading: Uses a panel of three high-performing LLMs to ensure scoring consistency.
  • Interactive dashboard: Includes a viewer to filter by lab, reasoning, or domain and compare model answers.
  • Local execution: Allows users to run their own evaluations using provided configuration files and the technical guide.

Related

  • Project
    MMMU-Benchmark/MMMU

    A massive multi-discipline benchmark for evaluating multimodal AI models on college-level subject knowledge and complex reasoning across 30 different subjects.

  • Dispatch
    OpenAI Research: Why Language Models Hallucinate

    OpenAI argues that language models hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty, rather than due to a mysterious glitch.

  • Dispatch
    TruthfulQA: Measuring how models mimic human falsehoods

    OpenAI introduces TruthfulQA, a benchmark designed to measure whether language models mimic common human misconceptions and falsehoods across 38 categories.

  • Project
    facebook/FAI-PEP

    A framework- and backend-agnostic benchmarking platform used to compare ML inference runtime metrics and detect performance regressions across various hardware and software environments.

  • Dispatch
    OpenAI Introduces LifeSciBench for Evaluating Agentic AI in Life Sciences

    OpenAI has released LifeSciBench, a benchmark of 750 expert-authored tasks designed to measure how AI systems handle complex, real-world life science research workflows rather than simple fact recall.