petergpt/bullshit-benchmark

BullshitBench measures whether AI models challenge nonsensical prompts instead of confidently answering them, created by Peter Gostev.

BullshitBench

BullshitBench is an open‑source benchmark that evaluates how well large language models (LLMs) notice and reject nonsensical or invalid prompts. It measures three grades of model behaviour:

  • Clear pushback – the model outright rejects the broken premise.
  • Partial challenge – the model flags the problem but still engages.
  • Accepted nonsense – the model treats the premise as legitimate.

The benchmark ships with two suites (V1 = 55 questions, V2 = 100 questions) covering 13 “nonsense techniques” across domains like software, finance, law, medicine and physics. Results for dozens of commercial models (e.g., Gemini 4 Argon, Claude Haiku 5.5, Mistral Large 4) are visualised in an interactive dashboard.

Quick facts

Item Detail
Scope Detecting and calling out nonsense in LLM outputs
Data JSON question sets, manifest files, CSV leaderboards; V1 = 55 Q, V2 = 100 Q
Scoring Three‑judge panel (Claude Sonnet 4.6, GPT‑5.2, Gemini 3.1 Pro). Average determines the grade.
How to view No build step; run a tiny Python HTTP server and open the local dashboard.
How to contribute Add new questions or model evaluations via the provided config.json/config.v2.json and the Technical Guide.
License MIT (brand assets have separate notices)
Last update 13 Oct 2026

How it works

  1. Question set – curated prompts that contain deliberately false premises or logical traps.
  2. Model runs – responses are collected via paid provider APIs (the repo includes scripts for this).
  3. Human‑like judges – three LLM judges grade each response; their average decides the category.
  4. Metrics – clear‑pushback rate = clear answers / (attempts – candidate refusals).
  5. Dashboard – filter by lab, reasoning level, domain; sort by push‑back; export PNG/CSV.

Getting started (viewing results locally)

# From the repo root
python3 -m http.server 8795 --bind 127.0.0.1

Then open http://127.0.0.1:8795/ in a browser. The Viewer Guide (viewer/next/README.md) explains the UI.

Running your own evaluation

  • Edit config.json (V1) or config.v2.json (V2) to specify which models and reasoning variants to test.
  • Follow the step‑by‑step instructions in docs/TECHNICAL.md for data collection, API calls, and grading.
  • The repository already contains the manifest files that pin exact question snapshots, making results reproducible.

Data assets

  • Manifests – JSON files that list every question, its metadata, and the locations of the response/grade files.
  • Leaderboards – CSVs summarising scores for each model variant.
  • Full responses – stored in bounded JSONL files; a helper reader (described in the Technical Guide) can re‑assemble them.

All data is released under the MIT license.

Who might use this?

  • LLM developers wanting a quick sanity‑check that their model does not hallucinate confidently on nonsense.
  • Researchers studying model alignment, refusal behaviour, or prompt robustness.
  • Product teams evaluating commercial APIs for safety‑critical applications.

Links

Related

  • Project
    LiveBench/LiveBench

    LiveBench is a contamination-free benchmark for LLMs that uses monthly updated questions and objective ground-truth answers to provide reliable performance evaluations.

  • Project
    MatthewCYM/VoiceBench

    VoiceBench is a benchmark suite for evaluating LLM‑based voice assistants. It supplies a Hugging‑Face dataset with audio‑driven QA and instruction‑following tasks, a command‑line toolkit to run a model, collect spoken responses, and automatically score them (via GPT‑4o‑mini for open‑ended sets). Results are posted on a public leaderboard, and the repo includes a curated list of recent voice‑assistant papers.

  • Dispatch
    BenchMIRT: Auditing LLM Benchmarks with Multidimensional Item Response Theory

    Hugging Face and AllenAI introduce BenchMIRT, a method for auditing LLM benchmarks at the prompt level to disentangle mixed signals like safety and general reasoning.

  • Dispatch
    LiveCodeBench Leaderboard: Contamination-Free Evaluation for Code LLMs

    Hugging Face has introduced the LiveCodeBench leaderboard, a new benchmark developed by researchers from UC Berkeley, MIT, and Cornell to evaluate LLM code generation and reasoning capabilities while preventing benchmark contamination using time-windowed problem sets.

  • Dispatch
    GPT-5.5 and GLM-5.2: Analyzing the Correlation Between Model Size and Hallucinations

    A technical analysis reveals that larger models like GPT-5.5 and DeepSeek V4 Pro exhibit higher hallucination rates on the AA-Omniscience benchmark compared to the smaller, MIT-licensed GLM-5.2.