Bullshit Detector Show HN Project – Agent Skills for Claim‑by‑Claim Fact‑Checking

TL;DR – What Bullshit Detector Does and Why It Matters

Bullshit Detector is a set of portable Agent Skills that automatically retrieve the text of any online content (YouTube, TikTok, articles, PDFs, tweets) and run a claim‑by‑claim fact‑check, producing a structured report with source citations and a 0‑10 “BS” score. It lets LLM‑powered agents verify hype‑driven content without manual research, addressing the growing problem of viral misinformation that spreads faster than human fact‑checking can keep up.

Core Functionality – Fact‑Checking as a Skill

  • Ingestion (fetch-content) – One script turns any URL into clean text + metadata. It extracts YouTube transcripts via yt‑dlp, TikTok captions (or falls back to local Whisper transcription), article bodies via readability, PDFs, and tweets via free endpoints. No API keys are required, and failures return actionable hints (e.g., paywall → paste, missing captions → Whisper).
  • Analysis (bullshit-detector) – The skill parses the normalized text, extracts individual claims, searches the web for independent sources, and assigns per‑claim verdicts:
    • ✅ confirmed
    • 🟡 plausible
    • 🟠 misleading
    • ❌ false
    • ❓ unverifiable It also scans for hype signals, performs incentive analysis, and aggregates a 0‑10 BS score.
  • Supporting Skills – summarize creates timestamped TL;DRs, explain offers ELI5‑style deep dives, and share formats reports for social platforms.

Design principle #3: Separation of fetching and judging – ingestion scripts are deterministic; analysis skills only see normalized text, making the detector reusable across media types (TikTok, podcasts, long‑form videos).

Quickstart – Get Running in 30 Seconds

  1. Install the lightweight dependency manager uv (or use any Python environment).
  2. Run the installer via npx skills@latest add SerhiiKorniienko/bullshit-detector and select the desired skills.
  3. Ask your agent:
    • is this bullshit? <url>
    • fact-check this video
    • summarize <url>
    • explain the part at 12:30

Two installation paths are offered:

  • skills.sh copies the markdown skills into your workspace for full customisation (works with Claude Code, Codex, OpenCode, etc.).
  • Claude Code plugin provides a read‑only, auto‑updating bundle for users who just want it to work out‑of‑the‑box.

Real‑World Example – 14 Ways to Make Money with AI (YouTube)

Running the detector on a 1.16 M‑view video produced the following report:

  • BS score: 5/10
  • Claims verified: 12 total → 4 confirmed, 2 plausible, 3 misleading, 0 false, 3 unverifiable.
  • Notable findings: a claim about "Renaissance, D.E. Shaw, Two Sigma only trade employees' money" was partially true (only one fund of one firm), and several marketplace statistics traced back to the platform’s own PR.

A second example on a 552 K‑view TikTok about a "hidden twin Sun" yielded a BS score of 9/10, exposing fabricated cosmology stitched onto genuine astronomical terminology.

Full reports are available in the repo:

Community Feedback – Highlights from Hacker News Comments

  • Effort vs. payoff – A commenter suggested a browser extension using local models (e.g., Ollama) would be less effort for end‑users. The project’s design intentionally targets agent‑centric workflows rather than a UI‑first extension.
  • Ground‑truth concerns – Users asked how “independent sources” are vetted and how contradictory sources are reconciled. The skill mandates citations for every verdict and leaves source credibility assessment to the LLM’s reasoning, which can be refined in future iterations.
  • Meta‑checking – Questions like “Who checks the checkers?” were raised. The open‑source nature allows anyone to audit the skill code and the underlying prompts, and the community can contribute improvements.
  • Use cases – Several commenters noted the tool would be valuable for social‑media platforms (TikTok, Reels, Shorts) where misinformation spreads fastest. The author acknowledges this and plans to add pre‑publish fact‑checking pipelines.
  • Skepticism – Some expressed doubt about the reliability of an LLM‑driven fact‑checker, calling it “vibe‑coded.” The project mitigates this by requiring external web sources rather than model memory alone.
  • Positive reception – Users praised the idea, noting that automated claim‑by‑claim verification could complement human fact‑checkers, especially for high‑volume content like the Washington Post’s Trump‑claim database.

Compatibility – Which Agents Can Use the Skills?

Agent / App Support Level Setup Guide
Claude Code CLI ✅ Full support Claude Code CLI guide
Claude Desktop (Code tab) ✅ Full support Code tab guide
Claude Desktop / claude.ai (Chat) ⚠️ Analysis only (no video fetch) Chat guide
OpenAI Codex ✅ Via skills.sh Codex guide
ChatGPT ⚠️ Paste‑driven workaround ChatGPT guide
OpenCode, Cursor, Gemini CLI, etc. ✅ Via skills.sh Other agents guide

Roadmap and Future Work

  • compare skill – Cross‑source comparison to identify which source is more reliable on a given claim.
  • transcribe skill – Production‑ready Whisper integration for caption‑less TikTok/Reels, currently in prototype.
  • Social‑media publishing – Direct posting of fact‑checked reports to TikTok, X, or other platforms.
  • Enhanced source vetting – Algorithms for weighting source credibility and handling contradictory evidence.

Getting Involved

The project is MIT‑licensed and welcomes contributions. Follow the author on X (@SerhiiFounder) or join the newsletter for updates on new detectors, adapters, and real‑world fact‑check reports.

License

MIT

Sources