ai-robots-txt/ai.robots.txt

A list of AI agents and robots to block.

What it solves

This project provides a centralized, community-maintained list of AI-related web crawlers. It helps website owners prevent AI systems from scraping their content by providing ready-to-use configuration files that tell these bots they are not welcome.

How it works

The project maintains a master list of AI crawler signatures. A GitHub action automatically converts this list into various formats, including a standard robots.txt file and specific configuration snippets for web servers like Nginx, Apache, Caddy, HAProxy, and Lighttpd. Users can download these files and apply them to their own server infrastructure to block the listed crawlers.

Who it’s for

This is for website administrators, developers, and site owners who want to control whether AI companies can crawl their content for training or data collection purposes.

Highlights

  • Provides a standardized robots.txt file following the Robots Exclusion Protocol.
  • Includes configuration snippets for multiple popular web servers (Nginx, Apache, Caddy, etc.).
  • Automates the generation of blocklists from a central source of truth.
  • Offers resources for tracking crawler behavior and reporting bots that ignore exclusion rules.

Related

  • Project
    jordan-gibbs/hyperresearch

    Hyperresearch is a deep research harness for AI coding agents (Claude Code, OpenAI Codex) that runs a 16-step pipeline to gather hundreds of sources, verify citations, and produce adversarial-audited reports with a persistent searchable vault.

  • Project
    ZJU-REAL/Easel

    Easel is an AI-powered social media workbench that automates the entire content creation lifecycle, from trend discovery and multi-format media production to cross-platform publishing and performance analysis.

  • Project
    browser-use/workflow-use

    A tool for creating deterministic, self-healing browser automation workflows by recording interactions or generating them from natural language, reducing reliance on constant LLM prompting.

  • Project
    arcboxlabs/arcbox

    An open-source, Rust-based container and VM runtime for macOS that provides a high-performance alternative to Docker Desktop and OrbStack, including support for AI agent sandboxes.