Cloudflare AI Traffic Management Update 2026

Cloudflare has transitioned from a binary "Block AI Bots" approach to a granular taxonomy that allows website owners to differentiate between AI search engines, autonomous agents, and model training crawlers. This shift addresses the "Faustian bargain" where small site owners previously had to choose between allowing AI training or losing search engine discoverability.

New AI Traffic Taxonomy

Cloudflare now classifies automated AI traffic into three primary use cases to provide site owners with more precise control over their content:

• Search: Bots that index content to answer questions later. These are expected to provide referral traffic or equitable compensation.
• Agent: Real-time automated behavior acting on a human's behalf (e.g., ChatGPT-User or browser-use agents like Gemini/Claude). These visit applications to complete specific tasks.
• Training: Crawlers that absorb data permanently into a model's architecture to improve its capabilities.

Cloudflare encourages bot operators to separate these functions into distinct crawlers to increase transparency for website owners.

Management Options and Default Changes

All Cloudflare customers, including those on the Free tier, can now manage AI traffic based on these three categories.

New Defaults (Effective September 15, 2026)

For all new domains onboarding to Cloudflare, the following defaults will apply to pages that display advertisements:

• Training and Agent bots: Blocked by default.
• Search bots: Allowed by default.

Impact on Multi-Purpose Crawlers

Crawlers that perform multiple roles (e.g., Googlebot, Applebot, and BingBot, which handle both Search and Training) will be subject to the most restrictive rule. Consequently, if a user selects to block "Training" traffic, these multi-purpose bots will be blocked entirely, even if "Search" is allowed.

BotBase and Enterprise Visibility

Cloudflare has launched BotBase, a searchable database for Enterprise Bot Management customers. BotBase provides a comprehensive view of all verified bots and agents, their classifications, and their detection IDs for use in security rules.

Expanded Bot Classifications

Beyond the three AI-specific categories, BotBase tracks several other behaviors:

• Transact: Checkout actions on behalf of users.
• Data Collection: Price scraping and competitive intelligence.
• Security Testing: Vulnerability scanning.
• SEO: Site auditing and accessibility checks.
• Ads Verification: Fraud detection and placement verification.
• Social/Link Preview: Messaging app previews.
• Feed Fetching: RSS and podcast aggregators.
• Monitoring & Operations: Uptime and health checks.

Content Use Signals and Verification

Cloudflare is introducing a way to block bots based on how they use content, moving beyond just who the bot is to what it does with the data.

Content Use Levels

Users can now define preferences via a new use signal in robots.txt (extending Content Signals):

• immediate: Interact only; store and reuse nothing.
• reference (Default): Index, excerpt, and link back.
• full: Summarize and reproduce.

Updated Verification Standards

The "Verified" bot status no longer implies a default "allow." Instead, a Verified label means a bot is eligible to be allowed if its specific category (e.g., Search) is permitted by the site owner. Bots that reproduce content in full cannot achieve Verified status.

Transitive Trust and the Forwarded Header

To handle agents run by third-party platforms on behalf of users, Cloudflare is proposing the use of the Forwarded header (RFC 7239). This allows a trusted operator (e.g., OpenAI) to pass their identity through intermediaries:

Forwarded: for="openai";use="reference"

Community Perspectives and Technical Concerns

While some users appreciate the monetization and control options, the Hacker News community raised several critical concerns regarding the implementation:

• Collateral Damage to Search Traffic: Several users reported that blocking AI training inadvertently blocked major search bots, leading to significant traffic drops.

  "I tried this, blocking AI training blocked the Google search bots and cut my traffic in half. I would not recommend."

• Centralization of Control: Critics expressed concern over outsourcing site access decisions to a dominant corporate entity.

  "I find it unsettling that we are willingly outsourcing the decision on who can access our sites to an increasingly dominant corporate entity."

• Effectiveness of the Honor System: Some skeptics argue that these signals are merely an "honor system" and that there is no technical way to prevent a bot from using content for training once it has been allowed to read the page for indexing.
• Pay-Per-Crawl Accessibility: Users noted that the promised "Pay-Per-Crawl" marketplace remains difficult to access, often requiring a request process that goes unanswered for smaller site owners.