The Cost of AI Crawlers: Lessons from git.kernel.org

AI Scrapers Consume 20% of git.kernel.org CPU Capacity

AI crawlers are creating a constant "background radiation" of system load on git.kernel.org, permanently tying up significant computing resources to produce data used exclusively for training large language models (LLMs). Currently, across five geo-distributed nodes with a total of 90 cores, approximately 14 to 16 cores are dedicated solely to rendering git commits as HTML for scrapers. This represents roughly 20% of the site's total capacity.

The Inefficiency of HTML Scraping vs. Git Cloning

Despite the availability of highly efficient methods for data acquisition, AI crawlers are opting for the most resource-intensive approach. While the entirety of the Linux kernel history and the LKML archives are available via git clone—allowing a scraper to download the entire history once and process it locally—bots are instead rendering every commit as HTML and parsing the resulting pages.

This inefficiency is compounded by the combinatoric explosion of available URLs. With approximately 1.48 million commits in linux.git and 922 forks, the number of valid URLs for commits, patches, plain renders, and diffs reaches into the billions. Scrapers are frequently targeting these billions of URLs, including those in abandoned forks, which generates massive server load without providing unique data.

The Evolution of Bot Mitigation Strategies

Efforts to block these crawlers have evolved through several stages as bots became more sophisticated:

User-Agent and IP Blocking: Initial attempts to block bots based on their self-identified User-Agents or obvious data-center IP ranges (e.g., Google Compute Engine) were quickly bypassed as bots began pretending to be standard web browsers.
Residential Proxy Networks: Crawlers transitioned to using millions of residential and mobile IPs, often routed through "proxy SDK monetization" schemes where household appliances (such as Smart TVs) are used as proxies. This makes IP-based banning ineffective, as each IP may only make a few requests before disappearing.
Proof-of-Work (PoW) Challenges: To flip the economic cost of scraping, git.kernel.org implemented Anubis, a proof-of-work system that requires clients to solve a mathematical challenge (SHA-256) before accessing content.

The Failure of Proof-of-Work as a Permanent Solution

While Anubis was initially effective, it has become an arms race. The system currently handles about 6 million daily requests for random commits; 66% are blocked by the challenge, but 33% are now solving the math and proceeding to the site.

Technical analysis from the community suggests that PoW is an unsustainable strategy because the cost of computation is asymmetric. High-powered scrapers can solve these challenges far more efficiently than legitimate users on mobile devices. For example, a user on an iPhone may find a high-difficulty challenge takes several seconds and causes the device to warm up, while a bot using optimized C kernels or specialized hardware can solve the same challenge in milliseconds.

Community Insights and Alternative Proposals

Discussion among developers and site administrators reveals that this is a systemic issue affecting many public resources, not just the Linux kernel.

Observed Patterns

Targeted vs. General Crawling: Some argue that bots are not specifically targeting git hosts but are simply following every available link in a general web crawl, leading them into "crawler traps" created by theCombinatoric nature of git web interfaces.
Synthetic Data Risks: The author notes that LLM-free data (like kernel commits) is highly prized because training LLMs on LLM-generated content can lead to a "digital prion disease" (model collapse).

Proposed Technical Mitigations

Client-Side Rendering: Moving the HTML rendering logic to the user's browser via JavaScript, turning the server into a simple object store for flat files and diffs.
Gated Access: Requiring authentication for HTML views while keeping git clone unrestricted, or implementing aggressive rate limiting for unauthenticated requests.
Tarpitting: Implementing "ioicaine-style" traps that serve bogus data or extremely slow responses to detected bots to waste their resources.
Monetization: Transitioning from proof-of-work to micropayments (e.g., L402) to cover the actual cost of the server resources used.

Current Status and Future Outlook

git.kernel.org is currently reducing functionality for anonymous users and turning off expensive features to mitigate the load. The administration emphasizes that while all data remains available for download, users may encounter more "hoops" to access it. The long-term solution remains unclear, as the demand for training data continues to grow and the infrastructure for residential proxy networks expands.