What happened to TheNumbers.com – AI scraping, cost spikes, and a stripped‑down return

What happened to TheNumbers.com

The site went down after a surge of automated traffic that drove up network and hosting expenses.

A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. ... Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.

Why it matters

The incident illustrates how unrestricted AI scraping can make free public resources financially unsustainable.

It feels like there is fundamentally missing infrastructure here that is needed to make these problems go away. Basically bots need to be (somehow) paying for the traffic they create, or prevented from creating it, or told to go away and then fined if they violate the request.

Community reactions

Commenters speculated about motives, mitigation strategies, and the broader impact on open data.

• Some suggested the downtime was tied to attempts to gain early access for prediction‑market betting: "If you could see The Numbers data before everyone else, every single week, you would have a significant edge over all the other traders."
• Others viewed the event as a warning about the erosion of the open web: "All new content will be behind paywalls. This will be the new way the Internet works."
• A few proposed technical fixes such as using Cloudflare, static site generation, or proof‑of‑work challenges to deter crawlers.

Possible mitigations discussed

Participants highlighted several approaches that could reduce the cost of bot traffic.

• Deploying a bot‑aware CDN or services like Cloudflare to block known scrapers.
• Re‑architecting the site as a static site generated from pre‑built data to minimize dynamic requests.
• Implementing rate limiting, challenge mechanisms (e.g., proof‑of‑work), or licensing models that shift costs to consumers of the data.
• Creating community‑maintained toolkits for traffic classification, access policies, and logging to help small sites defend against automated access.

Limitations of the source

The Hacker News post and its comments do not contain the full article from stephenfollows.com, so specific details such as exact traffic volumes, dates, or the site’s technical stack are not available. The summary above reflects only the information explicitly shared in the discussion.