AI Training and the Destructive Scanning of Rare Books

AI Companies Are Destructively Scanning Rare Books for Training Data

AI companies are bulk-buying rare books, scanning them using high-speed machines that cut the spines off, and shredding the original physical copies. This process, referred to in some court documents as "Project Panama," aims to obtain vast amounts of high-quality, human-generated text to train large language models (LLMs). The practice is particularly focused on pre-2022 books to ensure training data is free from AI-generated content.

The Role of ISBNdb and Anonymous Acquisition

A service called ISBNdb reportedly facilitates these operations by managing orders of up to one million books while keeping the buyers anonymous. To avoid public backlash, the service provides non-disclosure agreements (NDAs) and coaches clients to frame the destruction of books as "digital preservation."

Legal Justification and Fair Use

A federal judge has ruled that this destructive scanning practice constitutes fair use. The legal reasoning is that by eliminating the original physical copy, the company ensures that only one copy of the work exists at a time (in digital form), thereby limiting the market impact on the original work.

Strategic Acquisition and "Project Panama"

Reports indicate that Anthropic has been particularly aggressive in this pursuit, hiring the former head of Google Books partnerships to help acquire "all the books in the world." According to discussions surrounding the issue:

"The court documents even gave it a name, Project Panama, tens of millions spent, and a planning doc that flat out said the goal was to destructively scan every book in the world."

Technical and Cultural Implications

Loss of Primary Sources and Artifacts

The destruction of physical books removes the ability to verify digital copies against a primary source. Beyond the text, scanning fails to capture the physical artifacts of a book, including marginalia, binding, and printing quality, which are essential for historical research.

Risks of Historical Revisionism

Critics argue that once the physical originals are destroyed, AI companies gain the power to rewrite history. Without a physical reference, digital copies can be altered to fit specific political or corporate narratives without any means of verification.

The "Analog Hole" Strategy

Some analysts suggest this is a strategic response to copyright lawsuits. By purchasing physical copies and scanning them (the "analog hole"), AI companies can bypass expensive licensing deals with publishers and copyright holders.

Community Perspectives and Counterpoints

While many view this as a modern form of "book burning," some commenters offer a different perspective:

Copyright Law as a Driver: Some argue that overly restrictive US copyright laws force companies into these extremes to avoid "extortion fees" from publishers who keep books out of print.
Digital Accessibility: A few suggest that digitizing obscure books that would otherwise never be read is a net positive, provided the information is preserved.
Comparison to Library "Weeding": Some point out that libraries already engage in "weeding" (discarding old materials) at scale, though not typically for the purpose of training AI models.
Literary Parallels: Observers have noted the similarity between this practice and the "shred and scan" factory depicted in Vernor Vinge's 2006 novel Rainbows End.