AnswerDotAI/llms-txt

The /llms.txt file, helping language models use your website

What it solves

Web pages are designed for humans, often cluttered with navigation, ads, and JavaScript that make it difficult for AI agents to extract clean, relevant text. Additionally, full websites are often too large to fit into an LLM's context window, leading to wasted tokens and increased costs.

How it works

It proposes a standardized /llms.txt file placed at a website's root or subpath. This Markdown file provides a concise, expert-level overview of the site, guidance, and links to detailed Markdown versions of specific pages (e.g., page.html.md).

Agents are intended to read the llms.txt file first to understand the site's structure and then follow links to fetch only the specific detailed content they need. To help agents discover these files, the proposal recommends using standard HTML link relations like rel="alternate" for Markdown versions and rel="describedby" for the llms.txt file.

Who it’s for

  • Website owners and developers who want to make their documentation or content more accessible to AI agents.
  • AI agent developers who want a reliable, standardized way to fetch high-quality, concise context from the web.

Highlights

  • Standardized Format: Uses a specific Markdown structure (H1 title, blockquote summary, and H2-delimited file lists) that is both human-readable and programmatically parsable.
  • Context Efficiency: Reduces token waste by providing a curated map of the site rather than requiring the agent to scrape entire HTML pages.
  • Broad Compatibility: Designed to coexist with robots.txt and sitemap.xml without replacing them.
  • Ecosystem Support: Already adopted by major AI labs (OpenAI, Anthropic, Gemini) and various CMS platforms like Wix and GitBook.

Related

  • Dispatch
    The Rise of llm.txt: Can Machine-Readable Web Content Fix the Human Experience?

    The emergence of llm.txt files—simplified, Markdown-based summaries for AI—is sparking a debate on whether a 'clean web' for machines could inadvertently restore a text-centric, usable internet for humans.

  • Project
    okineadev/vitepress-plugin-llms

    A VitePress plugin that automatically generates LLM-friendly Markdown documentation (llms.txt and llms-full.txt) based on the llmstxt.org standard.

  • Project
    vercel-labs/lat.md

    A knowledge graph system for codebases that uses interconnected markdown files and source code annotations to provide AI agents and humans with scalable, searchable context.

  • Project
    paulpierre/markdown-crawler

    A multithreaded web crawler that converts website pages into markdown files to simplify document parsing and chunking for RAG and LLM fine-tuning.

  • Project
    jatinkrmalik/LLMFeeder

    A browser extension that converts web page content into clean Markdown, optimized for feeding into LLMs with built-in token counting and multi-tab support.