m92vyas/llm-reader

llm‑reader – Turn any webpage into LLM‑ready text

What it does

• Fetches a URL (or any HTML string) and strips away the noisy parts of a web page, producing clean, markdown‑style text that fits well into prompts for large language models.
• Handles tables, links, images and other structured elements so they can be extracted by an LLM with minimal token waste.
• Built on Playwright for fast, concurrent, headless browsing (replaces the older Selenium implementation).

Why it matters When you feed raw HTML to a model you waste tokens on tags, scripts and layout noise, and the model often misses the data you actually need (product names, prices, tables, etc.). llm‑reader automates the preprocessing step, letting you focus on the prompt and the downstream extraction logic.

Key components

• get_page_source(url): async helper that launches a Playwright browser, loads the page, and returns the raw HTML. You can replace it with any other scraper or paid API if you prefer.
• get_processed_text(html, url): async function that converts the HTML into a clean, markdown‑like representation optimized for LLM consumption.

Installation

pip install git+https://github.com/m92vyas/llm-reader.git
# Playwright setup (downloads browsers and system deps)
playwright install
playwright install-deps

Basic usage

import asyncio
from url_to_llm_text.get_html_text import get_page_source
from url_to_llm_text.get_llm_input_text import get_processed_text

async def llm_ready(url: str) -> str:
    html = await get_page_source(url)          # fetch raw HTML
    clean = await get_processed_text(html, url)  # turn into LLM‑friendly text
    return clean

text = asyncio.run(llm_ready("https://example.com"))
print(text)

You can then embed text in any prompt, optionally truncating it to fit your model’s context window.

Example: extracting product data The README shows a full workflow: fetch a category page from IKEA, run get_processed_text, then craft a JSON‑style prompt for OpenAI’s gpt‑4o‑mini. The model returns a structured list of products with name, link, image URL and price.

Features & highlights

• Async & concurrent: Playwright lets you scrape many pages in parallel.
• Markdown output: Preserves headings, lists, tables and links in a format LLMs understand well.
• Pluggable source: You can swap get_page_source for a proxy service, paid API (e.g., Scrappey) or your own Selenium setup.
• Open‑source MIT license – free to modify and embed in commercial pipelines.

Limitations

• The library only prepares text; you still need to write the LLM prompt and handle token limits yourself.
• Complex JavaScript‑heavy sites may still require additional waiting or custom Playwright scripts.
• No built‑in OCR or PDF handling – those are offered via the author’s separate services (ParseExtract, RedactLocal).

Further reading

• Detailed docs: https://github.com/m92vyas/llm-reader/wiki/Documentation
• Companion scraper: https://github.com/m92vyas/AI-web_scraper (adds search, multi‑page crawling, and prompt templating).

Who might use this

• Data‑engineers building RAG pipelines that need clean web text.
• Researchers extracting structured information (prices, specs, tables) from e‑commerce or news sites.
• Anyone who wants a free alternative to commercial services like Firecrawl or Jina Reader.