vakra-dev/reader

Reader – Production‑grade web‑scraping for AI agents

What it is – A Node‑JS library (and accompanying CLI) that wraps Playwright to give you a ready‑made, stealthy browser pool. It hides the usual headaches of web‑automation – Cloudflare challenges, TLS fingerprinting, proxy rotation, resource limits – and returns clean markdown or HTML that LLM‑powered agents can consume.

Key primitives

1. scrape – give a list of URLs, get back the main article (or full page) as markdown/HTML.
2. crawl – breadth‑first discovery of links from a seed URL, optionally scraping each page.
3. browser – spin up a stealthed Chrome session (CDP endpoint) that you can drive with Playwright or Puppeteer for custom logic.

Why it matters for AI – Retrieval‑augmented generation and autonomous agents need reliable, up‑to‑date web content. Reader provides a stable, production‑ready way to fetch that content without building a bespoke scraper that constantly breaks on anti‑bot measures.

Main features

• Stealth browser – TLS fingerprinting, navigator spoofing, WebRTC masking, webdriver=false.
• Browser pool – configurable size, automatic recycling after a set number of pages or minutes.
• Proxy tiers – built‑in support for standard (datacenter) and premium (residential) proxy pools, with health monitoring and rotation.
• Content cleaning – automatic main‑content extraction, removal of navbars, footers, pop‑ups, cookie banners.
• CLI & API – use npx reader … for quick scripts or import ReaderClient in TypeScript/JavaScript projects.
• Concurrency – batch scraping with configurable parallelism and progress callbacks.

Typical usage

import { ReaderClient } from "@vakra-dev/reader";
const r = new ReaderClient();
const res = await r.scrape({
  urls: ["https://example.com"],
  formats: ["markdown", "html"],
  proxyTier: "premium",
  batchConcurrency: 3,
});
console.log(res.data[0].markdown);
await r.close();

Or launch a full browser for custom navigation:

const sess = await r.browser();
const { chromium } = require("playwright-core");
const browser = await chromium.connectOverCDP(sess.wsEndpoint);
const page = await browser.newPage();
await page.goto("https://news.ycombinator.com");
console.log(await page.title());
await sess.close();

Installation – npm install @vakra-dev/reader (Node ≥ 18). Playwright’s Chromium is bundled; run npx playwright install chromium on first use.

Who should consider it – developers building LLM agents, RAG pipelines, or any automation that needs reliable, stealthy web access at scale.