laiso/site2pdf
Generate comprehensive PDFs of entire websites, ideal for RAG.
What it solves
It simplifies the process of converting an entire website into a single, portable PDF document. This is specifically designed to make it easier to feed website content into AI tools like Google NotebookLM or ChatGPT GPTs for Retrieval-Augmented Generation (RAG) and Question Answering (QA) tasks.
How it works
The tool uses Puppeteer to launch a headless Chrome browser, crawls a starting URL to find matching links based on a a provided pattern, and generates PDFs for each page concurrently. Finally, it merges these individual PDFs into one comprehensive document using the pdf-lib library.
Who it’s for
Developers and AI practitioners who need to convert web-based documentation or sites into a format that is easily uploadable to LLM-based knowledge bases.
Highlights
- AI-Ready: Specifically optimized for RAG and multimodal AI models by preserving visual formatting and images.
- Concurrent Processing: Speeds up the generation process by processing multiple pages in parallel.
- Flexible Filtering: Allows users to specify regex patterns to include only specific sections of a website.
- Portable Output: Consolidates multiple web pages into one shareable PDF file.
Related
- Project
- Project
- Project
- Project
- Project