drmingler/smart-llm-loader

smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications.

What it solves

SmartLLMLoader is designed to simplify the document preprocessing pipeline for LLM applications. It solves the problem of "messy" document extraction where traditional tools often lose the structure, tables, and semantic meaning of a document, which in turn degrades the performance of RAG (Retrieval-Augmented Generation) systems.

How it works

The package transforms documents into clean, LLM-ready markdown. It uses a combination of OCR for scanned images and PDFs, and integrates with various LLM providers via litellm to perform intelligent, context-aware chunking. Instead of splitting text by a fixed character count, it identifies semantic themes and preserves table formatting to ensure the LLM receives high-quality, structured data.

Who it’s for

Developers building RAG systems, chatbots, and document Q&A applications who need a reliable way to convert complex documents (including those with tables and scanned content) into structured chunks for vector stores.

Highlights

  • Multi-modal Support: Built-in OCR for scanned documents and images.
  • Context-Aware Chunking: Supports contextual, page-based, and custom chunking strategies to preserve semantic meaning.
  • Broad Integration: Seamlessly works with LangChain and LlamaIndex.
  • Provider Agnostic: Supports multiple LLM providers (e.g., Gemini, OpenAI, Anthropic) through litellm.
  • Structure Preservation: Maintains table formatting and adds intelligent metadata tagging to chunks.

Related

  • Project
    paulpierre/markdown-crawler

    A multithreaded web crawler that converts website pages into markdown files to simplify document parsing and chunking for RAG and LLM fine-tuning.

  • Project
    Dicklesworthstone/llm_aided_ocr

    An advanced OCR pipeline that uses LLMs to correct errors in raw Tesseract OCR output and transform it into high-quality, formatted Markdown documents.

  • Project
    docling-project/docling

    A document parsing tool that converts complex PDFs, Office files, and multimodal content into structured Markdown or JSON for use in generative AI and RAG pipelines.

  • Project
    datalab-to/marker

    A high-accuracy document conversion tool that transforms PDFs and other files into structured Markdown, JSON, or HTML, specializing in tables and equations.

  • Project
    aryn-ai/sycamore

    An AI-powered document processing engine for ETL and RAG that transforms unstructured documents into high-quality data chunks for vector databases.