Dicklesworthstone/llm_aided_ocr

Enhances Tesseract OCR output using LLMs (local or API) for error correction, smart chunking, and markdown formatting of scanned PDFs

What it solves

Raw OCR (Optical Character Recognition) output is often riddled with typos, formatting errors, and layout issues that make documents hard to read. This project provides a pipeline to clean up these errors and transform raw text into polished, well-formatted Markdown documents.

How it works

The system follows a multi-step processing pipeline:

  1. PDF Conversion: Converts PDF pages into images using pdf2image.
  2. OCR Extraction: Uses Tesseract OCR to extract raw text from those images, applying grayscale and binary thresholding to improve clarity.
  3. Text Chunking: Splits the raw text into manageable chunks with overlaps to maintain context for the LLM.
  4. LLM Correction: Sends these chunks to a Large Language Model (either via API like OpenAI/Anthropic or locally via llama_cpp) to fix OCR errors and restore the original structure.
  5. Formatting: Optionally converts the corrected text into Markdown, removing duplicate paragraphs and suppressing headers or page numbers.
  6. Quality Assessment: Uses an LLM to compare the final output against the original raw OCR text to provide a quality score.

Who it’s for

Users who need to digitize scanned PDFs or old documents and want a high-accuracy, readable version of the text without manual proofreading.

Highlights

  • Flexible LLM Backend: Supports cloud APIs (OpenAI, Anthropic) and local GGUF models for privacy or cost.
  • Asynchronous Processing: Uses asyncio to process text chunks concurrently when using APIs, speeding up the overall workflow.
  • Adaptive Token Management: Dynamically adjusts request sizes based on model limits and prompt length to avoid truncation.
  • Automated Quality Control: Includes a built-in LLM-based evaluation step to assess the accuracy of the correction process.

Related

  • Project
    drmingler/smart-llm-loader

    A Python package that converts documents and scanned images into clean, structured markdown chunks using OCR and LLM-powered semantic chunking for RAG systems.

  • Project
    ocrmypdf/OCRmyPDF

    A command-line tool that adds an OCR text layer to scanned PDFs, making them searchable and copy-pasteable while maintaining image quality.

  • Project
    pymupdf/pymupdf4llm

    A lightweight extension for PyMuPDF that converts PDFs and other documents into structured Markdown, JSON, and plain text optimized for RAG pipelines and LLM ingestion.

  • Project
    bhimrazy/receipt-ocr

    An OCR engine that extracts either raw text or structured JSON data from receipt images using Tesseract and LLMs.

  • Project
    PaddlePaddle/PaddleOCR

    PaddleOCR is an open‑source OCR and document‑AI toolkit that converts images/PDFs into structured, LLM‑ready JSON or Markdown. It offers multilingual scene‑text models (PP‑OCRv6), a lightweight vision‑language parser (PaddleOCR‑VL‑1.6) for tables, formulas, and ancient scripts, and a structure‑aware pipeline (PP‑StructureV3). Models run on CPU, GPU, XPU, NPU or via ONNX/TensorRT, and integrate with RAG/agent platforms like Dify and RAGFlow.