aryn-ai/sycamore

What it solves

Sycamore addresses the difficulty of processing unstructured data from complex documents (like PDFs, reports, and presentations) for use in AI applications. It solves the problem of poor data chunking and low recall in RAG (Retrieval-Augmented Generation) systems by preserving the semantic structure of documents, including tables, figures, and graphs, during the extraction process.

How it works

Sycamore uses a "DocSet" abstraction to manage the transformation and manipulation of unstructured documents. It integrates with Aryn DocParse, a GPU-powered API that uses a deep learning DETR AI model to segment, label, and perform OCR on documents. Users can apply high-level Python transformations to clean, enrich, and extract data, then load the resulting high-quality chunks into various vector databases or hybrid search engines.

Who it’s for

It is designed for developers and data engineers building ETL pipelines for RAG, LLM-based applications, and analytics on unstructured data.

Highlights

• Semantic Document Partitioning: Uses vision AI to preserve the structure of complex documents, including tables and infographics.
• Scalable Processing: Built on a Ray backend to handle large-scale data processing.
• Flexible Integration: Supports multiple vector databases including Pinecone, Weaviate, Qdrant, OpenSearch, ElasticSearch, and DuckDB.
• Extensible Transforms: Provides LLM-powered user-defined functions (UDFs) and automatic data crawlers for S3 and HTTP.