kreeben/resin

What it solves

Resin provides a high-performance foundation for building search engines and machine learning models. It solves the problem of efficiently storing and retrieving large volumes of data—such as strings, byte arrays, and vectors—while supporting fast key/value lookups and complex set operations across different data columns.

How it works

Resin implements a custom storage system using three specialized streams: a key stream (.key) for sorted keys, an address stream (.adr) for offsets and lengths, and a value stream (.val) for the actual data. It uses an append-only value stream to ensure stable offsets and linear scaling. The system supports both single-value storage and linked-value chains, allowing multiple values to be associated with a single key without duplicating the key itself. For text analysis, it includes utilities for string processing, bag-of-words/chars analysis, and vector operations.

Who it’s for

It is designed for developers building vector space search engines, custom vector databases, or applications that require fast, low-dependency on-disk key/value storage and text analysis tooling.

Highlights

• Append-only storage: Prevents fragmentation and ensures stable offsets for efficient caching.
• Vector space search: Built-in support for vector operations and similarity tooling.
• Set operations: Column-based architecture enables efficient unions, intersections, and joins.
• Text analysis utilities: Includes tools for analyzing strings and creating lexicons.
• Lightweight design: Minimal dependencies for easy extensibility.