Oxen-AI/Oxen

What it solves

Oxen is a data version control system designed specifically for large datasets. It addresses the performance bottlenecks and ergonomic issues found in tools like Git LFS, which can be slow and unwieldy when handling millions of files or terabytes of data, especially for machine learning training data and model weights.

How it works

Oxen provides a Git-like interface (using commands like init, add, commit, and push) to version data. It is built in Rust for high performance and offers a CLI, a Python package, and an HTTP API. It uses a merkle tree to cache metadata for fast access and specialized extractors for various file types. Users can host their data on Oxen.ai or self-host using the oxen-server.

Who it’s for

It is primarily for machine learning engineers and data scientists who need to version large-scale datasets (images, audio, video, text, and tabular data) and collaborate with teams.

Highlights

• Git-like Ergonomics: Uses familiar commands, making it easy for anyone who knows Git to adopt.
• High Performance: Optimized for indexing and syncing hundreds of thousands of files in seconds.
• Broad Data Support: Handles large files including model weights, Parquet, JSONL, and media files.
• Native Tabular Data Handling: Ability to index and query CSV, Parquet, and JSONL files natively.
• Flexible Deployment: Supports both a managed cloud service (Oxen.ai) and self-hosting via oxen-server.