Oxen-AI/Oxen

Lightning fast data version control system for large repositories of data. Feels like git, pushes and pulls like oxen.

What it solves

Oxen is a data version control system designed specifically for large datasets, addressing the limitations of Git and Git LFS. It allows users to version, track changes, and collaborate on massive amounts of data—including machine learning training data and model weights—without the the performance bottlenecks associated with traditional version control systems.

How it works

Oxen uses a Git-like interface (commands like init, add, commit, and push) to make versioning data as intuitive as versioning code. It is built in Rust for high performance, allowing it to index hundreds of thousands of files in seconds. The system supports various data types (images, audio, video, text, and tabular data like Parquet files) and uses specialized metadata extractors and a merkle tree for fast access. It can be used via a CLI, a Python package, or an HTTP API, and can be hosted on Oxen.ai or self-hosted via oxen-server.

Who it’s for

It is primarily aimed at machine learning engineers and data scientists who manage large-scale datasets for training and evaluation, as well as developers working with game assets or studio media.

Highlights

  • Git-like Ergonomics: Uses familiar commands, making it easy for anyone who knows Git to adopt.
  • High Performance: Optimized for indexing and syncing millions of files and terabytes of data.
  • Native Tabular Data Handling: Specifically designed to index and query CSV, Parquet, and JSONL files.
  • Flexible Deployment: Supports CLI, Python bindings, and self-hosting options.
  • Cross-Platform Support: CLI and Python client work on Linux, macOS, and Windows.

Related

  • Dispatch
    Oak v0.99.0: A Git Replacement Designed for AI Agents

    Oak v0.99.0 is a content-addressed version control system designed to optimize agentic workflows through lazy mounts, branch-per-session units of work, and machine-readable JSON surfaces.

  • Project
    treeverse/dvc

    A command line tool and VS Code extension for versioning data and ML models, tracking experiments, and managing reproducible ML pipelines.

  • Project
    t8y2/dbx

    DBX is a lightweight database management tool that supports over 100 different databases in a single 25 MB package, available as a desktop app, CLI, Docker image, and featuring a built-in AI assistant and MCP server.

  • Project
    sleuth-io/sx

    sx is a zero‑infrastructure vault system for sharing AI assets (skills, agents, rules, commands, hooks, MCP servers) across teams. It works via a synced folder or Git repo, offering an app and CLI that manage versioning, scoped installation, usage stats, and audit logs, and integrates with many AI assistants (Claude, Copilot, Cursor, Gemini, etc.).

  • Project
    aoci-spec/aoci-code

    AOCI‑CODE is an open‑source CLI and local MCP server that builds a Git‑versioned plain‑text index of a code repository (one concise line per file describing purpose, dependencies, public symbols, and constraints). The index lets AI coding assistants read the whole system in one pass, avoiding repeated code‑base scans, and is kept automatically up‑to‑date as the code changes.