fractalbits-labs/fractalbits

Ultra-high performance distributed storage system in the AI era. Powered by Fractal ART engine, Rust-native, io_uring, with multi-protocol (S3 object storage, POSIX FS, ...)

What it solves

FractalBits is a high-performance, S3-compatible object storage system designed to eliminate the high costs and operational complexity of services like AWS S3 Express One Zone. It specifically addresses the need for low-latency, high-throughput storage for small object workloads, which is often a bottleneck in AI training pipelines and data analytics workflows.

How it works

The system is built using Rust for the API gateway and control plane, and leverages io_uring for asynchronous I/O in the data plane. Its core is the Fractal ART (Adaptive Radix Tree) storage engine, which serves two purposes:

  1. Metadata Engine: Uses a full-path name approach to avoid heavy distributed transactions, enabling high scalability and native atomic rename support for objects and directories.
  2. Local Data Engine: An SSD-optimized path that allows performance to scale directly with NVMe hardware.

Who it’s for

It is primarily intended for engineers building AI training pipelines and data analytics platforms that require extremely fast access to millions of small objects and need atomic operations for managing datasets and checkpoints.

Highlights

  • Extreme Throughput: Capable of over 1.3 million 4KB-object reads per second with P99 latency around 3ms.
  • Atomic Renames: Provides native support for atomic renames of both objects and directories, a feature missing from standard S3.
  • Hardware Optimized: Specifically designed to scale with NVMe SSDs using a low-overhead I/O path.
  • Rapid Deployment: Supports "Bring Your Own Cloud" (BYOC) deployment to AWS or GCP in under 5 minutes.

Related

  • Project
    usnistgov/dioptra

    A software test platform from NIST for assessing the trustworthy characteristics of AI, providing tools to design, execute, and track experiments to ensure safety, fairness, and reliability.

  • Project
    pathwaycom/pathway-benchmarks

    A suite of reproducible benchmarks comparing the Pathway Live Data Framework's performance in streaming and batch data processing against Flink, Spark, and Kafka Streams.

  • Project
    langfuse/skills

    A collection of agent skills that teach AI coding assistants how to integrate, manage, and migrate to the Langfuse LLM engineering platform.

  • Project
    open-thoughts/OpenThoughts-Agent

    OpenThoughts‑Agent (OT‑Agent) is a research‑focused toolkit for generating training data, fine‑tuning, and evaluating small agentic language models. It bundles a datagen pipeline (Ray + vLLM), SFT support via LLaMA‑Factory, and a Harbor‑based evaluation framework, with launchers that work on SLURM clusters, cloud VMs (SkyPilot), or locally. The codebase is aimed at researchers building and benchmarking agentic LLMs.