rustfs/rustfs

RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.

RustFS – High‑Performance Distributed Object Storage (Rust)

What it is – RustFS is an open‑source, Apache‑2.0‑licensed object‑storage system written in Rust. It aims to combine the ease‑of‑use of MinIO with the memory‑safety and raw speed that Rust provides. The service speaks the S3 API (core features, versioning, encryption, lifecycle, etc.) and also supports OpenStack Swift, SFTP/FTPS/WebDAV, and a web console.

Why it matters for AI/ML – Modern AI pipelines need fast, reliable object stores for training data, model artifacts, and inference results. RustFS advertises "Data Lake Support" and "optimized for high‑throughput big data and AI workloads," making it a candidate for the storage layer of data‑intensive ML workloads.

Key Features (as listed in the README)

High performance – built in Rust for low‑overhead, memory‑safe execution.
Distributed, fault‑tolerant architecture – can run as a single‑node or multi‑node cluster with erasure‑coded pools.
Broad S3 compatibility – core S3 operations, versioning, object lock (WORM), server‑side encryption, lifecycle/ILM, bucket replication, event notifications, etc.
OpenStack Swift API & Keystone auth – native support for Swift and its authentication model.
Additional protocols – FTPS, WebDAV, SFTP (opt‑in cargo features).
KMS integration – Vault (KV2/Transit) and AWS KMS back‑ends; local/static back‑ends for testing.
Observability – built‑in logging, Prometheus metrics, Grafana dashboards, Jaeger tracing.
Web console – a graphical UI for bucket and object management.
Helm chart, Docker images, Nix flake, and x‑cmd installer – multiple deployment options.
Permissive Apache‑2.0 license – no “viral” AGPL restrictions.

Typical Use Cases

Use case	How RustFS helps
AI/ML data lake	S3‑compatible high‑throughput storage for large training datasets and model checkpoints.
Edge / IoT gateways	Small‑footprint Rust binary, no telemetry, suitable for on‑prem or edge deployments.
Hybrid cloud	Can be run on‑premises or in containers/Kubernetes, providing a private S3 endpoint that works with existing cloud tools.
Enterprise compliance	Apache‑2.0 license, GDPR/CCPA/APPI‑friendly, no hidden telemetry, supports OIDC/SSO for fine‑grained IAM.
Backup / Archival	Supports bucket replication, versioning, and object lock for immutable storage.

Getting Started (quick‑start options)

One‑click script – curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh

Docker – run the official image (non‑root user rustfs UID/GID 10001:10001). Example:

mkdir -p data logs && chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 \
    -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

Replace latest with a specific version tag if needed.

Docker Compose – a ready‑made docker-compose-simple.yml includes RustFS plus Prometheus, Grafana, and Jaeger for observability.
Helm chart – install on a Kubernetes cluster via the chart at https://charts.rustfs.com/.
Build from source – use the provided docker-buildx.sh script or make targets for multi‑arch Docker images, or compile the binary directly with Cargo.
Nix flake / x‑cmd – for NixOS users or those preferring the x-cmd installer.

Administration Highlights

Console – reachable at http://localhost:9001 (default creds rustfsadmin / rustfsadmin).
TLS – enable by mounting a certificate directory and setting RUSTFS_TLS_PATH.
Notifications – webhook support via RUSTFS_NOTIFY_* env vars; outbound connections must be whitelisted via RUSTFS_OUTBOUND_ALLOW_ORIGINS.
Pool expansion – single‑node single‑disk deployments cannot expand in‑place; multi‑drive pools require careful geometry (see the pool‑layout compatibility docs).
OIDC – optional roles claim mapping for Microsoft Entra ID; configure via RUSTFS_IDENTITY_OPENID_* environment variables.

License

Apache License 2.0 – permissive, commercial‑friendly, no “viral” clauses.

Where to Learn More

Docs & installation guide – https://docs.rustfs.com/en/installation
GitHub repo – https://github.com/rustfs/rustfs (CI badges, issue tracker, discussions)
Community – Discord link in the README, plus multilingual README links.

TL;DR

RustFS is a Rust‑based, Apache‑2.0‑licensed object storage system that offers full S3 compatibility, distributed fault‑tolerant operation, and a web console. It targets high‑throughput AI/ML data‑lake workloads while providing edge‑friendly deployment options (Docker, Helm, Nix, etc.) and strong security/compliance guarantees.