InfuseAI/ArtiVC

What it solves

ArtiVC provides a way to version large datasets on cloud storage, similar to how Git handles code. It eliminates the need for a separate server or gateway, allowing users to snapshot their data and switch between different versions without changing where their files are stored.

How it works

It operates as a command-line tool that integrates with existing cloud environments. It supports native integration with AWS S3, Google Cloud Storage, Azure Blob Storage, and remote filesystems via SSH. It also supports over 40 additional backends through Rclone. To minimize storage and transfer costs, it uses a specific file structure design that prevents duplicating content and reduces the number of files uploaded during new version pushes.

Who it’s for

Data scientists and engineers who manage large files on cloud storage or NFS and need a way to track changes, tag versions, and diff between commits of their data.

Highlights

• Cloud Native: Works directly with AWS, GCP, and Azure without requiring a dedicated server.
• Harnesses Existing Auth: Uses standard cloud CLI configurations (like aws configure or gcloud auth) for access.
• Efficient Storage: Reduces data duplication and optimizes transfer speeds during updates.
• Lacks Server Overhead: Operates entirely as a CLI tool, removing the need for infrastructure management.
• Broad Compatibility: Supports local, SSH, and 40+ Rclone-compatible backends.