treeverse/dvc

What it solves

DVC (Data Version Control) addresses the challenge of reproducibility in machine learning projects. It allows developers to version large datasets and ML models—which are too large for Git—while keeping the versioning metadata in Git, ensuring that specific versions of code, data, and models are always linked.

How it works

DVC acts as a layer on top of Git. It stores actual data and model files in a separate cache outside of the Git repository, while placing lightweight placeholder files in Git. To share and back up this cache, DVC supports various remote storage platforms including S3, Azure, Google Cloud, and on-premise network storage via SSH.

It also provides a system for defining computational graphs (pipelines) that connect code and data. These pipelines specify the dependencies and outputs of each step, allowing DVC to only run the steps impacted by changes.

Who it’s for

Machine learning engineers and data scientists who need to track experiments, version their data and models, and collaborate on reproducible ML pipelines.

Highlights

• Git for Data: Version and share data artifacts and models without needing a dedicated server.
• Lightweight Pipelines: Versioned data pipelines that only execute the steps affected by by changes.
• Local Experiment Tracking: Track and compare hyperparameters, metrics, and performance plots directly in a local Git repo.
• Cloud Storage Integration: Seamlessly push and pull data from cloud providers like S3, Azure, and GCS.
• VS Code Extension: A GUI for experiment tracking and data management within the IDE.