databricks/mlops-stacks

What it solves

Databricks MLOps Stacks provides a standardized, production-ready template for starting new machine learning projects. It eliminates the need for data scientists and operations engineers to manually configure CI/CD pipelines, resource management, and project structures from scratch, ensuring that best practices for productionization are integrated from the start.

How it works

The project is implemented as a Databricks asset bundle template. Users initialize a new project using the Databricks CLI (databricks bundle init mlops-stacks), which generates a customizable project structure based on user-supplied parameters.

It organizes the ML pipeline into three modular components:

• ML Code: Provides a structure for training and batch inference with unit-tested Python modules and notebooks.
• ML Resources as Code: Uses Databricks CLI bundles to define pipeline resources (like training and batch inference jobs) as code, allowing changes to be managed via pull requests.
• CI/CD: Pre-configured workflows for GitHub Actions, Azure DevOps, or GitLab to automate testing and deployment across dev, staging, and production environments.

Who it’s for

• Data Scientists: Who want to quickly iterate on ML code without worrying about the eventual refactoring for production.
• ML Engineers (MLEs): Who need to set up and govern the deployment pipelines and resource management for ML projects.

Highlights

• Production-Grade CI/CD: Pre-configured automation for testing and deploying code and resources across multiple environments (dev, staging, prod).
• Modular Design: Components can be adopted individually or customized to fit organizational best practices.
• Resource Governance: ML resources are managed as code, enabling auditing and deployment through pull requests rather than manual UI changes.
• Multi-Cloud Support: Compatible with AWS, Azure, and GCP.
• Optional Feature Store Integration: Includes optional components for managing feature engineering and training with Databricks Feature Store.