AgnostiqHQ/covalent

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

What it solves

Covalent simplifies the process of running compute-intensive AI, ML, and scientific research code across diverse environments. It removes the need for developers to manually manage complex cloud consoles, Terraform, or Infrastructure-as-Code (IaC) scripts, allowing them to deploy code to any cloud or on-prem cluster without changing their core business logic.

How it works

Covalent acts as an abstraction layer between the Python code and the underlying infrastructure. By using specific executor plugins, users can swap the execution environment (e.g., from a local machine to AWS, GCP, Azure, or a SLURM cluster) by changing a single line of code (the decorator). It effectively converts existing infrastructure into a serverless-like setup where functions can be dispatched to remote resources.

Who it’s for

  • AI/ML Practitioners and Developers: Those building LLMs, Generative AI applications, and other high-compute tasks who need a cloud-agnostic backend.
  • Researchers: Scientists using HPC clusters (like SLURM, PBS, LSF) who want to run experiments from their laptops without needing SSH or complex scripts.

Highlights

  • Cloud-Agnostic Execution: Seamlessly switch between different cloud providers and on-prem clusters.
  • Infrastructure Abstraction: Keeps business code independent from resource definitions.
  • Serverless Conversion: Turns on-prem SLURM clusters or cloud compute into serverless infrastructure.
  • Observability: Includes a user-friendly UI for real-time monitoring of jobs.
  • Extensive Plugin Ecosystem: Supports AWS, Azure, GCP, Kubernetes, SLURM, Dask, and IBMQ.

Related

  • Project
    quiltdata/quilt

    A scientific data management platform on AWS that turns cloud data into versioned, context-rich packages to ensure data is findable and reusable for AI and research.

  • Project
    fal-ai/fal

    A serverless Python runtime and client library for deploying and scaling ML models and pipelines in the cloud without managing infrastructure.

  • Project
    googlecolab/colabtools

    A collection of Python libraries that power Google Colaboratory, a zero-setup Jupyter notebook environment designed for machine learning research and education.

  • Project
    transformerlab/transformerlab-app

    An open-source machine learning platform that unifies training, fine-tuning, inference, and evaluation into a single interface for individuals and research labs.