bacalhau-project/bacalhau

What it solves

Bacalhau addresses the inefficiency and high cost of moving massive datasets across networks for processing. Instead of transferring data to a central compute cluster, it implements a "Compute Over Data" (CoD) approach, executing jobs directly where the data resides to reduce latency, eliminate egress costs, and maintain data sovereignty.

How it works

Bacalhau uses a distributed orchestration framework consisting of a single binary that can act as a client, orchestrator, or compute node. An orchestrator coordinates job scheduling, while compute nodes execute tasks using engines like Docker or WebAssembly. It integrates with various storage providers (S3, IPFS, HTTP/HTTPS, local) to locate and process data in situ, allowing users to submit jobs either declaratively via YAML or imperatively via a CLI.

Who it’s for

It is designed for organizations and developers managing large-scale data (petabytes) across distributed environments, such as those performing distributed machine learning, log processing, edge computing, or cross-organizational computation on protected datasets.

Highlights

• Compute Over Data: Processes jobs where data was created to minimize movement.
• Single Binary: Simplifies setup and scaling by combining client, orchestrator, and node roles.
• Flexible Execution: Supports multiple engines including Docker and WebAssembly.
• Broad Storage Support: Integrates with S3, IPFS, and local storage.
• Data Sovereignty: Enables processing of sensitive data within security boundaries without raw data exposure.