bacalhau-project/bacalhau — Gitpedia

<p align="center"> <a href="https://github.com/bacalhau-project/bacalhau"> <img src="./docs/logo/Bacalhau-horizontal.svg" alt="Bacalhau" width="300"/> </a> </p> <h1 align="center">Globally Distributed Compute Orchestrator ⚡<br>Compute Over Data (CoD)</h1> <br> <p align="center"> <a href="https://github.com/bacalhau-project/bacalhau/blob/main/LICENSE" alt="License"> <img src="https://img.shields.io/badge/license-Apache-green" /> </a> <a href="https://github.com/bacalhau-project/bacalhau/releases/" alt="Release"> <img src="https://img.shields.io/github/v/release/bacalhau-project/bacalhau?display_name=tag" /> </a> <a href="https://github.com/bacalhau-project/bacalhau/pulse" alt="Activity"> <img src="https://img.shields.io/github/commit-activity/m/bacalhau-project/bacalhau" /> </a> <a href="https://github.com/bacalhau-project/bacalhau/graphs/contributors"> <img src="https://img.shields.io/github/contributors/bacalhau-project/bacalhau" alt="Bacalhau contributors" > </a> <a href="https://www.bacalhau.org/"> <img alt="Bacalhau website" src="https://img.shields.io/badge/website-bacalhau.org-red"> </a> <a href="https://bit.ly/bacalhau-project-slack" alt="Slack"> <img src="https://img.shields.io/badge/slack-join_community-red.svg?color=0052FF&labelColor=090422&logo=slack" /> </a> <a href="https://twitter.com/intent/follow?screen_name=BacalhauProject"> <img src="https://img.shields.io/twitter/follow/BacalhauProject?style=social&logo=twitter" alt="follow on Twitter"> </a> </p> <p align="center"> <a href="https://github.com/bacalhau-project/bacalhau/actions/workflows/main.yml"> <img src="https://github.com/bacalhau-project/bacalhau/actions/workflows/main.yml/badge.svg" alt="Main Workflow" /> </a> <a href="https://github.com/bacalhau-project/bacalhau/actions/workflows/release.yml"> <img src="https://github.com/bacalhau-project/bacalhau/actions/workflows/release.yml/badge.svg" alt="Release Workflow" /> </a> <a href="https://github.com/bacalhau-project/bacalhau/actions/workflows/pr-checks.yml"> <img src="https://github.com/bacalhau-project/bacalhau/actions/workflows/pr-checks.yml/badge.svg" alt="PR Checks Workflow" /> </a> </p>

What is Bacalhau?

Bacalhau is an open-source distributed compute orchestration framework designed to bring compute to the data. Instead of moving large datasets around networks, Bacalhau makes it easy to execute jobs close to the data's location, drastically reducing latency and resource overhead.

Why Bacalhau?

⚡ Fast job processing: Jobs in Bacalhau are processed where the data was created and all jobs are parallel by default
💰 Low cost: Reduce (or eliminate) ingress/egress costs since jobs are processed closer to the source
🔒 Secure: Data scrubbing and security can happen before migration, with a granular, code-based permission model
🚛 Large-scale data: Process petabytes of data efficiently without massive data transfers
🏢 Data sovereignty: Process sensitive data within security boundaries without requiring it to leave your premises
🤝 Cross-organizational computation: Allow specific vetted computations on protected datasets without exposing raw data

Key Features

Single Binary Simplicity: Bacalhau is a single self-contained binary that functions as a client, orchestrator, and compute node—making it incredibly easy to set up and scale
Modular Architecture: Support for multiple execution engines (Docker, WebAssembly) and storage providers through clean interfaces
Orchestrator-Compute Model: A dedicated orchestrator coordinates job scheduling, while compute nodes run tasks
Flexible Storage Integrations: Integrates with S3, HTTP/HTTPS, IPFS, and local storage systems
Multiple Job Types: Support for batch, ops, daemon, and service job types for different workflow requirements
Declarative & Imperative Submissions: Define jobs in YAML (declarative) or pass arguments via CLI (imperative)
Publisher Support: Output results to local volumes, S3, or other storage backends

Getting Started

Quick Installation

bash
# Install Bacalhau CLI (Linux/macOS)
curl -sL https://get.bacalhau.org/install.sh | bash

# Verify installation
bacalhau version

For the complete quick start guide, including running your first job, see our Quick Start Documentation.

Use Cases

Bacalhau's distributed compute framework enables a wide range of applications:

Log Processing: Process logs efficiently at scale by running distributed jobs directly at the source
Distributed Data Warehousing: Query and analyze data across multiple regions without moving large datasets
Fleet Management: Efficiently manage distributed nodes across multiple environments
Distributed Machine Learning: Train and deploy ML models across a distributed compute fleet
Edge Computing: Run compute tasks closer to the data source for applications requiring low latency

Documentation

📚 Read the Bacalhau docs guide here! 📚

The Bacalhau documentation contains all the information you need to get started:

Community & Contributing

Bacalhau has a very friendly community, and we are always happy to help:

Join the Slack Community and go to the #general channel - it is the easiest way to engage with other members in the community and get help

If you are interested in contributing to the Bacalhau project:

Set up your local environment
Check out our Contributing Guide
For issues and feature requests, please open a GitHub issue

We are excited to hear your feedback!

Open Source

This repository contains the Bacalhau software, covered under the Apache-2.0 license, except where noted (any Bacalhau logos or trademarks are not covered under the Apache License, and should be explicitly noted by a LICENSE file.)

Bacalhau is a product produced from this open source software, exclusively by Expanso, Inc. It is distributed under our commercial terms.

Others are allowed to make their own distribution of the software, but they cannot use any of the Bacalhau trademarks, cloud services, etc.

We explicitly grant permission for you to make a build that includes our trademarks while developing Bacalhau software itself. You may not publish or share the build, and you may not use that build to run Bacalhau software for any other purpose.

We have borrowed the above Open Source clause from the excellent System Initiative