geerlingguy/beowulf-ai-cluster

AI Cluster deployed with Ansible on Random computers with random capabilities

What it solves

It provides a streamlined way to set up and benchmark distributed AI clusters. Instead of manually configuring multiple machines to work together to run large models, this project automates the deployment of various distributed inference tools across a cluster of nodes.

How it works

The project uses Ansible playbooks to automate the installation, compilation, and execution of AI software across multiple hosts. It supports three primary distributed AI frameworks:

  • llama.cpp (RPC mode): Configures nodes as RPC servers to share the workload of a single model.
  • distributed-llama: Sets up worker nodes to perform distributed inference.
  • Exo: Provides a setup playbook for manual deployment and testing.

Who it’s for

It is designed for users who want to test distributed AI clustering tools, compare the performance of different hardware nodes, or benchmark how multiple machines scale when running large language models.

Highlights

  • Automated Deployment: Uses Ansible to handle setup and compilation across all cluster nodes.
  • Multiple Framework Support: Includes benchmarks for llama.cpp (individual and cluster), distributed-llama, and Exo.
  • Flexible Configuration: Allows users to define their cluster inventory and hardware-specific build options via configuration files.
  • Manual Debugging: Provides detailed instructions for running manual benchmarks to debug larger models.

Related

  • Project
    dell/omnia

    An open-source deployment toolkit using Ansible to automate the provisioning of servers, schedulers like Slurm and Kubernetes, and AI frameworks for compute clusters.

  • Project
    b4rtaz/distributed-llama

    A distributed inference engine that clusters home devices over Ethernet to share RAM and accelerate LLM performance using tensor parallelism.

  • Project
    ggml-org/llama.cpp

    A high-performance C/C++ implementation for local LLM and VLM inference, supporting a wide range of hardware through quantization and hardware acceleration.

  • Project
    docker/model-runner

    A tool for managing and deploying AI models via Docker, streamlining the process of pulling and serving LLMs using backends like llama.cpp and vLLM.

  • Project
    Mesh-LLM/mesh-llm

    A distributed inference system that pools GPUs and memory across multiple machines to serve large language models via an OpenAI-compatible API.