lablup/backend.ai

What it solves

Backend.AI is a container-based computing cluster platform designed to simplify the hosting and management of machine learning frameworks and programming languages across heterogeneous hardware. It solves the problem of resource allocation and isolation for multi-tenant environments, allowing users to on-demand access to diverse accelerators (GPUs and NPUs) without complex manual configuration.

How it works

The platform uses a custom orchestrator called "Sokovan" to allocate and isolate computing resources for computation sessions. It operates through several core components:

• Manager: Acts as the central API gateway and orchestrator for session scheduling and user authentication.
• Agent: Manages the lifecycle of Docker containers (kernels) on individual compute nodes.
• Storage Proxy: Provides an abstraction layer called "vfolders" (virtual folders) to manage network-based storage (NFS/SMB) across sessions.
• App Proxy: Routes traffic to in-container services like Jupyter, VSCode, and web-based terminals.
• Webserver: Hosts the user interface for session management.

Who it’s for

It is designed for organizations managing compute clusters for data scientists and AI researchers who need secure, isolated, and flexible access to shared hardware accelerators.

Highlights

• Broad Accelerator Support: Native plugins for CUDA, ROCm, Google TPU, Intel Gaudi, Tenstorrent, Graphcore IPU, and various NPUs (FuriosaAI, Rebellions, HyperAccel).
• Multi-tenant Isolation: Container-based isolation with a programmable sandbox (Jail) and system call filtering.
• ** uma-on-demand sessions**: Supports direct browser-based access to Jupyter, VSCode, and SSH/SFTP/SCP.
• Extensible Architecture: Plugin-based system for accelerators and monitoring tools (e.g., Datadog, Sentry).
• API-First: All functions are exposed via REST and GraphQL APIs.