Avarok-Cybersecurity/atlas

Pure Rust Inference Engine

What it solves

Atlas is a high-performance LLM inference engine designed to eliminate the dependency hell and instability often found in Python-based inference stacks. It aims to reduce reliance on expensive Cloud APIs by maximizing local inference speed through hardware-specific optimizations, allowing powerful models to run efficiently on local hardware.

How it works

Built entirely in Rust, Atlas uses a modular, "plug-and-play" architecture based on traits. This allows it to decouple the HTTP API and scheduler from the actual model logic and hardware backends. Instead of general-purpose kernels, Atlas employs hardware- and model-specific kernels (e.g., custom CUDA kernels for NVIDIA GB10) to achieve 2-3x speed increases. The system supports a variety of architectures including MoE, SSM, and Hybrid models, and utilizes a registry system to auto-discover the correct kernels at build time based on the hardware, model, and quantization target.

Who it’s for

It is designed for software engineers and the AI community who want a stable, high-speed local inference environment. It is particularly useful for those deploying on NVIDIA GB10 (DGX Spark) hardware, though it is architected to eventually support AMD, Apple Silicon, and Intel.

Highlights

  • Pure Rust Implementation: Avoids the complexity and instability of polyglot codebases.
  • Hardware-Specific Kernels: Custom-tuned kernels for specific hardware/model combinations to maximize throughput.
  • Modular Architecture: Trait-based design allows for easy integration of new models and hardware targets.
  • AI-Friendly Codebase: Specifically structured to encourage and facilitate AI-generated pull requests and contributions.
  • Broad Model Support: Ships with support for a wide array of modern architectures including Qwen, Gemma-4, Mistral, and Nemotron.

Related

  • Project
    lablup/mlxcel

    A high-performance Rust-based inference runtime and server for LLMs and VLMs on Apple Silicon and NVIDIA CUDA, removing the need for Python environments during deployment.

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Project
    EricLBuehler/mistral.rs

    A fast, flexible LLM inference engine written in Rust that supports multimodal models, automatic quantization, and a built-in agentic runtime for tool and code execution.

  • Project
    pegainfer-project/pegainfer

    A pure Rust and CUDA LLM inference engine that removes framework runtimes like PyTorch to provide faster startup times and a smaller memory footprint.

  • Project
    warpfront/hipfire

    A fast local LLM inference engine for AMD GPUs written in Rust and HIP, featuring an OpenAI-compatible API and a custom dispatch layer to minimize overhead.