Avarok-Cybersecurity/atlas

Pure Rust Inference Engine

What it solves

Atlas is a high-performance LLM inference engine designed to run large models efficiently across a wide range of hardware, from edge devices to data center racks. It eliminates the need for heavy dependencies like Python or PyTorch, providing a lightweight, single-binary runtime that maximizes throughput and reduces latency, particularly for agentic workloads involving frequent tool calls and long context windows.

How it works

Written in pure Rust and CUDA, Atlas uses hand-tuned CUDA kernels tailored to specific hardware, model, and quantization targets. It implements several advanced optimization techniques:

  • Speculative Decoding: Supports both MTP (Multi-Token Prediction) draft heads and DFlash block diffusion to increase tokens per second.
  • Memory Management: Utilizes radix-tree prefix caching for multi-turn conversations and FP8/NVFP4 quantization to reduce memory footprints.
  • SLAi Scheduler: An SLO-aware scheduler that reorders concurrent sequences to optimize verification batches and prefill efficiency.
  • Expert Parallelism: Enables scaling across multiple nodes (e.g., 2x GB10) for massive models.
  • Hardware Abstraction: Uses a plug-and-play architecture allowing the same CUDA source to be compiled for different targets (e.g., NVIDIA and AMD via SCALE).

Who it’s for

Atlas is built for developers and organizations deploying LLMs in production environments where performance, low overhead, and hardware flexibility are critical. It is specifically tuned for agentic-coding workloads (like Claude Code or Cline) that require high context windows and fast tool-use loops.

Highlights

  • Pure Rust Runtime: Ships as a single ~75 MB binary with no Python or PyTorch dependencies.
  • Broad Hardware Support: Verified on NVIDIA DGX Spark (GB10/Blackwell) and AMD Strix Halo, with early bring-up for Apple Silicon.
  • OpenAI Compatible: Serves OpenAI, Anthropic, and Responses APIs.
  • High Concurrency: Out-performs vLLM across a wide range of concurrency levels (C=1 to C=128).
  • Advanced Quantization: Supports NVFP4 and FP8 quantization for weights and KV caches.

Related

  • Project
    EricLBuehler/mistral.rs

    A fast, flexible LLM inference engine written in Rust that supports multimodal models, agentic tool execution, and OpenAI/Anthropic compatible APIs.

  • Project
    flashrt-project/FlashRT

    A high-performance real-time inference engine optimized for small-batch, latency-sensitive AI workloads across NVIDIA, AMD, and Ascend hardware.

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Project
    lablup/mlxcel

    A high-performance native Rust runtime for LLM, VLM, and audio inference on Apple Silicon, CUDA, and ROCm, removing Python from the production request path.

  • Project
    lucasjinreal/Crane

    Crane is a Rust‑based inference engine built on the Candle ML library. It loads GGUF models (LLMs, vision‑language, OCR, ASR, TTS, etc.) and runs them fast on CPU, CUDA, Apple Metal, or AMD ROCm, providing an OpenAI‑compatible server and easy‑to‑use Rust APIs.