zengxiao-he/tessera

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

What it solves

Tessera provides a complete, from-scratch implementation of an LLM stack designed to bridge the gap between model distillation and efficient inference. It addresses the complexity of building an end-to-end pipeline by integrating training, custom GPU acceleration, and high-performance serving into a single, cohesive codebase.

How it works

The project follows a distillation-first workflow: it trains a small "student" model from a larger "teacher" model using custom-built sharded training (FSDP/ZeRO-3). Once trained, the student model is deployed using a custom inference engine that features paged KV caching, continuous batching, and speculative decoding. The system uses Triton and CUDA kernels for performance, with a Rust-based gateway (via PyO3) to handle HTTP requests and admission control.

Who it’s for

This project is for developers and researchers who want to understand or build the low-level mechanics of LLM systems, including custom GPU kernels, distributed training, and high-throughput serving architectures, without relying on massive, opaque frameworks.

Highlights

  • Full-Stack Implementation: Includes custom Triton/CUDA kernels, FSDP/ZeRO-3 training, and a paged-attention inference engine.
  • Advanced Serving: Features continuous batching, speculative decoding, and a Rust-based API gateway for production-ready request handling.
  • Distillation-Focused: Built specifically to distill large teacher models into efficient, small student models.
  • Verification-First: Includes a JAX/XLA reference implementation to verify PyTorch results and extensive testing for numerical parity across training and inference components.

Related

  • Project
    Mirrowel/LLM-API-Key-Proxy

    A self-hosted universal LLM API proxy that provides OpenAI and Anthropic compatible endpoints for multiple providers with built-in key rotation and failover.

  • Project
    Layr-Labs/d-inference

    Darkbloom’s d‑inference repo provides a Swift‑based, Apple‑Silicon‑only runtime that runs LLM inference locally with MLX and exposes an OpenAI‑compatible API. It’s the on‑device side of the Darkbloom platform, with clear separation from the backend/console code kept in a sibling repository.

  • Project
    FutureMLS-Lab/OSCAR

    OSCAR is a 2-bit KV cache quantization method that uses offline spectral covariance-aware rotations to reduce memory usage by 8x while maintaining near-BF16 accuracy.

  • Project
    matlab-deep-learning/MATLAB-Deep-Learning-Model-Hub

    A curated hub of pretrained deep learning models for MATLAB, covering computer vision, NLP, audio, Lidar, and robotics for use in inference and transfer learning.