ekzhang/resin

What it solves

Resin provides a set of low-level building blocks for developers who want to create custom LLM inference engines. Unlike "all-in-one" inference engines, it allows you to build specialized systems for new models or specific hardware without being locked into a pre-packaged solution.

How it works

Resin separates the inference process into two parts: a high-performance Rust "Engine" and a model-specific runner. The Rust Engine handles the heavy lifting—such as batch scheduling, memory management for caches, and metrics—while you write the forward pass (the actual model logic) as kernels optimized for your specific hardware. This modular design allows you to plug in different runners for CUDA, PyTorch, or Apple Silicon (MLX) while using the same core scheduling logic.

Who it’s for

This project is intended for developers and researchers who want to experiment with or optimize LLM inference systems. It is not designed for general production use; the authors recommend existing tools like vLLM or SGLang if you need a ready-to-use production engine. Resin is specifically for those who want to "hack" on inference, develop new model support, or push performance boundaries on custom hardware.

Highlights

• Modular Architecture: Decouples the batch scheduler from the model-specific forward pass.
• High-Performance Core: Includes a Rust-based scheduler, slab-based cache pool, and support for speculative decoding.
• Hardware Agnostic: Supports diverse backends including CUDA (with graph support) and Apple Silicon (via MLX).
• Performance Tools: Comes with built-in benchmarking, profiling, and an "autoresearch" workflow to help diagnose bottlenecks and validate performance improvements.
• Flexible Deployment: Includes optional HTTP serving compatible with common APIs like OpenAI and Anthropic.