perplexityai/pplx-garden

What it solves

It provides a collection of high-performance inference technologies designed to optimize how large language models (LLMs) are deployed and run, focusing on communication efficiency, tokenization, and hardware-specific acceleration.

How it works

The repository acts as a "garden" of specialized tools:

• fabric-lib: Uses RDMA (Remote Direct Memory Access) for point-to-point communication to speed up data transfer in LLM systems, specifically for Mixture-of-Experts (MoE) dispatch and combine kernels.
• pplx-unigram: A high-performance Unigram tokenizer encoder optimized for CPU performance.
• lily: A dedicated inference server written in Rust and Metal that allows Qwen3.6-35B-A3B to run on Apple Silicon via an OpenAI-compatible API.

Who it’s for

ML engineers and researchers working on LLM inference optimization, distributed systems for trillion-parameter models, and on-device AI deployment on Apple hardware.

Highlights

• RDMA Integration: Enables efficient P2P communication for massive models on AWS EFA.
• Hardware Acceleration: Optimized Metal-based inference for Apple Silicon.
• CPU-Optimized Tokenization: Improved performance for Unigram tokenizer encoders.
• Disaggregated Architecture: Supports disaggregated prefill and decode processes.