t8/hypura

What it solves

Hypura allows users to run Large Language Models (LLMs) on Apple Silicon Macs that are too large to fit into the machine's physical memory (RAM/GPU). It prevents the system crashes and "swap-thrashing" that typically occur when trying to load massive models on consumer hardware with limited memory capacity.

How it works

Hypura acts as a storage-tier-aware inference scheduler. It analyzes the model architecture and hardware capabilities to distribute model tensors across three tiers: GPU (Metal), RAM, and NVMe storage.

• GPU (Metal): Stores critical, frequently accessed components like attention layers, norms, and embeddings.
• RAM: Holds overflow layers that cannot fit in the GPU working set.
• NVMe: Stores the remaining layers, which are streamed on-demand using direct I/O and prefetched ahead of the forward pass.

For Mixture-of-Experts (MoE) models, it uses "expert-streaming," which only loads the specific experts needed for a token, reducing I/O by 75%. For dense models, it uses "FFN-streaming," where Feed-Forward Network tensors are streamed through a dynamic pool buffer. It also includes a neuron cache to minimize redundant reads from the SSD.

Who it’s for

Developers and AI enthusiasts using Apple Silicon Macs who want to run high-parameter models (like Llama 70B or Mixtral 8x7B) that would otherwise exceed their available unified memory.

Highlights

• Prevents OOM Crashes: Enables the runnability of models that would crash vanilla llama.cpp.
• Zero Overhead: Models that already fit in memory run at full Metal GPU speed.
• Hardware-Aware: Automatically profiles GPU, RAM, and NVMe bandwidth to optimize tensor placement.
• Ollama Compatible: Includes an HTTP API that makes it a drop-in replacement for tools that integrate with Ollama.
• SSD Safe: Performs read-only operations during inference to avoid SSD wear.