Epistates/pmetal

PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.

What it solves

PMetal is a comprehensive machine learning platform designed specifically for Apple Silicon. It eliminates the need to leave the Apple ecosystem for high-performance AI development by providing a unified suite of tools for low-level GPU/ANE optimization, model training, and deployment on macOS.

How it works

Written in Rust, PMetal leverages a modular architecture of 20 specialized crates to interface directly with Apple's Metal GPU and Neural Engine (ANE). It uses custom Metal shaders for critical operations like FlashAttention and fused kernels to maximize hardware efficiency. The platform provides multiple interfaces for different user needs:

  • Desktop GUI: A Tauri and Svelte application for visual model management and training.
  • Terminal TUI: A feature-rich control center for monitoring metrics and configuring runs.
  • CLI: A command-line tool for rapid execution of tasks like fine-tuning and quantization.
  • SDKs: Embeddable Rust and Python libraries for integrating these capabilities into other applications.
  • Multi-Mac Cluster: A distributed system that uses mDNS and Thunderbolt/Ethernet to link multiple Macs for shared training and inference.

Who it’s for

  • AI Researchers and Developers working on Apple Silicon who need a full-stack ML environment.
  • Rust and Python Developers wanting to integrate high-performance ML capabilities into their own apps.
  • Mac Power Users who want to fine-tune, merge, or quantize LLMs locally using a GUI or TUI.

Highlights

  • Deep Hardware Integration: Native support for M1 through M5 chips, including ANE integration and UltraFusion topology awareness.
  • Extensive Training Suite: Supports SFT, LoRA, QLoRA, DoRA, GRPO, and knowledge distillation.
  • Broad Model Support: Compatible with Llama, Qwen, DeepSeek, Mistral, Gemma, Phi, and more.
  • Advanced Model Ops: Built-in tools for model merging (14 strategies), GGUF quantization, and LoRA fusion.
  • Distributed Computing: Thunderbolt-aware clustering for multi-machine gradient all-reduce training.