Local LLM Setup on M4 Pro Mac Mini

Local LLM Infrastructure on Apple Silicon

Running a local LLM server on an M4 Pro Mac Mini with 48GB of unified memory allows for a private, cost-predictable AI backend that handles approximately 80% of daily tasks without relying on cloud APIs. This setup utilizes oMLX for inference, Tailscale for secure cross-device networking, and a mix of Mixture-of-Experts (MoE) and dense models to balance reasoning depth with system performance.

The Hardware and Software Stack

The core of the setup is an M4 Pro Mac Mini (48GB RAM), which serves as an always-on server. The software stack is designed for rapid deployment and cross-device accessibility:

  • Inference Server: oMLX, which provides an admin dashboard with a built-in HuggingFace browser and KV cache persistence to SSD, reducing re-computation time for agentic workflows.
  • Networking: Tailscale, creating a private mesh network (tailnet) that connects the Mac Mini to an iPhone and MacBook without exposing ports to the public internet.
  • Primary Models:
    • Qwen3.6-35B-A3B-OptiQ-4bit: Used for complex reasoning and depth. As an MoE model, it has 35B total parameters but only 3B active parameters per token.
    • Gemma-4-E4B-it-OptiQ-4bit: A lightweight model used for routine formatting and simple chat tasks.
  • Client Interfaces:
    • Hermes: An agent backend running on the Mac Mini, accessed via a desktop client on MacBook and Telegram on iOS.
    • Apollo (iOS): Used for quick, Claude-like throwaway queries via the oMLX endpoint.
    • Raycast AI: Integrated into macOS for miscellaneous tasks.
    • Pi: Utilized as a dedicated coding agent.

Memory Management for Local Models

Effective local LLM deployment depends on understanding the distinction between total parameters and active parameters, particularly in Mixture-of-Experts (MoE) architectures.

Dense vs. MoE Memory Footprint

In a dense model, every parameter is activated for every token, meaning a 27B model in 4-bit quantization requires roughly 14GB of RAM just for weights. In contrast, an MoE model like Qwen3.6-35B-A3B has 35B total parameters but only 3B active parameters per token. While all 35B parameters must reside in unified memory (taking ~20GB in 4-bit), the GPU/Media Memory footprint during inference is significantly lower, resembling a 6B dense model.

Hardware Compatibility Checklist

To determine if a model will fit on specific Apple Silicon hardware, the following calculation is recommended:

  1. Quantized File Size: A 4-bit model is roughly the number of parameters in gigabytes (e.g., 35B $\approx$ 17-20GB).
  2. OS Overhead: Subtract 6-8GB for macOS.
  3. Context Window: Allocate 8-16GB for the KV cache to support long conversations.
  4. Buffer: Maintain a 10-15% memory buffer to prevent the system from swapping to SSD, which severely degrades performance.

Strategic Advantages of Local Inference

Moving from cloud APIs to local compute addresses several operational and financial risks:

  • AI Sovereignty and Privacy: Local hardware eliminates the risk of third-party data exposure and protects against government-mandated model restrictions or sudden changes in API terms of service.
  • Cost Predictability: Local inference replaces variable per-token billing with a flat hardware purchase and electricity cost.
  • Performance: Local setups eliminate network roundtrips, providing lower latency for everyday tasks. The M4 Pro's media engine enables near-instant responses for most prompts.
  • No Rate Limits: Local compute removes the throttling common in paid API tiers.

Community Insights and Performance Benchmarks

While the M4 Pro is highly capable, community discussions highlight several nuances regarding performance and hardware limits.

Performance Data

Users reporting similar setups on M1 Max (32GB) using oMLX have noted the following token generation (TG) and prompt processing (PP) speeds:

  • Qwen3.6-35B-A3B-OptiQ-4bit: PP 342.6 tok/s, TG 44.4 tok/s.
  • Qwen3.6-35B-A3B-mxfp4: PP 389.6 tok/s, TG 47.6 tok/s.
  • Qwen3.8-27B-4bit: PP 66.3 tok/s, TG 11.8 tok/s.

Counterpoints and Limitations

Community members have raised several critical considerations for those attempting this setup:

"Running a large model locally comes down to one thing: how much RAM it actually needs in memory. Not completely true. It's memory AND memory bandwidth. You can have 1tb of memory but if you have awful memory-bandwidth you'll also have slow tok/s."

Other users noted that while local models are excellent for 80% of tasks, prefill latency can still be a bottleneck on Apple Silicon compared to dedicated H100/B300 clusters. Additionally, some questioned the privacy benefits of using Telegram as a frontend for local models, as Telegram bot accounts are not end-to-end encrypted.

Sources

Related