cactus-compute/cactus

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

What it solves

Cactus is a hybrid edge-cloud AI engine designed specifically for mobile devices and wearables. It enables high-performance local inference for text, speech, and vision models while providing a mechanism to automatically hand off complex queries to the cloud when local model confidence is low.

How it works

The system is built as a layered stack:

  • Cactus Engine: Provides OpenAI-compatible APIs for multimodal AI tasks and manages the hybrid cloud handoff logic.
  • Cactus Graph: A zero-copy computation graph for efficient tensor operations.
  • Cactus Kernels: Optimized CPU/GPU kernels (ARM NEON SIMD) tailored for mobile hardware like Apple, Samsung, and Pixel devices.
  • Cactus Quants: A custom rotation-based quantization technique that supports weight tensors from 4-bit down to 1-bit.
  • Cactus Transpiler: A Python-based tool that converts PyTorch models into the Cactus runtime graph format.

Who it’s for

Developers building AI-powered mobile and wearable applications who need low-latency, offline-capable inference with the option of cloud fallback for higher accuracy.

Highlights

  • Hybrid Execution: Automatically routes hard queries to the cloud based on local model confidence thresholds.
  • Multimodal Support: Local execution of LLMs, VLMs (Vision Language Models), and transcription models (e.g., Whisper, Parakeet).
  • Broad Hardware Optimization: Specialized kernels for mobile CPUs and GPUs across various Android and iOS devices.
  • Extreme Quantization: Supports mixed-precision and uniform quantization from 4-bit to 1-bit.
  • Cross-Platform Bindings: Official bindings for Swift, Kotlin, Flutter, React Native, Python, and Rust.