cactus-compute/cactus

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

What it solves

Cactus is a hybrid edge-cloud AI engine designed to run high-performance AI models on mobile devices and wearables. It addresses the challenges of running large models on hardware with limited resources by providing optimized kernels, efficient quantization, and a mechanism to automatically hand off difficult queries to the cloud when local confidence is low.

How it works

Cactus uses a layered architecture consisting of an Engine, a Graph, and Kernels. The Engine provides OpenAI-compatible APIs for text, speech, and vision. The Graph handles zero-copy computation, and the Kernels are optimized for specific hardware (Apple, Samsung, Pixel). It employs a custom rotation-based quantization technique (CQ) to reduce model size and memory usage while maintaining accuracy.

Who it’s for

Developers building AI-powered mobile and wearable applications who need low-latency, offline-first AI capabilities with an optional cloud fallback.

Highlights

  • Hybrid Edge-Cloud Execution: Automatically routes hard queries to the cloud based on local model confidence thresholds.
  • Multimodal Support: Integrated support for text LLMs, Vision Language Models (VLMs), and audio transcription (e.g., Whisper, Parakeet).
  • Custom Quantization: Rotation-and-codebook quantization (CQ) supporting bit-widths from 1-bit to 4-bit.
  • Broad Hardware Optimization: Optimized ARM NEON SIMD kernels for various mobile chipsets.
  • Extensive Bindings: Provides native bindings for Swift, Kotlin, Flutter, React Native, Python, and Rust.
  • Integrated Tooling: Includes a CLI for model conversion, benchmarking, and serving as an OpenAI-compatible local server.

Related

  • Dispatch
    Cactus Hybrid Gemma 4: On-Device LLMs with Confidence-Based Cloud Handoff

    Cactus Hybrid introduces a post-trained Gemma 4 E2B model that provides a confidence score for every answer, enabling efficient routing to larger cloud models when on-device confidence is low.

  • Project
    omnigent-ai/omnigent

    Omnigent is an open‑source meta‑harness that unifies many LLM coding agents (Claude Code, Codex, Cursor, etc.) under one CLI and web UI, letting you run, combine, and govern them across devices and cloud sandboxes, with built‑in collaboration and policy features.

  • Project
    chorus-codes/chorus

    A local orchestration tool that uses multiple AI subscriptions to provide parallel, multi-vendor code reviews to catch bugs and biases that a single AI model would miss.

  • Project
    ModelCloud/GPTQModel

    A comprehensive LLM quantization toolkit that compresses models using methods like GPTQ and AWQ with hardware acceleration for NVIDIA, AMD, Intel, and Huawei NPUs.

  • Project
    PINTO0309/PINTO_model_zoo

    A comprehensive collection of AI models pre-converted and quantized across various frameworks like TensorFlow, PyTorch, ONNX, and CoreML for easier deployment on edge devices.