omnimind-ai/OmniInfer

Unified, efficient, and easy-to-use inference infrastructure for edge AI || 面向端侧 AI 的统一、高效、易用的推理基础设施

OmniInfer – Quick Overview

What it is – OmniInfer is a cross‑platform inference engine that lets you run large language models (LLMs) and vision‑language models (VLMs) locally, on anything from a desktop PC to a phone. It hides the messy details of model compilation, hardware‑specific optimisations, and API wiring, offering a single command‑line tool (omniinfer) and an OpenAI‑/Anthropic‑compatible HTTP server.

Key capabilities

  • Multi‑backend support – Works with a growing list of runtimes (llama.cpp, stable‑diffusion.cpp, MNN, MLX, TurboQuant, LiteRT‑LM, ExecuTorch QNN, etc.). The tool can automatically pick the best backend for the current device.
  • LLM & VLM inference – Handles pure‑text models as well as vision‑language models such as Qwen‑VLA, with a terminal UI for chatting and a browser dashboard for vision tasks.
  • Cross‑platform – Pre‑built installers for Linux x64, macOS arm64, Windows x64, plus Android (Gradle) and iOS (Swift package) modules.
  • Local API compatibility – Exposes OpenAI‑style and Anthropic‑style endpoints (omniinfer serve) so existing client code can talk to a locally‑hosted model.
  • Hardware‑aware optimisation – Offers knobs for context length, GPU off‑loading, KV‑cache size, and backend‑specific launch flags.

Typical workflow

  1. Install the CLI with a one‑liner script (curl/PowerShell).
  2. Run omniinfer; the TUI suggests a compatible backend, downloads it if needed, and lets you pick a locally‑available GGUF model.
  3. Chat directly in the terminal, or start omniinfer serve to get an HTTP API for other apps.

Why it matters – Running LLMs/VLMs locally avoids cloud costs, latency, and data‑privacy concerns. OmniInfer’s “write‑once, run‑anywhere” approach makes it easier for developers, researchers, and hobbyists to experiment with state‑of‑the‑art models (e.g., Qwen‑3.8‑27B) on modest hardware.

Documentation & community – The repo includes detailed guides for installation, CLI usage, API reference, model loading, remote access, benchmarking, and mobile embedding. Contributions are welcomed via a standard CONTRIBUTING.md.

TL;DR

OmniInfer = a fast, multi‑backend, cross‑platform CLI + local API for running LLMs and VLMs on any device, with easy installation and hardware‑aware optimisation.

Related

  • Project
    sophgo/LLM-TPU

    LLM-TPU is an open‑source toolkit that compiles HuggingFace LLM/VLM checkpoints into SOPHGO TPU‑specific bmodels and provides ready‑to‑run Python/C++ demos, supporting quantization, KV‑cache, dynamic shapes, and multi‑chip parallelism.

  • Project
    llmmanorg/llmman

    llmman is a CLI/daemon that runs any LLM‑style agent (Claude Code, Codex, Copilot, etc.) on any model—local or hosted—by treating models as OCI container images. It supports multiple back‑ends (llama.cpp, vLLM, sglang, mlx‑lm), provides a unified Ollama/OpenAI/Anthropic API, enables registry‑to‑registry model transfer, signing with cosign, and aggregation of several machines into one endpoint.

  • Project
    flashinfer-ai/flashinfer

    FlashInfer is a GPU‑kernel library (with a Python package) that provides ultra‑fast, low‑precision attention, GEMM, MoE, and sampling operations for LLM inference. It automatically selects the best backend (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM) for a wide range of NVIDIA GPUs, from Turing up to Blackwell, and is already used by popular inference servers like vLLM and TensorRT‑LLM.

  • Project
    openvinotoolkit/model_server

    OpenVINO Model Server (OVMS) is a C++ inference server that serves LLMs, vision‑language models, and classic deep‑learning models via OpenAI‑compatible REST or KServe gRPC APIs. Optimized for Intel CPUs, GPUs and NPUs, it provides features like continuous batching, streaming, model versioning, hot‑reload, and Prometheus metrics. Deployable via Docker, bare‑metal binaries, or Kubernetes, it supports many model formats (TensorFlow, ONNX, OpenVINO IR, GGUF, etc.) and integrates with standard client libraries.

  • Project
    ztxz16/fastllm

    FastLLM is a C++‑based, PyTorch‑free inference engine for large language models. It supports dense and MoE models, hybrid GPU/CPU/NUMA/disk deployment, multi‑GPU tensor parallelism, speculative decoding (MTP/DSpark/DFlash2), many quantization formats, and offers a Python CLI, web UI, and OpenAI‑compatible API server. Install via `pip install ftllm`, run a model with `ftllm run`, or launch a service with `ftllm server`. Extensive docs cover model‑specific guides, performance tuning, and community support.