omnimind-ai/OmniInfer

Unified, efficient, and easy-to-use inference infrastructure for edge AI || 面向端侧 AI 的统一、高效、易用的推理基础设施

What it solves

OmniInfer provides a high-performance, cross-platform inference engine for running Large Language Models (LLM) and Vision-Language Models (VLM) locally. It removes the complexity of model compilation, hardware adaptation, and deployment, allowing users to run AI models on various devices with minimal configuration.

How it works

OmniInfer acts as an abstraction layer over multiple backend engines (such as llama.cpp, MLX, MNN, and ExecuTorch QNN). It allows users to switch between these backends seamlessly to find the best performance for their specific hardware. It can be used via a Terminal UI for local chat, a CLI for serving OpenAI- and Anthropic-compatible API endpoints, or embedded as a module in Android and iOS applications.

Who it’s for

Developers and users who want to run LLMs and VLMs locally and privately on desktop (Linux, macOS, Windows) or mobile (Android, iOS) devices without dealing with the manual setup of various inference runtimes.

Highlights

  • Cross-platform support: Runs on Linux, macOS, Windows, Android, and iOS.
  • Multi-backend flexibility: Supports a variety of engines including llama.cpp, MLX, MNN, and TurboQuant.
  • API Compatibility: Provides local endpoints that are compatible with OpenAI and Anthropic APIs.
  • Hardware-aware optimization: Optimized for token generation speed and minimal memory footprint across different accelerators.
  • VLM and VLA support: Handles vision-language models and vision-language-action (VLA) policies for robotics simulators like LIBERO.

Related

  • Project
    AlexsJones/llmserve

    A TUI-based model manager and server launcher that auto-detects inference backends and local model files to quickly spin up LLM servers.

  • Project
    orailnoor/cross-platform-llm-client

    A cross-platform AI chat client that unifies local on-device LLM inference via llama.cpp with cloud API access for OpenAI, Anthropic, and Google Gemini.

  • Project
    mudler/LocalAI

    An open-source AI engine that lets you run LLMs, vision, audio, and image models locally on any hardware with drop-in OpenAI-compatible APIs.

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without a server.

  • Project
    Ma-Zhuang/OmniNWM

    OmniNWM is a research codebase that builds a panoramic, multi‑modal world model for autonomous‑driving simulation. It jointly generates RGB, semantic, depth, and 3‑D occupancy maps from a vehicle trajectory, uses normalized Plücker ray‑maps for precise control, and provides occupancy‑based dense rewards for closed‑loop policy evaluation. The repo includes installation steps (including a manual patch to `transformers`), data preparation instructions for nuScenes, pretrained checkpoint download links, and scripts for inference, out‑of‑distribution testing, and staged training.