OpenSQZ/MiniCPM-V-CookBook

Cook up amazing AI applications effortlessly with MiniCPM / MiniCPM-V / MiniCPM-o

What it solves

This project provides a comprehensive set of implementation guides and deployment recipes for the MiniCPM series of models (MiniCPM-V, MiniCPM-o, and MiniCPM LLM). It simplifies the process of integrating multimodal AI capabilities—including vision, speech, and text—into applications, ranging from high-performance enterprise servers to edge devices like iPhones and Android phones.

How it works

The cookbook serves as a centralized hub for ready-to-run examples and technical instructions. It organizes these "recipes" into four main categories:

  • Inference: Ready-to-run examples for vision tasks (single/multi-image QA, video QA, OCR, grounding), audio tasks (speech-to-text, text-to-speech, voice cloning), and text reasoning.
  • Fine-tuning: Guides for customizing models using frameworks like Transformers, LLaMA-Factory, SWIFT, and Align-anything.
  • Serving: Deployment instructions for various backends including vLLM, SGLang, llama.cpp, Ollama, and MLX, as well as web demos using FastAPI, Gradio, and OpenWebUI.
  • Quantization: Instructions for compressing models using GGUF, BNB, AWQ, GPTQ, and FP8/INT4 to reduce resource consumption.

Who it’s for

  • Individuals: Users looking for easy local inference via Ollama or llama.cpp.
  • Enterprises: Developers needing high-throughput, scalable GPU inference via vLLM or SGLang.
  • Researchers: AI researchers using advanced frameworks for model development and experimentation.
  • Edge Developers: Developers building multimodal AI apps for iOS, Android, and HarmonyOS.

Highlights

  • Omni-modal capabilities: Support for text, vision, and speech interaction.
  • Broad hardware support: Deployment paths for Apple Silicon, mobile devices, and high-end GPUs.
  • ** Endgame deployment**: Specific guides for on-device AI on iPhone and iPad.
  • Comprehensive quantization: Multiple methods to support various efficiency needs.

Related

  • Project
    Liquid4All/cookbook

    A comprehensive collection of examples, tutorials, and applications for deploying and fine-tuning Liquid AI's open-weight Liquid Foundation Models (LFMs) and LEAP SDK on mobile, desktop, and edge devices.

  • Project
    OpenBMB/MiniCPM-V-Apps

    OpenBMB’s MiniCPM‑V‑Apps repo provides three on‑device demo apps (iOS, Android, HarmonyOS) that run MiniCPM‑V multimodal LLMs locally via a fork of llama.cpp. It includes build scripts, model‑download instructions, and hardware requirements for models ranging from 1 B to a 35 B MoE that streams experts from SSD.

  • Project
    OpenBMB/MiniCPM-V

    Open‑source multimodal LLMs (MiniCPM‑V 4.7 and MiniCPM‑o 4.5) that run on‑device, support vision‑language (images, video, docs) and full‑duplex audio‑visual interaction, with MoE inference optimised for phones.

  • Project
    huggingface/huggingface-gemma-recipes

    A collection of minimal recipes and notebooks for implementing multimodal inference, fine-tuning, and RAG with the Gemma family of models.

  • Project
    UbiquitousLearning/mllm

    A fast, lightweight multimodal LLM inference engine for mobile and edge devices that enables local execution of VLMs and LLMs across various NPUs and CPUs.