Yuliang-Liu/Monkey

Monkey (LMM): Image Resolution and Text Label Are Important Things for Large Multi-modal Models (CVPR 2024 Highlight)

What it solves

Monkey is a series of Large Multi-modal Models (LMMs) designed to improve how AI understands images, specifically by focusing on the importance of high image resolution and detailed text labels. It addresses the limitation of many multimodal models that struggle with high-resolution images and fine-grained visual details, such as text within images (OCR).

How it works

The project implements several variations of the model, including Monkey-Chat, TextMonkey, and Mini-Monkey. These models leverage existing architectures (like QwenVL, InternLM2, and miniCPM) and enhance them through image resolution enhancement and token compression methods. The project provides a full pipeline for data generation, training (via fine-tuning scripts), and inference, allowing the models to process high-resolution images and answer complex visual questions.

Who it’s for

Researchers and developers working on multimodal AI, specifically those interested in high-resolution image understanding, document parsing, and Visual Question Answering (VQA).

Highlights

  • High-Resolution Support: Specifically optimized for high-resolution images to improve accuracy in tasks like OCR and detailed visual analysis.
  • Model Zoo: Includes various versions such as Monkey-Chat for general chat, TextMonkey for OCR-free document understanding, and Mini-Monkey for multi-scale adaptive cropping.
  • Comprehensive Tooling: Provides training code, a data generation pipeline, and evaluation scripts for 14 different VQA datasets.
  • Competitive Performance: Ranked highly on OpenCompass in the Multimodal Model category.

Related

  • Project
    bilibili/Index-Translate

    A family of multilingual, multimodal translation models that support text, image, and audio inputs, featuring advanced instruction following for terminology, formatting, and syllable-controlled dubbing.

  • Project
    potamides/DeTikZify

    DeTi*k*Zify is an open‑source multimodal LLM that converts sketches or raster scientific figures into editable TikZ code. It supports image‑to‑TikZ generation, text‑to‑TikZ via Ti*k*Zero adapters, and iterative refinement with Monte‑Carlo Tree Search. Installable via pip, it requires a full TeX Live, Ghostscript, and Poppler installation, and runs on GPUs (8‑b models) or CPU (1‑b). The repository provides a CLI/web UI, Python API examples, model weights on Hugging Face, and scripts to recreate the training datasets.

  • Project
    jd-opensource/JoyAI-VL-Interaction

    An 8B-scale vision-language interaction model and system that enables AI to proactively respond to real-time video streams in under a second without waiting for user prompts.

  • Project
    jd-opensource/JoyAI-Echo

    JoyAI-Echo is a research-focused toolkit for generating long-form, coherent audio-visual content and interactive world models that maintain narrative and environmental continuity.