mudler/locate-anything.cpp

What it solves

It provides a fast, lightweight, and dependency-free way to run NVIDIA's LocateAnything-3B model for open-vocabulary object detection. By porting the model to C++ and ggml, it eliminates the need for a Python runtime during inference and allows the model to run efficiently on CPUs and GPUs.

How it works

The project is a C++17 inference port of a Vision Language Model (VLM) consisting of a Qwen2.5-3B language model, a MoonViT vision tower, and a 2-layer MLP projector. It performs detection in "token space," where the model generates coordinate tokens (from 0 to 1000) that are then decoded into bounding boxes. It supports multiple decoding modes (hybrid, slow, and fast) and utilizes GGUF quantization to reduce model size and increase speed without sacrificing detection accuracy.

Who it’s for

Developers and researchers who need to perform object detection based on text prompts (open-vocabulary) in environments where Python is unavailable or where high-performance CPU/GPU inference is required.

Highlights

• High Performance: Significantly faster than the official PyTorch implementation on both CPU and GPU.
• Zero Python Dependency: Inference is handled entirely in C++, removing the Python runtime requirement.
• Quantization Support: Offers various GGUF quantization levels (e.g., q8_0, q4_k) to balance memory usage and speed.
• Validated Accuracy: Box detections are validated to be identical or near-identical to the official NVIDIA implementation.
• Flexible Deployment: Includes a CLI tool and a C-API for integration into other applications (such as LocalAI).