Maelic/RelateAnything

RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs

What it solves

RelateAnything provides a way to predict the relationships between objects in an image in real-time. Unlike traditional systems, it is "open-vocabulary," meaning you can define the specific relations you want to look for (e.g., "riding", "beside", "in front of") at inference time without needing to retrain the model.

How it works

The system uses a DINOv3 backbone to process the image and turn object regions into visual tokens. A pair sampler identifies candidate relations, and a transformer refines these using the scene context. A vocabulary head then scores these pairs against text embeddings of the user-provided predicate strings. This allows the model to identify both spatial relations (where things are) and semantic relations (what they are doing) from a single forward pass.

Who it’s for

This tool is designed for developers and researchers working on scene graph generation, computer vision, and real-time object relationship analysis.

Highlights

  • Open-Vocabulary: Define custom relation phrases at runtime without retraining.
  • Real-Time Performance: High-speed inference (e.g., 16.8ms per image with ViT-S+ on an RTX 3080).
  • Flexible Inputs: Works with any object detector or segmenter; it only requires image pixels and bounding boxes.
  • Multiple Graph Types: Can produce a single ranked list of relations or separate spatial and semantic graphs.
  • Browser-Ready: Includes a demo that runs locally on the device using WebGPU or WASM.

Related

  • Project
    mudler/locate-anything.cpp

    A C++/ggml inference port of NVIDIA's LocateAnything-3B that enables fast, open-vocabulary object detection on CPU and GPU without a Python runtime.

  • Project
    zju3dv/MatchAnything

    MatchAnything is a research project that trains a single neural network to find correspondences between images from any visual modality (e.g., RGB, infrared, sketches). Pre‑trained weights are hosted on Hugging Face, and an online demo lets users try cross‑modality matching instantly. The approach aims to replace many specialized matching pipelines with one universal model, useful for robotics, remote sensing, medical imaging, and AR.

  • Project
    mkturkcan/DART

    A training-free framework that converts SAM3 into a real-time multi-class open-vocabulary detector using TensorRT optimization and distilled backbones.

  • Project
    om-ai-lab/VLX-Seek

    VLX-Seek is a fine-grained perception VLM that replaces coordinate generation with region-token retrieval for more stable and efficient object localization in embodied AI.

  • Project
    facebookresearch/dinov2

    A self-supervised learning framework that produces robust visual features from 142M images, enabling high-performance computer vision tasks without fine-tuning.