mkturkcan/DART

What it solves

DART converts the Segment Anything Model 3 (SAM3) into a real-time, multi-class, open-vocabulary detector. It addresses the latency issues of large vision models by enabling high-speed object detection (boxes and scores) without requiring retraining of the base model.

How it works

DART uses a training-free framework that splits the model into two separate TensorRT FP16 engines: a backbone (e.g., ViT-H/14) and an encoder-decoder. The text encoder remains in PyTorch and caches embeddings to the GPU, allowing users to change target classes in milliseconds. To further increase speed, the project provides distilled student backbones (like RepViT or TinyViT) and supports block pruning (removing entire blocks or masking sub-blocks) to reduce computational load.

Who it’s for

Computer vision developers and engineers who need real-time open-vocabulary detection on GPUs, particularly those targeting NVIDIA hardware (RTX 4080, Jetson AGX Orin) using TensorRT for optimization.

Highlights

• Real-time Performance: Achieves up to 15.8 FPS on a single RTX 4080 for 4 classes at 1008px resolution.
• Open-Vocabulary: Detects any class specified by the user via text prompts without retraining.
• Flexible Backbones: Supports full ViT-H, pruned versions, and distilled student models (RepViT, TinyViT, EfficientViT) for various quality-speed tradeoffs.
• Optimization Suite: Includes tools for block pruning analysis, self-distillation for quality recovery, and TensorRT export scripts.
• Video Support: Features inter-frame pipelining to reduce latency and integrated ByteTrack for multi-object tracking.