lyuwenyu/RT-DETR

🎯 What is RT‑DETR?

RT‑DETR (Real‑Time Detection Transformer) is an open‑source implementation of a family of object‑detection models that combine the DETR (Detection Transformer) architecture with a focus on real‑time speed. The authors show that, with the right design choices, DETR‑style detectors can out‑perform classic YOLO‑type models while still running at hundreds of frames‑per‑second on modern GPUs.

---

🚀 Main Highlights

Feature	Details
Two codebases	Implementations in PyTorch and PaddlePaddle for both the original RT‑DETR and the newer RT‑DETRv2.
Model zoo	Multiple backbone sizes (R18, R34, R50‑m, R50, R101, HGNetv2‑L/X) with pretrained weights on COCO and Objects365.
Speed / Accuracy	Example: RT‑DETR‑R18 reaches 46.5 AP (63.8 AP@50) on COCO at 217 FPS (TensorRT FP16 on an NVIDIA T4). Larger models trade a bit of speed for higher AP (up to 56.2 AP).
Real‑time tricks	"Bag‑of‑Freebies" such as better data augmentation, sliced inference for small objects, and support for ONNX/TensorRT/OpenVINO deployment.
Extensible	Torch‑hub entry, Hugging‑Face Transformers integration, and a profiling script (run_profile.py) for parameter/FLOP analysis.
Vision Foundation Model (VFM) boost	The newest RT‑DETRv4 (linked from the repo) leverages VFMs to improve performance without extra inference latency.
License	Apache 2.0 (see badge in README).

---

📦 Getting Started (PyTorch example)

# clone the repo
git clone https://github.com/lyuwenyu/RT-DETR.git && cd RT-DETR

# install dependencies (example)
pip install -r requirements.txt

# download a pretrained checkpoint (e.g., RT‑DETR‑R18)
wget https://path/to/rt_detr_r18.pth -O rtdetr_pytorch/weights/r18.pth

# run inference on an image
python rtdetr_pytorch/tools/infer.py \
    --config rtdetr_pytorch/configs/rt_detr_r18.py \
    --weights rtdetr_pytorch/weights/r18.pth \
    --image path/to/image.jpg

The repository also provides:

• run_profile.py – quick stats on model parameters and FLOPs.
• hubconf.py – load a model with torch.hub.load('lyuwenyu/RT-DETR', 'rt_detr_r18').
• Deployment scripts for ONNX, TensorRT and OpenVINO.

---

📊 Performance Snapshot

Model	Input	Dataset	AP (val)	AP@50 (val)	Params (M)	FLOPs (G)	TensorRT FP16 FPS (T4)
RT‑DETR‑R18	640×640	COCO	46.5	63.8	20	60	217
RT‑DETR‑R34	640×640	COCO	48.9	66.8	31	92	161
RT‑DETR‑R50‑m	640×640	COCO	51.3	69.6	36	100	145
RT‑DETR‑R101	640×640	COCO	54.3	72.7	76	259	74
RT‑DETRv2‑S	640×640	COCO	48.1 (+1.6)	65.1	20	60	217
RT‑DETRv2‑L	640×640	COCO	53.4 (+0.3)	71.6	42	136	108

Finetuned models on COCO + Objects365 achieve even higher AP (up to 56.2) with the same speed.

---

🛠️ Where It Fits

• Real‑time edge applications (e.g., robotics, AR/VR, video analytics) where latency matters more than squeezing the last fraction of AP.
• Research on transformer‑based detection, providing a solid baseline and a set of “freebie” tricks to compare against.
• Deployment pipelines thanks to ready‑made ONNX/TensorRT/OpenVINO export scripts and Torch‑Hub support.

---

📚 Citation

If you use RT‑DETR or RT‑DETRv2 in a publication, please cite the corresponding arXiv papers (provided in the README).

---

TL;DR: RT‑DETR is a well‑maintained, high‑speed object‑detection transformer library with multiple backbones, pretrained weights, and deployment tooling, making it a practical choice for anyone needing fast, accurate detection on GPUs or edge devices.