Tau-J/rtmlib

RTMPose series (RTMPose, DWPose, RTMO, RTMW) without mmcv, mmpose, mmdet etc.

rtmlib – lightweight pose‑estimation library

What it is – rtmlib is a tiny Python package that lets you run human‑ and animal‑pose estimation with the popular RTMPose and ViTPose models without pulling in the heavy OpenMMLab stack (mmcv, mmpose, mmdet, …). It only depends on numpy, opencv‑python, opencv‑contrib‑python and onnxruntime, and can optionally use OpenVINO, TensorRT or the GPU‑enabled ONNX Runtime for speed.

Why it matters – Pose estimation is a core computer‑vision task used in AR/VR, sports analytics, robotics, animation and many AI‑powered apps. By stripping away the large OpenMMLab dependencies, rtmlib makes it easy to embed pose inference in lightweight environments (e.g., a simple script, a web UI, or an edge device) while still giving you access to state‑of‑the‑art models.

Quick start (single image)

import cv2
from rtmlib import Wholebody, draw_skeleton

img = cv2.imread('demo.jpg')
whole = Wholebody(
    to_openpose=False,          # False = mmpose style skeleton
    mode='balanced',            # 'performance' | 'lightweight' | 'balanced'
    backend='onnxruntime',
    device='cpu'
)
keypoints, scores = whole(img)
img = draw_skeleton(img, keypoints, scores, kpt_thr=0.5, to_openpose=False)
cv2.imshow('pose', img)
cv2.waitKey(0)

The same API works on a webcam or video stream via the PoseTracker class.

Main building blocks

Layer Class (high‑level) What it does
Detector YOLOX, RFDETR, RTMDet Detect humans (or any COCO class) in an image.
Pose estimator RTMPose, ViTPose Predict key‑point coordinates for body, hand, whole‑body, or 3‑D pose.
Solution Wholebody, Body, Hand, Animal, Custom, Wholebody3d Combine a detector + a pose estimator into a single callable that returns keypoints and confidence scores.
Utility draw_skeleton, draw_bbox Visualise the results on the original image.

All of these can be instantiated with a mode shortcut (performance, lightweight, balanced) or by passing explicit ONNX model URLs/paths for the detector and pose estimator.

Model zoo (download handled automatically)

Detectors – YOLOX variants (nano, tiny, s, m, l, x) trained on HumanArt+COCO (detect real humans and cartoon characters) or plain COCO.

Pose estimators – dozens of ONNX checkpoints:

  • RTMPose (t, s, m, l, x) for 17‑keypoint body, 21‑keypoint hand, 26‑keypoint HALPE, 133‑keypoint whole‑body, and 3‑D variants.
  • RTMO (one‑stage) versions of the same scales.
  • ViTPose++ (s, b, l) for 17‑keypoint and 133‑keypoint bodies, plus 25‑keypoint fine‑tuned models.

If the original OpenMMLab download server is unreachable, the library falls back to a mirror on HuggingFace (huggingface.co/Tau-J/RTMPose).

How to install

# from PyPI (recommended)
pip install rtmlib -i https://pypi.org/simple

# or from source
git clone https://github.com/Tau-J/rtmlib.git && cd rtmlib
pip install -r requirements.txt
pip install -e .   # editable install
# optional GPU back‑ends
pip install onnxruntime-gpu openvino

Web UI

A minimal Gradio interface (webui.py) lets you upload an image or stream from a webcam and see the pose overlay instantly:

pip install gradio   # if not already present
python webui.py

The UI demonstrates the same Wholebody/draw_skeleton pipeline under the hood.

Typical use‑cases

  • Rapid prototyping of pose‑based features in research notebooks or small apps.
  • Edge deployment where installing the full OpenMMLab stack is impractical.
  • Animal pose estimation (bird, cat, dog, horse, etc.) using the Animal solution with OpenPose‑style skeletons.
  • Custom pipelines – plug any ONNX detector or pose model you like via the low‑level YOLOX, RTMPose, ViTPose classes.

License & citation

The repository follows the same licensing as the underlying OpenMMLab models (check the LICENSE file). If you use rtmlib in a publication, cite the original RTMPose / ViTPose papers and the rtmlib GitHub URL as indicated in the Citation section of the README.

Bottom line – rtmlib gives you a clean, dependency‑light wrapper around cutting‑edge pose‑estimation models, with ready‑made high‑level APIs, a small model zoo, and optional GPU/accelerator back‑ends for fast inference.

Related

  • Project
    ultralytics/yolov5

    YOLOv5 is a PyTorch‑based, open‑source object detection/segmentation/classification model suite from Ultralytics, offering multiple model sizes, simple Python/CLI inference, one‑line training, and export to many deployment formats.

  • Project
    Tencent/YOLO-Master

    A YOLO-style framework for real-time object detection that uses Mixture-of-Experts (MoE) to adaptively allocate computation based on scene complexity for higher precision and lower latency.

  • Project
    PRBonn/rko_lio

    RKO‑LIO is a sensor‑agnostic LiDAR‑Inertial odometry library that fuses LiDAR scans and IMU data to estimate robot pose. It works on ROS bags (ROS 1/2) via a simple Python CLI (`pip install rko_lio`) or as a ROS 2 package (`apt install ros-<distro>-rko-lio`). Only the IMU‑to‑base and LiDAR‑to‑base extrinsics are required; no per‑sensor modeling is needed. The system includes optional real‑time visualization with the Rerun SDK and is backed by a peer‑reviewed IEEE RA‑L paper.

  • Project
    lyuwenyu/RT-DETR

    RT‑DETR is an open‑source, real‑time object‑detection transformer library (PyTorch & Paddle) with multiple backbone sizes, pretrained weights, and deployment tools (ONNX/TensorRT/OpenVINO). It achieves YOLO‑level speed (up to 217 FPS on an NVIDIA T4) while delivering competitive AP (46‑56 on COCO). The repo includes profiling scripts, Torch‑hub integration, and recent extensions like RT‑DETRv4 that leverage vision foundation models.

  • Project
    AIFrontierLab/TorchUMM

    TorchUMM is a unified, config‑driven toolkit for running, evaluating, and fine‑tuning a wide range of multimodal models (text‑to‑image, image understanding, editing). It supplies adapters for 14 back‑bones, supports 10+ benchmarks, offers post‑training methods, and can run locally, on AMD HPC clusters, or via Modal cloud. The library standardises inference/evaluation, making fair, reproducible comparisons easy.