torinmb/mediapipe-touchdesigner

What it solves

This plugin allows TouchDesigner users to integrate GPU-accelerated computer vision tasks from Google's MediaPipe without needing to install complex local libraries or external dependencies. It provides a self-contained way to perform real-time tracking and segmentation in a visual programming environment.

How it works

The plugin runs a local web server and an embedded Chromium browser within TouchDesigner. It executes MediaPipe's JavaScript implementation via WebAssembly, which enables GPU acceleration. The detection data is then sent from the browser back to TouchDesigner via a local WebSocket server and decoded from JSON into usable data formats (CHOPs and TOPs).

Who it’s for

Creative coders, interactive installation artists, and developers using TouchDesigner who need real-time face, hand, pose, and object tracking for visual effects or interactive experiences.

Highlights

• Broad Model Support: Includes face detection, face landmark tracking, hand tracking, object detection, pose tracking, image segmentation, and image classification.
• Zero Installation: Runs as standalone .tox files with no external setup required on Mac or PC.
• GPU Acceleration: Leverages the browser's GPU capabilities for high-performance vision tasks.
• Flexible Input: Supports webcam feeds and external video sources via Spout (Windows) or Syphon/OBS (Mac).
• Performance Monitoring: Provides real-time metrics for detection time, draw time, and frame delay.