xrdevrob/QuestCameraKit

What it solves

QuestCameraKit provides a set of Unity samples to help developers build mixed reality (MR) experiences on Meta Quest 3 and 3S that can interact with the physical environment using the device's cameras. It simplifies the implementation of computer vision tasks like object detection, QR code tracking, and real-world color sampling, as well as integrating multimodal AI for environmental awareness.

How it works

The project is a collection of Unity scenes that leverage the Meta Passthrough Camera API and other specialized tools. It implements several distinct workflows:

• Computer Vision: Uses the Unity Inference Engine to run YOLO models for real-time object detection and Meta MRUK for native QR code tracking.
• Multimodal AI: Integrates with OpenAI to send camera frames and voice recordings for visual questioning and text-to-speech responses.
• Visual Effects: Implements custom shaders for stereo camera mapping, frosted glass, and portal effects.
• Connectivity: Uses SimpleWebRTC and Unity WebRTC to stream camera video to external peers.
• Environment Interaction: Combines environment raycasting with camera pixel sampling to pick colors from the physical world.

Who it’s for

Unity developers building for Meta Quest 3/3S who want to integrate camera-based vision, AI, and streaming capabilities into their mixed reality applications.

Highlights

• Real-time Object Detection: Bundled YOLO model for projecting detection markers into 3D space.
• Multimodal AI Integration: Voice-to-text and image-to-text pipeline using OpenAI.
• Native QR Tracking: Spatial tracking of QR codes using Meta MRUK.
• Camera-based Shaders: Specialized materials for MR effects like frosted glass.
• WebRTC Streaming: Ability to stream the headset's camera feed to another device.