oculus-samples/Unity-PassthroughCameraApiSamples

What it solves

This project provides Unity developers with a way to access raw camera data from Meta Quest 3 and 3S headsets. It solves the problem of aligning digital content with the real world by providing precise timestamps, camera metadata (intrinsics and extrinsics), and simultaneous access to both left and right camera feeds.

How it works

It utilizes the PassthroughCameraAccess component from the Mixed Reality Utility Kit (MRUK). This component allows developers to integrate camera feeds into their Unity scenes, which can then be used for custom GPU effects, environment analysis, and machine learning models.

Who it’s for

Unity developers building mixed reality (MR) experiences for Quest 3/3S devices who need direct access to the headset's camera textures and pose information for advanced spatial awareness or computer vision tasks.

Highlights

• Multi-Object Detection: Uses the Unity Inference Engine (Sentis) to perform real-world object recognition.
• Camera-to-World Alignment: Transforms 2D camera coordinates into 3D world space rays for precise spatial mapping.
• Environment Adaptation: Includes a brightness estimation sample to adapt the experience based on real-world lighting.
• Custom GPU Effects: Provides a shader sample for applying real-time effects to the camera texture.