google/xrblocks

What it solves

XR Blocks is designed to accelerate the prototyping of immersive experiences that combine Extended Reality (XR) and Artificial Intelligence. It removes the complexity of building these applications by providing a lightweight, cross-platform library that works across XR devices (like Android XR) and desktop browsers.

How it works

Built on top of three.js, the library provides a high-level SDK that abstracts the hardware-specific complexities of WebXR. It integrates on-device machine learning models (via TensorFlow Lite/PyTorch) for gesture and head motion recognition, and connects to cloud-hosted Gemini models for multimodal understanding and conversational AI. It also includes a desktop simulator to allow developers to test experiences without needing a physical XR headset.

Who it’s for

Developers and researchers interested in creating AI-native XR applications, specifically those targeting Chrome and WebXR-supported devices.

Highlights

• AI-Native Integration: Seamless connection to Gemini for multimodal AI and live conversations.

• Advanced Interaction: Built-in support for hand tracking, custom gestures, and head gesture recognition (nodding/shaking).

• World Understanding: Tools for depth sensing, plane detection, scene meshes, and semantic segmentation.

• Agent Context: Provides an agent-facing scene context, including semantic trees and Set-of-Mark screenshots, to help AI agents understand the 3D environment.

• Unified Interaction: A single pipeline for handling mouse, gaze, hand rays, and direct touch.

• Spatial UI: A system for composing world-space cards and view-space overlays using flex-layout panels.