Intent-Lab/VisionClaw

What it solves

VisionClaw transforms Meta Ray-Ban smart glasses into a real-time AI assistant. It allows users to interact with their environment through voice, enabling the AI to see what the user sees and perform actions on their behalf across various apps and services.

How it works

The system integrates the Meta Wearables DAT SDK with the Gemini Live API via a WebSocket connection. The app captures audio from the microphone and video frames (throttled to ~1fps) from the glasses' camera, streaming them to Gemini for real-time visual and auditory context.

To enable action-taking, the project optionally integrates with OpenClaw, a local gateway that provides Gemini with access to over 56 tools and connected apps for tasks like sending messages, managing shopping lists, and performing web searches.

Who it’s for

Developers and researchers interested in embodied AI, smart glasses integration, and real-time multimodal AI agents.

Highlights

• Real-time Multimodal Interaction: Uses Gemini Live for native audio and vision processing without needing separate speech-to-text steps.
• Agentic Capabilities: Integrates with OpenClaw to execute real-world tasks across messaging apps, smart home devices, and notes.
• Cross-Platform Support: Available for both iOS and Android.
• POV Streaming: Includes a WebRTC implementation to share the glasses' point-of-view live to a browser viewer.
• Phone Mode: Allows testing the entire pipeline using a smartphone camera instead of smart glasses.