reality-opened/openreality

OpenReality – Turning Phone Videos into AI‑Queryable 3D Worlds

What it does

• You upload a short video taken on a phone. The service (or a self‑hosted server) runs a SLAM‑style reconstruction pipeline that turns the footage into a persistent 3‑D scene containing geometry, detected objects and a report.
• The resulting scene can be queried by an LLM‑based AI assistant (Claude, Codex, Cursor, etc.) via the Model Context Protocol (MCP). The assistant can ask the system to measure distances, plan a navigation path, export robot‑training data, or simply answer “what’s in the room?”.

Key components

Component	Role	How you get it
core/	Low‑level reconstruction library (VGGT‑SLAM 2.0) that tracks the camera, builds dense geometry, calibrates metric scale, runs object detection/segmentation, and exports splats.	Source code in the repo (mirrored from a private repo).
server/	Backend that receives videos, runs the heavy GPU reconstruction job, stores the persistent scene, and exposes a plain REST API for measurement, planning, agents and data export.	Source code in the repo; can be run locally on a GPU box or on Modal.
mcp/	41 MCP “tools” that wrap the server’s endpoints, plus an offline simulator for development without a GPU. Published as the npm package openreality-mcp.	Install via npm i -g openreality-mcp or run with npx openreality-mcp serve.

How you use it

1. Add the tools to your AI assistant – e.g. claude mcp add openreality -- npx -y openreality-mcp serve (or the equivalent for Codex/Cursor). The assistant now knows commands like measure, plan path, export robot data.
2. Log in once (npx openreality-mcp login). A revocable API key is stored locally.
3. Upload a video – either through the hosted UI at open-reality.io or by asking the assistant to upload a local file.
4. Ask questions – “What objects are in the scan?”, “How far is the couch from the window?”, “Give me a path from the door to the desk”, etc.

Self‑hosting

• Clone the repo and run python -m server.selfhost --data-dir ~/openreality-data on a machine with a GPU, or deploy the Modal version (modal deploy modal_selfhost.py).
• The server fetches the 3‑D reconstruction model (VGGT‑1B) under a CC‑BY‑NC‑4.0 license; commercial use requires the hosted service or a separate commercial license.

Why it matters

• Bridges the gap between raw visual data and actionable AI agents: an LLM can now see a space, measure it, and plan actions for robots.
• Provides a single, open‑source stack (reconstruction, backend, MCP tooling) that can be run entirely on your own hardware, preserving privacy and avoiding vendor lock‑in.
• The MCP interface makes the capabilities discoverable to any AI assistant that supports the protocol, turning the 3‑D world into a set of typed, reliable tool calls rather than a vague text prompt.

License & caveats

• Code in this repo is BSD‑2‑Clause. The reconstruction model is CC‑BY‑NC‑4.0 (non‑commercial only). For commercial projects you must use the hosted service or obtain a separate license.
• mcp/ is the only part actively developed in this public repo; server/ and core/ are curated mirrors of private repositories, updated manually.

Bottom line OpenReality lets you convert everyday phone videos into structured 3‑D scenes that an AI assistant can query and act upon, with a self‑hostable stack and a clean tool‑based interface.