LiteReality/LiteReality-Agent

LiteReality‑Agent – Turn a phone‑captured room scan into a physics‑ready 3D simulation

What it is – An open‑source, end‑to‑end toolkit that takes RGB‑D scans produced by the LiteReality iOS scanning app and automatically builds a fully articulated, simulation‑ready indoor scene. The pipeline runs a series of AI‑driven modules (layout cleaning, object articulation, material generation, physics parameter inference) and finally exports the result as MuJoCo‑compatible URDF/MJCF files, Blender scenes, or GLB assets.

Why it matters – Robotics researchers and developers often spend hours manually cleaning scans, adding colliders, and assigning masses before a scene can be used for simulation or reinforcement‑learning experiments. LiteReality‑Agent automates that whole workflow, letting you go from a 30‑second phone sweep to a MuJoCo scene with realistic mass, inertia, joints and collision geometry in a single command.

Key capabilities (as described in the README)

Capability	What the repo does
Scan ingestion	Accepts the `room.usdz` package produced by the LiteReality Scanner (RGB frames, depth, geometry).
Layout cleaning	A layout agent resolves noisy raw scans into a collision‑free floor plan before any objects are built.
Articulated asset generation	Uses the Articraft approach to infer object parts, joints, masses, inertias and colliders; each object becomes a separate rigid body.
Material & visual refinement	Calls LLM‑backed agents (`codex` with GPT‑6, `claude`) and an image‑generation API (OpenAI or Gemini) to synthesize reference textures and improve visual fidelity.
Physics export	Produces URDF and MJCF files that can be loaded directly into MuJoCo; also outputs Blender (`.blend`) and GLB files for Unity/Unreal/web viewers.
Live preview	With `--live` the CLI streams a web viewer that shows the scene being built step‑by‑step.
Shake test	A built‑in MuJoCo “shake” stage measures how objects move, confirming that physics parameters are sensible.

Typical workflow

Capture – Use the free LiteReality Scanner iOS app to walk through a room (produces room.usdz).
Upload – Copy the scan folder to the machine where you will run the toolkit.
Setup – Install the Python environment with uv, point BLENDER_PATH to Blender 5.x, and provide API keys (OPENAI_API_KEY or GEMINI_API_KEY) plus a Modal token for the hosted GPU models.
Run – One command does everything:
```
uv run litereality run scans/<your‑scan>
```
Stage 1 (scene init) creates a deterministic seed room; Stage 2 (authoring) runs the agentic passes that refine geometry, add articulation, and polish materials.
Simulate – Export to MuJoCo and optionally run a shake test:
```
uv run litereality stage simulate run/<scan> --shake
```
Inspect – View the final GLB/Blender files locally or in the interactive web viewer.

Installation highlights

Package manager: uv (fast, deterministic lockfile). Install with uv sync --frozen --extra modal --group dev.
Dependencies: Blender 5.x, a GPU‑capable model backend (TRELLIS + GroundingDINO). By default these heavy models run on Modal (free tier), so a Mac without a GPU can still use the toolkit.
Environment variables: API keys for image generation, Modal credentials, BLENDER_PATH, and the directory containing scans.
One‑time model deployment: uv run litereality setup provisions the remote containers.

Who should use it

Robotics researchers needing realistic indoor environments for sim‑to‑real transfer, manipulation planning, or reinforcement learning.
Game/VR developers who want quick, physics‑accurate prototypes of real rooms.
Computer‑vision labs exploring 3‑D reconstruction, layout reasoning, or articulated object detection.
Educators demonstrating end‑to‑end pipelines from sensor data to simulation.

Current status (as of the README)

Version 0.0 released 2026‑08‑01.
Simulation‑ready (MuJoCo export) added 2026‑09‑12.
Procedural physics (mass, inertia, joints) integrated 2026‑09‑09 via Articraft.
Layout agent introduced 2026‑09‑07.
Paper published on arXiv (2610.01863) 2026‑10‑01.

Citation

If you use the system in research, cite the accompanying arXiv paper:

@misc{huang2026literealityagentagenticinteractable3d,
  title = {LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstruction},
  author = {Zhening Huang and Yueyan Li and Johnathan Chiu and Xiaoyang Lyu and Matt Zhou and Yuxin Yao and Joan Lasenby and Shangzhe Wu},
  year = {2026},
  eprint = {2610.01863},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2610.01863}
}

Bottom line – LiteReality‑Agent is a genuine, research‑grade open‑source system that bridges mobile RGB‑D scanning and physics‑based simulation, automating many steps that traditionally required manual 3‑D modeling and rigging.