LiteReality/LiteReality-Agent

LiteReality‑Agent – AI‑driven indoor 3D scene reconstruction

What it is

• An open‑source, end‑to‑end toolkit that turns a handheld RGB‑D scan of a room (produced by the LiteReality Scanner iOS app) into a fully‑articulated, graphics‑ready 3D scene.
• The heavy lifting is done by an agentic pipeline: a language model (Claude or Codex) orchestrates vision models (TRELLIS, GroundingDINO) and Blender to generate, place, and refine objects until the virtual room matches the real capture.

Why it matters

• Bridges the gap between consumer‑grade scanning (a phone app) and production‑grade assets usable in Blender, Unity, Unreal, or WebGL.
• Automates tedious manual steps such as object identification, pose estimation, and material baking, making high‑quality indoor reconstructions accessible to creators without deep 3D expertise.

---

Key components (as described in the README)

Component	Role
LiteReality Scanner (iOS)	Captures RGB frames, depth, and an Apple RoomPlan room.usdz file in a single walkthrough.
Agentic controller (Claude Code or OpenAI Codex)	Reads the scan, decides which objects to add/adjust, and issues commands to the downstream models.
TRELLIS	Generates 3D geometry for detected objects.
GroundingDINO	Provides 2‑D object detection and grounding to guide TRELLIS.
Blender 5.x	Performs the final scene assembly, material baking, and export to GLB/BLEND formats.
Modal (optional)	Hosts TRELLIS and GroundingDINO in the cloud, so a local GPU is not required.

---

How it works (high‑level flow)

1. Scan a room with the LiteReality app – you get a folder containing RGB images, depth maps, and room.usdz.
2. Upload the scan to the machine where you will run the toolkit.
3. Run uv run litereality run <scan>.
   • Scene init (deterministic): converts the raw capture into a seed room – a rough layout with basic geometry.
   • Authoring (agentic): the LLM‑driven agent iteratively compares the seed to the original scan, calls TRELLIS/GroundingDINO to create missing objects, refines poses, and improves materials. Flags --polish add a quality‑pass, --live streams the reconstruction live.
4. View/export the result with uv run litereality view …. The output includes:
   • room_preview/Room.glb – ready for game engines or web viewers.
   • room_preview/Room.blend – a full Blender project.
   • room/ – the complete asset hierarchy.

---

Getting started (from the README)

# 1. Install the Python environment (uv is the package manager)
uv sync --frozen --extra modal --group dev
cp .env.example .env   # then edit the .env file with your keys

# 2. Deploy the vision models (once per workspace)
uv run litereality setup

# 3. Verify the installation
SANITY_DEEP=1 uv run python sanity.py

# 4. Run a reconstruction (example scans are provided)
uv run litereality run example-scans/<scan>

System requirements

• macOS (Apple Silicon) or Linux with a ≥24 GB NVIDIA GPU.
• Blender 5.x installed; BLENDER_PATH must point to the directory.
• API keys for image generation (OpenAI or Gemini) and a Modal account (or a local GPU) for TRELLIS & GroundingDINO.

---

Typical use cases

• Game/VR asset creation – quickly generate interior environments from a phone scan.
• AR interior design – produce a manipulable 3D model that can be edited or re‑textured.
• Robotics simulation – feed realistic indoor maps into simulation platforms.
• Research – study agent‑driven 3D reconstruction pipelines without building the stack from scratch.

---

Limitations & current status

• A formal technical report is still “coming soon”; the repository provides a citation but no peer‑reviewed paper yet.
• Heavy components (TRELLIS, GroundingDINO) require either a paid Modal account or a high‑memory GPU; on‑device execution on typical laptops is not supported.
• The pipeline relies on external LLM APIs (Claude or Codex); usage costs and rate limits apply.
• Some steps are optional (--live, --polish) and may increase runtime considerably.
• Tests that need a GPU, a scan, or live rendering are excluded by default, so full end‑to‑end validation requires the appropriate hardware.

---

Quick reference cheat‑sheet

Command	What it does
uv run litereality run scans/<scan>	Full pipeline (init + authoring).
uv run litereality run scans/<scan> --through seed	Only the deterministic scene‑init stage.
uv run litereality stage author run/<scan> --force --polish --live	Re‑run authoring with polishing and live preview.
uv run litereality view run/<scan>	Open the generated GLB/BLEND in the default viewer.
uv run litereality setup	Deploy TRELLIS & GroundingDINO (Modal or local GPU).

---

Bottom line – LiteReality‑Agent is a genuine, open‑source AI‑augmented system that turns a consumer phone scan into a ready‑to‑use 3D indoor scene, leveraging LLM orchestration, vision models, and Blender. It is aimed at creators and researchers who need high‑quality indoor reconstructions without manually modeling every object.