wesbos/3d-model-anything

Capture iPhone LiDAR and photos, reconstruct editable Three.js rooms, and explore them in a browser.

What it does

3D Model Anything is a workflow for turning a real room into an editable 3D model you can view in a web browser. You capture a room with an iPhone (recording both the room's geometry and photographs), then an AI coding agent uses that captured data to build a Three.js scene — a 3D model that can be modified and viewed from different angles.

The project is not a one-click converter. It's an iterative process: the iPhone app records the raw evidence, and a separate "skill" (a set of instructions for an AI coding agent) guides the agent through measuring, modeling, and comparing rendered views against the original photos to check accuracy. The current focus is on rooms and interiors.

What's in the repository

  • The iPhone capture app (source code only; an App Store release isn't included here and shouldn't be assumed to exist yet)
  • A web-based 3D viewer for exploring the reconstructed room (dollhouse, walking, overhead, wireframe, and knolling views)
  • The "scan-to-threejs" agent skill — instructions the AI agent follows to do the reconstruction
  • A demo office scene built from real captured data

How to use it briefly

  • View the demo: Requires Node.js 22.12 or newer. Run npm ci, then npm run dev, and open http://localhost:5188. The demo office includes only the reconstructed model and selected textures — original raw scans and photos aren't distributed, so the Scan and Overlay features are disabled for the demo but would work with your own captured data.
  • Capture your own space: You need an iPhone or iPad with LiDAR. Non-LiDAR devices can import and review scans but not capture. Export a room via the system share sheet, then extract it into a data/captures-in/<room>/ directory.
  • Run the AI agent: Open the repository in an agent that supports local skills, point it at the skill at .agents/skills/scan-to-threejs/SKILL.md, and ask it to rebuild a room from your extracted export. Creates an editable scene and compares rendered views with the original photos.

Requirements and extras

The skill's helper scripts need Python 3, plus Pillow for photo comparisons. Optional depth reconstruction also needs NumPy and Open3D. Capture scripts use the agent-browser CLI. Each new room goes in demos/<id>/ and is registered in demos/index.ts.

Privacy and licensing

Only the office demo is distributed — no other rooms, original scan bundles, or development history. Keep your own captures private, and check assets before publishing. The reusable code and skill are MIT licensed; the supplied office images and textures have separate asset terms, and third-party brands or products shown are not covered by the code license.

Related

  • Project
    Friedrich-M/UniMate

    UniMate is a diffusion‑based text‑to‑animation model that can generate motion for any rigged 3‑D skeleton (humans, animals, robots, etc.) without per‑skeleton fine‑tuning. It is trained on the UniML3D dataset (13.8 k clips, 7.4 k diverse skeletons) and provides full training, data‑processing, and inference pipelines, plus a tool to bring your own GLB/FBX/GLTF rigs into the system.

  • Project
    NVlabs/nvdiffrec

    A differentiable rendering framework that extracts triangular 3D models, materials, and lighting from multi-view images through joint optimization.

  • Project
    isl-org/Open3D-ML

    An extension of Open3D that provides tools, pretrained models, and pipelines for 3D machine learning tasks like semantic segmentation and object detection.

  • Project
    weihaox/awesome-neural-rendering

    A curated, topic-organized hub of neural-rendering papers and resources, maintained via pull requests, with clear per-topic docs files and a standard entry format.