AlayaLab/WorldSculpt

🌍 What is WorldSculpt?

WorldSculpt is a research‑grade system that turns ordinary RGB video (plus instance masks and rough 3D boxes) into a single, compositional mesh of a whole scene – even when the scene contains hundreds of objects. It does this by re‑using a single‑object generative prior (the Pixal3D model) and fine‑tuning it to handle occluded, multi‑view inputs. The result is a dense, editable 3‑D mesh that preserves object‑level geometry.

---

🚀 Key Features

Feature	What it means
Single‑object prior for complex scenes	Leverages a model trained on individual objects to reconstruct whole rooms or outdoor environments with many items.
Finetuned Pixal3D	Adapts the Pixal3D generative model to accept occluded multi‑view video frames.
UE‑MeshyScene benchmark	A new dataset (UE‑MeshyScene) with per‑frame, per‑instance annotations for evaluating compositional meshification.
Marble demo	Shows the pipeline can also convert a 3‑DGS “Marble” world into object‑level meshes.
ComfyUI integration	Community‑built node‑based UI for easy drag‑and‑drop use inside ComfyUI.
Open‑source checkpoints & data	Model weights and benchmark data are hosted on Hugging Face for immediate download.

---

📦 Getting Started (High‑level)

1. Base environment – Install the Microsoft TRELLIS.2 stack first (the repo points to its installation guide).
2. Additional deps – Install natten (CUDA‑aware), utils3d, and a list of Python packages (torch, transformers, diffusers, etc.).
3. Download models – Pull the WorldSculpt checkpoints and the Pixal3D prior from Hugging Face.
4. Download data – Grab the UE‑MeshyScene or Marble demo datasets from the same hub.
5. Run inference – Use the provided inference.sh script, specifying the input scene folder, output folder, checkpoint location, and a few hyper‑parameters (e.g., SS_STEP, SHAPE_STEP, FACE_BUDGET).

---

🛠️ Example Command

INPUT_ROOT=./input/UE-MeshyScene \
OUTPUT_ROOT=./output \
CKPT_ROOT=./pretrained \
SAMPLER=official SS_STEP=15000 SHAPE_STEP=15000 \
RENDER=1 FACE_BUDGET=1000000 GPU=0 \
./inference.sh scene_00001

The script reads transforms.json inside the scene folder, runs the model, and writes a mesh (and optional renders) to ./output.

---

🎨 Using with ComfyUI

A community member, Terry Jia, created a ComfyUI‑WorldSculpt plugin. It wraps the same inference pipeline into ComfyUI nodes, letting you build visual workflows without writing shell commands. See the linked GitHub repo for installation and example node graphs.

---

📚 Resources

• Project page – https://alaya-lab.github.io/WorldSculpt (demo video, paper link, visual results)
• ArXiv paper – https://arxiv.org/abs/2609.05416 (full technical details)
• Model hub – https://huggingface.co/AlayaLab/WorldSculpt (checkpoints)
• Dataset hub – https://huggingface.co/datasets/AlayaLab/Worldsculpt_data (UE‑MeshyScene, Marble)
• ComfyUI integration – https://github.com/jtydhr88/ComfyUI-WorldSculpt

---

📖 Citation

If you use WorldSculpt in research, cite the arXiv pre‑print (provided in the README).