lightningpixel/modly

Modly – AI‑powered image‑to‑3D desktop app

What it is – Modly is a cross‑platform desktop application (Windows, Linux, Apple‑Silicon macOS) that lets you turn a single photograph into a 3D mesh. All processing runs locally on your GPU using open‑source AI models, so no cloud service is required.

How it works – The UI is built with Electron/Node.js and talks to a Python backend (FastAPI). The backend loads a chosen diffusion‑based 3D generation model (e.g., Hunyuan3D, TripoSG, Trellis2) and runs inference on the image you provide. The resulting mesh can be inspected, smoothed, decimated, and saved as a GLB/OBJ file.

Key features

• Local GPU inference – Supports NVIDIA, AMD (via ROCm) and Apple‑Silicon GPUs.
• Extension system – Models and processing steps are packaged as extensions (GitHub repos with a manifest.json). Official extensions for several state‑of‑the‑art 3D generators are listed, and users can add their own.
• Multi‑source model weights – An extension can pull parts of a model from multiple Hugging Face repos, with integrity checks before download.
• Shared weight groups & variants – Extensions can share common files across nodes and expose quantized variants (e.g., Q4_K_M, Q5_K_M) with size/VRAM hints.
• Workflow editor – Drag‑and‑drop graph where you connect nodes such as Image → Generate Mesh → Add to Scene.
• CLI agent – A pure‑Python command‑line tool (tools/modly-cli/agent.py) can drive the app programmatically (health check, list models, run a generation, query run status, etc.).
• Live RAM indicator – Shows current memory usage in the top bar.
• In‑app mesh post‑processing – Smoothing and decimation are built‑in.

Getting started

1. Clone or download the repo (or use the installer from the Releases page).
2. Install JavaScript dependencies: npm install.
3. Set up the Python backend:

   cd api
   python -m venv .venv
   source .venv/bin/activate   # or .venv\Scripts\activate on Windows
   pip install -r requirements.txt
   

4. Run the development version: npm run dev (or launch the packaged app with launch.bat / launch.sh).

Platform notes

• AMD GPUs – Detected automatically; ROCm‑compatible PyTorch wheels are used without a separate ROCm install.
• macOS – Only Apple‑Silicon is supported; the app uses native window controls.
• Packaging – macOS Apple‑Silicon builds can be created with npm run package:mac.

Extending Modly

• Create a GitHub repo containing a manifest.json and the required entry files (model node, process node, etc.).
• Install via the UI: Models → Install from GitHub → paste the repo URL.
• Extensions can declare:
  • model_sources for downloading weights from Hugging Face.
  • weight_groups to share files between multiple nodes.
  • weight_variants to expose quantized or precision variants, with optional size/VRAM metadata.

CLI usage examples

# Check the app is running
python tools/modly-cli/agent.py health

# List available models
python tools/modly-cli/agent.py model list

# Generate a mesh from an image
python tools/modly-cli/agent.py generate --image ./cat.png --output ./cat.glb

The CLI prints machine‑readable JSON with run IDs, status, and any error messages, making it suitable for automation scripts or agents.

Community & support – A Discord server is provided for announcements, bug reports, and user feedback.

License – MIT. Forks must retain attribution to the original Modly project as required by the license.

---

All details above are taken directly from the repository’s README.