fabiotosi92/ZipDepth

📦 What is ZipDepth?

ZipDepth is a lightweight, zero‑shot monocular depth‑estimation model introduced at ECCV 2026. It predicts dense depth maps from a single RGB image without any task‑specific fine‑tuning. The key selling point is its accuracy‑efficiency trade‑off: it reaches the performance of much larger transformer‑based foundation models while keeping the model size to ~6 M parameters and running at >1 k FPS on modern GPUs (and comfortably on mobile/edge hardware).

---

🔑 Core ideas

Component	What it does
Encoder	Four‑stage hierarchy built from RepVGG re‑parameterizable convolutions. Early stages use Strip Pooling Attention for horizontal/vertical context; later stages add Squeeze‑and‑Excitation and a Global Context Block.
Neck	Multi‑scale pooling (SPPF) plus a Cross‑Scale Fusion module to blend features before decoding.
Decoder	Compact Feature‑Pyramid‑Network (FPN) with a Convex Upsampling head that yields sub‑pixel‑accurate depth maps.
Two checkpoints	zipdepth_base.pth (GPU‑friendly, uses torch.nn.Unfold) and zipdepth_base_npu.pth (unfold‑free, easier to convert to ONNX/CoreML/TFLite for mobile/NPUs).

---

🚀 Getting started

# Clone and install
git clone https://github.com/fabiotosi92/ZipDepth
cd ZipDepth
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt   # PyTorch ≥2.4 is required
pip install -e .

Inference (single image, folder, or video)

python scripts/infer.py \
  --checkpoint checkpoints/zipdepth_base.pth \
  --input assets/examples/im0.jpg \
  --input-size 384          # short side, rounded to multiple of 32

Optional flags: --fp16 (half‑precision), --compile (torch.compile), --npu (use NPU‑compatible checkpoint), --no-colormap / --save-raw for raw .npy depth output.

Exporting the model

• ONNX (recommended for mobile/edge):

  python scripts/export.py \
    --ckpt checkpoints/zipdepth_base_npu.pth \
    --format onnx --height 384 --width 384 --npu
  

• TorchScript (traced or frozen) for pure PyTorch deployment.

---

📊 Performance snapshot

Dataset	Params	GFLOPs (384×384)	Typical latency (RTX 3090)
NYUv2, KITTI, ETH3D, ScanNet, DIODE	~6.1 M	~0.9 G	0.8 ms (TensorRT FP16) → ≈1300 FPS

The paper’s Pareto plot shows ZipDepth dominating the lightweight‑model region while staying within a few percent of the best large‑scale models.

---

🛠️ Training (research‑oriented)

• Data: 14 M RGB‑depth pairs generated by knowledge‑distillation from Depth Anything V2 Large across 17 public image domains (e.g., COCO, Cityscapes, MegaDepth, etc.). The list of file paths is provided as training_files.txt.gz; the actual images must be sourced from the original datasets.
• Pipeline: scripts/prepare_index.py builds a JSON index → optional conversion to a NumPy mem‑map for fast I/O.
• Launch (single‑GPU example):

  python scripts/train.py --config configs/default.json
  

  Multi‑GPU via torchrun and standard DDP.
• Loss: scale‑and‑shift invariant depth loss + gradient regularisation, optimized with AdamW and a One‑Cycle LR schedule.

---

📱 Deploying on‑device

1. Choose the NPU‑friendly checkpoint (zipdepth_base_npu.pth).
2. Export to ONNX (scripts/export.py --format onnx --npu).
3. Convert the ONNX model to the target runtime (ONNX Runtime Mobile, CoreML, TFLite, NCNN, etc.).
4. The upsampling head uses only operators that are widely supported on mobile NPUs, ensuring a clean conversion.

---

🙏 Acknowledgements & Citation

The authors thank the creators of Marigold (for evaluation protocol) and Depth Anything V2 (for pseudo‑labels). If you use ZipDepth in research, cite:

@inproceedings{tosi2026zipdepth,
  title     = {ZipDepth: Bringing Lightweight Zero-Shot Monocular Depth Anywhere, on Any Device},
  author    = {Tosi, Fabio and Bartolomei, Luca and Poggi, Matteo and Mattoccia, Stefano},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

---

📧 Contact

For questions, reach out to the authors at the email addresses listed in the README.