VCIP-RGBD/DFormer

DFormer – RGB‑D Semantic Segmentation Toolkit

What it is

• A research‑grade code base that implements a family of models (DFormer, DFormerv2, DFormer++) for semantic segmentation of RGB‑D images (color + depth). The models are described in papers published at ICLR 2024, CVPR 2025, and IEEE TPAMI 2026.
• The repository also contains tools for RGB‑D pre‑training, dataset preparation, training, evaluation, and inference.

Key capabilities

Feature	Details
Unified RGB‑D encoder	Processes color and depth jointly instead of using separate streams.
Geometry‑guided attention (DFormerv2)	Generates a geometry prior from depth and injects it into the attention mechanism, improving 3‑D scene understanding.
Efficiency‑focused variants (DFormer++)	Tiny/Small/Base models that balance accuracy (up to 59 % mIoU on NYU‑Depth v2) with low MACs and parameter counts.
Pre‑training on RGB‑D ImageNet	Scripts to pre‑train encoders on a large‑scale RGB‑D dataset, then fine‑tune on downstream segmentation benchmarks (NYU‑Depth v2, SUN‑RGBD, etc.).
Benchmarking utilities	FLOPs/parameter counting, latency measurement, and ready‑made config files for common datasets.
Visualization & demo	Simple inference script and a Hugging Face Space demo showing geometry‑attention maps.

How to get started

1. Create the environment (Python 3.10, PyTorch 2.1, CUDA 11.8) and install required packages:

   conda create -n dformer python=3.10 -y
   conda activate dformer
   conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 -c pytorch -c nvidia
   pip install mmcv==2.1.0 -f https://download.openmmlab.com/mmcv/dist/cu118/torch2.1/index.html
   pip install tqdm opencv-python scipy tensorboardX tabulate easydict ftfy regex timm
   # optional: install TransNeXt swattention CUDA extension for faster local attention
   

2. Download data and checkpoints – the README lists Google Drive, OneDrive and BaiduNetdisk links for NYU‑Depth v2, SUN‑RGBD, and pre‑trained weights. Place them under datasets/ and checkpoints/ following the shown folder tree.
3. Train – pick a config (e.g., local_configs.NYUDepthv2.DFormerPP_B) and run:

   bash train.sh local_configs.NYUDepthv2.DFormerPP_B
   

   Checkpoints are saved under checkpoints/<model>/.
4. Evaluate – after training, run:

   bash eval.sh
   

5. Inference / visualization – generate segmentation maps with:

   bash infer.sh
   

6. Measure efficiency – FLOPs/params:

   PYTHONPATH="$(dirname $0)/..":$PYTHONPATH python benchmark.py --config local_configs.NYUDepthv2.DFormer_Large
   

   Latency (device‑specific):

   python utils/latency.py --config local_configs.NYUDepthv2.DFormer_Large
   

Performance snapshot (NYU‑Depth v2 mIoU)

Model	Params	MACs	mIoU
DFormer++‑T	17.3 M	29.5 G	57.0
DFormer++‑S	37.2 M	56.4 G	~58.1
DFormer++‑B	66.7 M	97.5 G	59.0

Related projects

• DFormer‑SOD – RGB‑D salient object detection (separate repo).
• DFormer‑Jittor – implementation using the Jittor deep‑learning framework (a Chinese‑origin alternative to PyTorch).
• RGB‑D ImageNet pre‑training – code for building large‑scale RGB‑D pre‑trained encoders.

Citation If you use the code, cite the three papers listed in the README (ICLR 2024, CVPR 2025, TPAMI 2026).

License

• Non‑commercial use only (as stated in the LICENSE section).

---

This summary is based solely on the information provided in the repository’s README.