zju3dv/ENeRF

ENeRF – Efficient Neural Radiance Fields for Interactive Free‑viewpoint Video

What it is

• A research implementation of Efficient Neural Radiance Fields (ENeRF), a method that speeds up NeRF‑based view synthesis so that interactive (≈20‑50 FPS) free‑viewpoint video becomes feasible.
• Targets both static scenes (DTU, NeRF synthetic/LLFF) and dynamic human capture (ZJU‑MoCap) and even an outdoor dataset released by the authors.

Key capabilities

• Training a generalizable ENeRF model on the DTU multi‑view dataset.
• Fine‑tuning on a specific scene (e.g., a DTU scan, ZJU‑MoCap sequence, or the ENeRF‑Outdoor “actor1” data).
• Evaluation with standard metrics (PSNR, SSIM, LPIPS, depth error) and real‑time rendering speed reporting.
• An optional GUI for interactive rendering of human captures.

Installation (from the README)

1. Create a conda environment

   conda create -n enerf python=3.8
   conda activate enerf
   

2. Install PyTorch 1.9.0 (CUDA 11.1) and related packages:

   pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 \
       -f https://download.pytorch.org/whl/torch_stable.html
   

3. Install remaining Python dependencies

   pip install -r requirements.txt
   

4. Set a workspace directory where datasets, checkpoints and results will live:

   export workspace=$PATH_TO_YOUR_WORKSPACE
   

Datasets & pretrained models

• DTU (pre‑processed multi‑view data) – required for the baseline model and fine‑tuning.
• NeRF synthetic & LLFF – for evaluating on standard NeRF benchmarks.
• ZJU‑MoCap – human capture data used for the interactive GUI.
• ENeRF‑Outdoor – a new outdoor dataset released by the authors.
• A pre‑trained DTU model can be downloaded and placed at $workspace/trained_model/enerf/dtu_pretrain/latest.pth.

Typical workflow

1. Training (generalizable model on DTU):

   python train_net.py --cfg_file configs/enerf/dtu_pretrain.yaml
   

   Multi‑GPU training is supported via torch.distributed.
2. Fine‑tuning on a specific scan (example: DTU scan 114):

   cd $workspace/trained_model/enerf
   mkdir dtu_ft_scan114
   cp dtu_pretrain/138.pth dtu_ft_scan114
   cd $codespace   # directory containing the ENeRF code
   python train_net.py --cfg_file configs/enerf/dtu/scan114.yaml
   

   The README notes that 3 k and 11 k iterations take ~11 min and ~40 min on an i9‑12900K + RTX 3090.
3. Evaluation – e.g., on DTU:

   python run.py --type evaluate \
       --cfg_file configs/enerf/dtu_pretrain.yaml \
       enerf.cas_config.render_if False,True \
       enerf.cas_config.volume_planes 48,8 \
       enerf.eval_depth True
   

   Sample output shows PSNR ≈ 27.6, SSIM ≈ 0.957, LPIPS ≈ 0.089 and ~21.8 FPS at 512×640.
4. Interactive rendering (human capture):

   python gui_human.py --cfg_file configs/enerf/interactive/zjumocap.yaml
   

   Mouse/keyboard controls are listed in the README.

Performance highlights from the README

• DTU evaluation: 27.6 dB PSNR, 0.957 SSIM, 0.089 LPIPS, 21.8 FPS.
• ZJU‑MoCap: 31.48 dB PSNR, 0.971 SSIM, 0.042 LPIPS, 49.2 FPS.
• Real‑time rendering is achieved on a high‑end desktop (i9‑12900K + RTX 3090).

Citation If you use the code in research, cite the SIGGRAPH Asia 2022 paper:

@inproceedings{lin2022enerf,
  title={Efficient Neural Radiance Fields for Interactive Free-viewpoint Video},
  author={Lin, Haotong and Peng, Sida and Xu, Zhen and Yan, Yunzhi and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},
  booktitle={SIGGRAPH Asia Conference Proceedings},
  year={2022}
}

---

Bottom line: ENeRF is a full‑stack open‑source implementation for training, fine‑tuning, evaluating, and interactively rendering neural radiance fields, with a focus on speed‑efficient inference suitable for free‑viewpoint video applications.