ant-research/4DAnyone

4DAnyone – Turning a Single Monocular Video into a Full‑View 4D Capture

What it does – 4DAnyone is a research‑grade neural model that takes an ordinary, single‑camera video of a person and synthesises dozens of synchronized, view‑consistent videos from new camera angles. The generated multi‑view footage can then be fed to a 4‑D Gaussian Splatting (4DGS) pipeline to obtain a dynamic 3‑D reconstruction (a “4‑D” capture) of the subject.

Key capabilities

• Multi‑view generation – From one input clip it can produce 6, 24, 48 or any multiple of views that are evenly spaced around the subject, with configurable pitch layers and yaw spans.
• Consumer‑GPU friendly – Peak GPU memory is kept under 24 GB (fits a RTX 4090) and a 121‑frame clip is generated in ~27 s on a single RTX 4090.
• Two model variants –
  • 4DAnyone‑Base – the original denoising schedule (high quality, slower).
  • 4DAnyone‑Turbo – a distilled four‑step denoiser that is enabled by default; it is ~5.6× faster with comparable quality.
• Plug‑and‑play inference – Models and example data are downloaded automatically on first run; a few command‑line flags let you control view count, camera layout, GPU usage and whether to use the Turbo model.
• Downstream 4DGS support – The repo includes a short guide for feeding the generated videos into nerfstudio’s 3‑D Gaussian Splatting pipeline.

Installation

# Clone the repo and its submodule (GVHMR for motion recovery)
git clone https://github.com/ant-research/4DAnyone.git
cd 4DAnyone
git submodule update --init third_party/GVHMR

# Create a fresh conda environment (Python 3.11) and install deps
conda create -n 4danyone python=3.11 -y
conda activate 4danyone
pip install -r requirements.txt

Optional speed‑ups: install FlashAttention‑3 or SageAttention; the code will auto‑detect the best backend.

Running inference (example for a 24‑view full orbit):

python inference.py \
    --video_path data/source/pexels/2785536-uhd_2160_3840_25fps.mp4 \
    --views_per_layer 24   # 24 cameras evenly spaced around 360°

Other common layouts are demonstrated in the README (6‑view, 48‑view three‑layer, 24‑view frontal arc). The main arguments are:

• views_per_layer – must be divisible by 4 or 6.
• layer_pitches – list of pitch angles (degrees) for each horizontal ring.
• start_yaw / yaw_span – control horizontal coverage.
• gpu_ids – select GPUs for parallel stages.
• enable_turbo – toggle the Turbo model (on by default).

Output layout

data/
├─ gvhmr/results/<clip>/          # motion‑recovery intermediate results
└─ fdanyone/<clip>/
   ├─ metadata.json               # settings, timings, resource usage
   ├─ cameras.json                # final camera rig description
   ├─ skeletons/00.mp4 … <N‑1>.mp4
   └─ videos/
      ├─ sparse/…                # default 24‑view proposals (low‑density)
      └─ dense/…                 # full set of generated target views

The videos can be directly consumed by the nerfstudio guide (docs/nerfstudio.md) to build a 3‑D Gaussian Splatting model; future work aims to add an end‑to‑end 4DGS pipeline.

Hardware requirements – A single RTX 4090 (or any GPU with ≥24 GB VRAM) is sufficient for the default 24‑view generation. Multi‑GPU setups can be used for parallel pose/VAE stages.

Roadmap highlights

• ✅ Peak memory < 32 GB (now < 24 GB).
• ✅ 5.58× speedup with Turbo model.
• ✅ 1.42× end‑to‑end speedup for the full 24‑view pipeline.
• ✅ Upcoming support for direct 4DGS reconstruction.

Citation – If you use the code or the ideas, please cite the arXiv paper:

@article{jin2026fdanyone,
  title={4DAnyone: Create Anyone in 4D from a Casual Monocular Video},
  author={Jin, Yudong and Xie, Tao and Zhang, Qihang and Shen, Zehong and Xu, Zhen and Shen, Yujun and Bao, Hujun and Zhou, Xiaowei and Xu, Yinghao},
  journal={arXiv preprint arXiv:2608.20335},
  year={2026},
  url={https://arxiv.org/abs/2608.20335}
}

---

TL;DR – 4DAnyone is a research implementation that turns a casual single‑camera video of a person into a dense set of synthetic camera views, enabling 4‑D reconstruction with consumer‑grade GPUs. It ships ready‑to‑run inference scripts, automatic model downloading, and a clear guide for downstream 3‑DGS rendering.