ActionMesh – Turning Videos into Animated 3D Meshes

What it does

ActionMesh is a research‑grade model that creates a full 3‑D animated mesh from a short video (16‑31 frames). It can work in two ways:

1. Video → 4D – give only a video and the system builds a new mesh that moves exactly as the subject does.
2. {Video + 3D} → 4D – supply an existing static mesh (e.g., a GLB file) and the model animates it to follow the motion in the video while keeping the original topology and texture.

The output can be a series of per‑frame .glb files, a single animated .glb that Blender can import, or a rendered video of the animation.

Why it matters

• Bridges vision and graphics – combines video understanding (via DINOv2, SAM2, background‑removal) with 3‑D diffusion to synthesize geometry and motion.
• Fast enough for interactive demo – on an H100 GPU the default run takes ~75 s, and a “fast” flag drops it to ~45 s.
• Low‑RAM option – can run on a Colab T4 (12 GB) with --low_ram, making the tech accessible to researchers without huge GPUs.
• Open‑source benchmark – the accompanying ActionBench dataset (128 video ↔ animated point‑cloud pairs) lets the community evaluate and compare future methods.

How to get started (quick‑start)

# clone and install
git clone git@github.com:facebookresearch/actionmesh.git && cd actionmesh
git submodule update --init --recursive
pip install -r requirements.txt -e .

# simple video‑to‑mesh run (replace paths as needed)
python inference/video_to_animated_mesh.py \
    --input assets/examples/davis_camel \
    --blender_path /path/to/blender   # optional, for .glb export

Add --fast for the quicker inference mode, and --low_ram if you are on a modest GPU (e.g., Colab).

Key components & dependencies

Component	Role	Install / Source
PyTorch 2.4 / CUDA 12.1	Core deep‑learning framework
PyTorch3D	Rendering animated meshes (needed for {video+3D} mode and benchmark evaluation)
Blender 3.5.1	Converts per‑frame meshes into a single animated GLB
TripoSG	Image‑to‑3‑D generator used as a backbone for shape priors
DINOv2‑large	Vision encoder providing robust frame features
RMBG‑1.4	Automatic background removal when masks are absent
Diffusers / Transformers	Underlying diffusion and transformer implementations

Input requirements

• Video: .mp4 or a folder of PNG frames.
• Length: 16‑31 frames (extra frames are ignored).
• Masks are optional – if missing, the pipeline runs RMBG to cut out the foreground. For best results the README recommends using the SAM2 demo to produce clean masks.

Output formats

Output	What you get	How to use
mesh_000.glb, mesh_001.glb, …	One mesh per timestep – useful for custom rendering pipelines
animated_mesh.glb	Single file with embedded animation – drag‑and‑drop into Blender 3.5.1
rendered.mp4	Video rendered with PyTorch3D – quick visual check

Benchmarks & performance

Hardware	Mode	Time	Quality note
H100 GPU	Default	~75 s	Highest quality
H100 GPU	--fast	~45 s	Slight quality drop
Colab T4 (12 GB)	--low_ram + --fast	~1 min 30 s	Acceptable for prototyping

Community resources

• Hugging Face demo – interactive web UI at facebook/ActionMesh.
• Colab notebook – ready‑to‑run notebook (ActionMesh.ipynb).
• ActionBench dataset – hosted on Hugging Face (facebook/actionbench) for evaluation.

License & citation

The code is released under the license in the repository’s LICENSE file. If you use the method in research, cite the CVPR 2026 paper (full bibtex provided in the README).

---

All information above is taken directly from the repository’s README; no external speculation added.