LordLiang/DrawingSpinUp

DrawingSpinUp: 3D Animation from Single Character Drawings

What it is: A research project (SIGGRAPH Asia 2024) that turns a single 2D drawing of a character into a fully 3D, animated version of that character — while preserving the original hand-drawn style.

The problem: Normally, animating a 2D drawing in 3D requires manually building a 3D model, adding textures, and rigging it for animation. This project automates the whole pipeline from just one flat image.

How it works (3 main steps):

1. Contour removal — The system first removes the dark outline of the drawing using a neural network (FFC-ResNet, borrowed from the LaMa inpainting project). This gives a clean, color-only version of the character.

2. 3D character reconstruction — The cleaned image is fed into a multi-view generation model (based on Wonder3D) that hallucinates what the character looks like from different angles. These views are then fused into a single textured 3D mesh (an .obj file).

3. Stylized animation — The 3D character is rigged (using Mixamo, a free auto-rigging tool) and animated with a motion clip (e.g., jumping, dabbing). The system then trains a per-character style-translation model that re-applies the original drawing's hand-drawn look onto the rendered 3D frames, so the final animation looks like the original sketch is moving, not like a generic 3D model.

Key technical notes:

• The style translator is trained per character (a few minutes on one GPU), but once trained it can be applied to any new animation frames without retraining.
• Requires a single RTX 4090 or 2080 Ti GPU, Python 3.8, PyTorch 2.0, CUDA 11.8.
• The repo includes a toy example (a character named ff7ab74a67a443e3bda61e69577f4e80) with pre-made rigged animations so users can run the full pipeline end-to-end immediately.
• Users can also supply their own 512×512 character drawing plus a foreground mask.

Who it's for: Researchers and hobbyists interested in 3D reconstruction, character animation, and style transfer. The code is research-grade — it requires some comfort with Python, GPU setup, and command-line workflows.

Notable dependencies: LaMa (inpainting), Wonder3D (multi-view 3D reconstruction), Few-Shot-Patch-Based-Training (style transfer), Blender (for rendering frames), and Mixamo (auto-rigging).