Fine-Tuning TRELLIS.2 for 3D Generation with LoRA: A Practical Guide

TL;DR: Hugging Face released a practical fine-tuning guide, scripts, and Docker containers for adapting TRELLIS.2, a 3D generation model, to custom 3D assets with LoRA. On a 188-object footwear subset of Google Scanned Objects, fine-tuning only the two geometry-controlling DiTs improved held-out geometry substantially while leaving the texture stage frozen.

What Hugging Face released

Hugging Face published a guide and companion tooling for fine-tuning TRELLIS.2 with LoRA. The release includes fine-tuning scripts and two Docker containers: one for preprocessing 3D meshes into TRELLIS.2 latents, and one for training LoRA adapters. The containers can run locally with Docker or on Hugging Face Jobs.

The guide walks through an end-to-end experiment on footwear from the Google Scanned Objects dataset. It covers which pipeline components to fine-tune, which to freeze, how the training loop works, and how the adapted model performs on held-out objects.

The companion repository is suvadityamuk/trellis2-finetuning, and the trained adapters are published at suvadityamuk/trellis-2-finetune-gso.

Why fine-tune a 3D generation model?

General-purpose 3D generation models can produce plausible objects but fail on details that matter for a specific application. The guide uses footwear as an example: the pretrained TRELLIS.2 model usually produced shoes that looked correct, but the soles were often inaccurate.

The experiment uses 188 footwear objects from GSO, which contains 1,030 objects in total. Nine of the 188 objects are held out for evaluation, and the remaining 179 are used for training. The goal is to test whether LoRA adapters can make the soles more accurate for unseen footwear objects.

Choosing what to freeze matters as much as choosing what to fine-tune.

How TRELLIS.2 works

TRELLIS.2 generates a 3D object through five main stages: remove the image background, encode the image, predict the coarse structure, generate detailed geometry, and generate materials. The pipeline then exports the result as a textured mesh.

Image encoding

BiRefNet removes the background from the input image. DINOv3-L then encodes the image into feature tokens. The generation models use these tokens to guide the 3D object.

Sparse structure generation

The sparse-structure flow DiT predicts the object's coarse shape. It is a diffusion transformer that produces a small latent of size 16³ × 8. A VAE decoder then converts this latent into a 64 × 64 × 64 occupancy grid. Each voxel in the grid indicates whether that location is part of the object.

The guide adds LoRA adapters to this model because its output guides the geometry stage.

O-Voxel and structured latents

O-Voxel is the 3D format used by TRELLIS.2. It is a sparse voxel grid that keeps only the voxels on the object's surface. Each voxel stores local geometry, such as a mesh vertex, and material attributes. A CPU can convert a mesh into O-Voxel in seconds, and a GPU can convert O-Voxel back into a mesh in seconds.

Structured sparse latents come from TRELLIS. They keep 3D data in sparse tensors, which makes computation faster. The model uses only approximately 10% of the voxels in the grid, so it does not spend compute on empty voxels. The shape flow DiT and texture flow DiT are sparse transformers, while the sparse-structure flow DiT is a dense transformer.

Geometry generation

The shape flow DiT uses the occupancy grid and DINOv3 feature tokens to produce a latent. Another VAE decoder converts this latent into O-Voxel geometry at a resolution of 512 × 512 × 512.

The guide adds LoRA adapters to this DiT because it controls the object's shape.

Material generation and mesh creation

The texture flow DiT uses the shape latent and DINOv3 features as input. It defines the materials and textures of the geometry's surface. A VAE decoder then converts the latent into base color, metallic, roughness, and alpha values for each voxel.

The guide does not add LoRA adapters to this stage. The footwear dataset has diffuse color maps but lacks the separate material maps needed to train the texture flow DiT. The base model already produces good colors and textures, and the problem being addressed is the shoe's shape.

Finally, the pipeline combines geometry and material attributes into a textured mesh and exports it as a GLB file.

What to fine-tune and what to freeze

The guide adds LoRA adapters only to the sparse-structure flow DiT and the shape flow DiT. These two DiTs control the object's shape and geometry. The texture flow DiT is frozen to preserve its existing capabilities.

Training the texture flow DiT on the available diffuse color maps would be problematic because those maps include baked lighting effects. The dataset does not provide separate material maps. If the texture model were trained on diffuse maps, it could learn to represent lighting as a surface property. It could also forget what it learned before, a problem known as catastrophic forgetting.

LoRA is preferred over full fine-tuning because the footwear subset is small: 188 objects from GSO. Full fine-tuning costs much more and can overwrite the model's learned priors. The guide's configuration trains approximately 9.3 million adapter parameters in total across both targeted DiTs.

The LoRA configuration uses rank 8 adapters in the attention and feed-forward MLP layers of the sparse-structure flow DiT. It uses rank 16 adapters in the attention layers of the shape flow DiT. The rank sets the size of the adapter matrices; a higher rank lets the adapter learn more.

Inside the training pipeline

Preprocessing into latents

The preprocessing container uses Blender to render 16 views of each object. These views serve as conditioning images during LoRA training. Blender runs without a display through the xvfb-run utility. The container also processes each object through the TRELLIS.2 preprocessing pipeline.

The pipeline uses TRELLIS.2's frozen encoders to convert input meshes into training latents. It saves those latents so later runs do not have to encode the meshes again. During training, one random view is selected for each sample as the conditioning image.

For the 188 footwear objects, the full preprocessing run took approximately 1 hour on an a100-large Hugging Face Jobs instance. The preprocessing command supports options such as --holdout 0.05, --views 16, and --workers 8. The holdout setting keeps 9 of the 188 objects for evaluation.

Preparing LoRA adapters

The guide uses Hugging Face's PEFT library to insert LoRA adapters into the selected layers. TRELLIS.2 has a custom forward pass, so the standard PeftModel() wrapper does not work. Instead, the code calls inject_adapter_in_model(), which adds adapters directly to the model while preserving its custom interface.

Trainable adapter weights are kept in 32-bit floating point for numerical stability. The forward pass uses bfloat16 autocasting, which lets PyTorch use lower precision for some operations and reduces memory use. Autocasting does not change how the frozen backbone weights are stored.

Training objective and loop

The training objective is rectified flow matching, the same objective TRELLIS.2 used during pretraining. The guide notes that this objective is also used in image-generation models such as Stable Diffusion 3 and Flux 2.

The training loop updates only the LoRA parameters in the sparse-structure and shape DiTs. All original model weights remain frozen. The optimizer is AdamW with betas set to (0.9, 0.95).

For each sample, the loop samples a timestep t. The sparse-structure flow DiT uses a logit-normal timestep schedule, while the shape flow DiT uses a uniform schedule, based on the TRELLIS.2 configuration. The loop generates random noise matching the data shape, mixes clean data with noise based on t, predicts velocity at t × 1000 to match pretraining conditions, and calculates loss as the mean squared error between predicted and target velocity.

The training setup uses a batch size of 4 with 4 gradient accumulation steps, giving an effective batch size of 16. Gradient checkpointing saves GPU memory by recomputing some intermediate values during backpropagation. For classifier-free guidance dropout, the loop sets the conditioning to zero for a random fraction of samples. After gradient accumulation, it clips gradients and performs an optimizer step. It logs loss every 10 steps and saves an adapter checkpoint every 500 steps.

Running the jobs

The guide provides two container images: cr.hf.co/suvadityamuk/trellis2-finetuning-prep:latest for preprocessing and cr.hf.co/suvadityamuk/trellis2-finetuning-train:latest for training. The same commands and flags work on Hugging Face Jobs and with local Docker; only the Docker options differ.

Before running, users need a Hugging Face token with access to DINOv3, because DINOv3 is a gated model. If using a custom dataset, meshes must be uploaded to the Hugging Face Hub first.

A representative training command for the sparse-structure flow DiT is:

export HF_TOKEN=hf_xxx

hf jobs run -d --flavor a100-large --timeout 12h --secrets HF_TOKEN \
  cr.hf.co/suvadityamuk/trellis2-finetuning-train:latest \
  trellis2-train --config ss \
    --latents your-username/my-meshes-trellis2 \
    --push-to your-username/my-trellis2-lora \
    --run-name ss_lora \
    --push-path ss_lora \
    --trackio-space your-username/trellis2-trackio

The training container supports two modes: ss for the sparse-structure flow DiT and shape for the shape flow DiT. Each mode references a different configuration file. The container records training metrics with Trackio, and if --push-to is set, it uploads adapter checkpoints to the Hugging Face Hub.

Both Hugging Face Jobs commands support resumption. Preprocessing skips objects that are already finished, and training continues from the newest checkpoint in the --push-to repository.

Results

The guide trained two adapters: one for the sparse-structure DiT and one for the shape DiT. Each run used one A100 80 GB GPU on Hugging Face Jobs. The sparse-structure run took 7.2 hours, and the shape run took 3.6 hours.

The training loss of the sparse-structure flow DiT falls quickly at the start and then more slowly. The loss of the shape flow DiT stays at approximately the same level. The guide cautions that these curves show how the training objective changes, but they do not directly measure the quality of generated geometry. Flow-matching loss varies with sampled objects, noise, and timesteps, so outputs must also be compared on held-out objects.

On the nine held-out objects, the fine-tuned model produced soles closer to the reference scans. The guide reports the following mean metrics:

Metric Base model With footwear adapters Direction
F-score @ 0.02 0.276 0.803 Higher is better
Chamfer distance 0.174 0.033 Lower is better
Normal consistency 0.530 0.804 Higher is better

F-score @ 0.02 combines precision and recall at a matching threshold of 2% of the scan's bounding-box diagonal. Chamfer distance measures the distance from points on one surface to the nearest points on the other surface in both directions. Normal consistency measures how closely surface directions agree.

With both adapters, the F-score is better on all nine held-out objects.

Practical lessons and future directions

The guide's main lesson is that the training target must agree with the problem and the available data. For the footwear experiment, the structure and shape stages were fine-tuned, while the texture stage was frozen because the dataset lacked the material maps needed for texture training.

The main lesson is to select a training target that agrees with the problem and the data that you have.

The guide suggests testing whether the same choices help other 3D generation models, such as Pixal3D and UniLat3D. It also identifies Gaussian-splat and world-generation models, such as HunyuanWorld 2.0 and NVIDIA Lyra 2.0, as possible next steps. Those models use different representations and training pipelines, so they may require different fine-tuning methods.

Sources