ReMu

Reconstruct Multi-layer 3D Clothed Humans from Images

This repository is co-developed by master's students Nauryzbay Koishekenov, Onat Vuran, and their supervisors Hsuan-I Ho and Tianjian Jiang at ETH Zurich. It accompanies our series of works on reconstructing clothed humans as multi-layer, simulation-ready garment meshes from images.

ReMu reconstructs 3D garments from single-image or multi-image inputs.

The reconstructed garments can also be simulated with the optional ContourCraft demo.
See docs/contourcraft_demo.md.

Install

We test our code with Python 3.12, PyTorch 2.5.1, and CUDA 12.1 on Ubuntu 22.04 with one A100 GPU. The GPU toolchain must be visible at build and run time (nvdiffrast / kaolin compile CUDA plugins lazily). nvcc 12.1 rejects GCC > 12, so point CC/CXX to GCC 12 explicitly:

export CUDA_HOME=/usr/local/cuda-12.1
export PATH=$CUDA_HOME/bin:$PATH
export CC=$(which gcc-12) CXX=$(which g++-12)

Install Python packages

# core + stages 1-4 (note the flags — see below)
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt                  # unified project dependency list
pip install --no-build-isolation git+https://github.com/NVlabs/nvdiffrast.git

Install dependency-sensitive packages

Install these packages without dependency resolution so they do not drift transformers/diffusers/tokenizers/OpenCV away from the tested stack:

pip install --no-deps supervision==0.25.1
SAM2_BUILD_CUDA=0 pip install --no-deps git+https://github.com/facebookresearch/sam2.git
pip install --no-deps git+https://github.com/luca-medeiros/lang-segment-anything.git

Framework

ReMu is organized as a four-step framework:

data  ─▶  reconstruction  ─▶  registration & penetration removal  ─▶  refinement

Backends are interchangeable at each stage. For example, single_image first inpaints a BMVC-style set of layer images, and the later stages can then use either SiTH or UP2You. We provide two main preset configs:

• bmvc — reproduces our published results (SiTH reconstruction + neural-UDF refinement).
• remu — the upgraded pipeline (single-image diffusion layering, UP2You reconstruction, mesh-based detail-restoring refinement).

Step	bmvc backend	remu backend
1. Data	multi-layer images	single image → multi-layer images (SD3 inpaint + pose ControlNet)
2. Reconstruction	SiTH	UP2You
3a. Registration (3D seg + canonicalize)	knn_lbs or diffuse_lbs	-
3b. Penetration removal	layerwise (shared)	layerwise (shared)
4. Refinement	neural UDF (cleaner)	mesh (faster)

Additional wild1 / wild2 presets are included for the bundled in-the-wild examples.

See docs/reference.md for per-stage details, data conventions, and advanced usage.

Data preparation and setup

The repository uses this local folder layout for data and generated artifacts:

remu/
  examples/      downloaded example data and prompts
  checkpoints/   SiTH / UP2You checkpoints
  body_models/   licensed SMPL-X body model files
  outputs/       generated runs

Copy the resource path file once:

cp configs/paths.example.yaml configs/paths.local.yaml

configs/paths.local.yaml stores only resource roots. Edit these paths if you want to keep assets in different directories:

weights_root: checkpoints
body_model_root: body_models
work_root: outputs

Example data

Fetch the example inputs:

python scripts/download_data.py --subset examples

The example data layout is intentionally compact:

examples/
  prompts.json
  4d_dress_00127/
    input_img/       00127_Outer.png, 00127_Inner.png, 00127_Lower.png
    smplx_params/    per-layer *_smplx.pkl + matching *_smplx.obj
  wild_5/
    input_img/       5_Outer.png, 5_InOuter.png, 5_Inner.png, 5_Lower.png
    smplx_params/    per-layer *_smplx.pkl + matching *_smplx.obj

Dataset roots live in the presets. The bundled presets default to examples/4d_dress_00127 (bmvc, remu) and examples/wild_5 (wild1, wild2). Override that per run with --dataset-root <path> if your data lives elsewhere.

External models and checkpoints

We do not release any model weights of our own; the core pipeline is optimization-based. You still need these external assets:

body_models/smplx/SMPLX_MALE.npz
body_models/smplx/SMPLX_FEMALE.npz
body_models/smplx/SMPLX_NEUTRAL.npz
checkpoints/sith/recon_model.pth
checkpoints/up2you/pretrained_models/
checkpoints/up2you/human_models/

SiTH's public reconstruction checkpoint can be placed with:

mkdir -p checkpoints/sith
wget -O checkpoints/sith/recon_model.pth \
    https://files.ait.ethz.ch/projects/SiTH/recon_model.pth

UP2You weights are hosted on Hugging Face:

mkdir -p checkpoints/up2you
huggingface-cli download Co2y/UP2You --local-dir checkpoints/up2you

The remu single-image inpainting stage uses Stability AI's Stable Diffusion 3 model. Before running it, accept the model terms on Hugging Face and run huggingface-cli login, or set data.single_image.model_kwargs.base_model to a local or mirrored SD3 checkpoint.

Run

One command runs all four stages from a preset and threads each stage's output into the next:

python scripts/run.py --preset bmvc --subject 00127 --gender male      # reproduce BMVC
python scripts/run.py --preset remu --subject 00127 --gender male      # upgraded pipeline
python scripts/run.py --preset wild2 --subject 5 --gender male         # bundled wild example

Pass --gender male|female. The wrong gender silently corrupts registration and scale. SiTH SMPL-X fitting always uses the male model, but its feed-forward reconstruction model can consume meshes from different gender templates.

Outputs land in outputs/<preset>/<subject>/ with a fixed 4-directory layout:

outputs/<preset>/<subject>/
  data/            output_img/ + smplx_params/ + optional back_images/
  recon/           per-layer *_reco.obj meshes + smplx_params/
  registration/    canonicalized + de-penetrated meshes
  refine/          final textured OBJ + MTL + albedo/normal maps

Partial runs

Each stage reads exclusively from the previous stage's directory, so you can skip stages whose outputs you already have (--steps is a comma-separated subset of data,reconstruction,registration,refinement):

# start from prebuilt recons (skip data + reconstruction)
python scripts/run.py --preset bmvc --subject 00127 --gender male \
    --steps registration,refinement --recon-dir outputs/bmvc/00127/recon

# refinement only, on an existing registration dir, forcing the mesh backend
python scripts/run.py --preset bmvc --subject 00127 --gender male --steps refinement \
    --refine-backend mesh --reg-dir outputs/bmvc/00127/registration

On an A100, SiTH reconstruction is ~1.5 min/layer and neural_udf refinement dominates (~10–12 min for the joint fit); a full bmvc chain is ~15 min/subject. Per-stage drivers live under scripts/{data,recon,register,refine}/ for advanced or standalone use — see docs/reference.md.

Garment-simulation demo (optional)

The refined layered garments can be driven through ContourCraft (Grigorev et al., SIGGRAPH 2024) for physically based cloth simulation. This is an optional downstream application — scripts/run.py stops after refinement. The simulation runs in the same ReMu environment (no separate conda env, no PyTorch3D).

Additional packages

pip install -r requirements-cc.txt \
    -f https://data.pyg.org/whl/torch-2.5.0+cu121.html

# CCCollisions (build from source)
git clone https://github.com/NVIDIA/cuda-samples.git
export CUDA_SAMPLES_INC=$(pwd)/cuda-samples/Common
git clone https://github.com/Dolorousrtur/CCCollisions.git
cd CCCollisions && pip install --no-build-isolation . && cd ..

Usage

# Simulate (output goes to outputs/<preset>/<subject>/simulation/)
python scripts/simulate_contourcraft.py \
    --refine-dir outputs/bmvc/00127/refine \
    --params-dir outputs/bmvc/00127/refine/smplx \
    --subject 00127 --layers Lower,Inner,Outer \
    --pose-seq examples/amass/cmu/01_01_stageii.npz \
    --speed 0.5 --n-sim-frames 300

# Play back with AITViewer
python scripts/play_simulation.py outputs/bmvc/00127/simulation/simulation.pkl \
    --texture-dir outputs/bmvc/00127/refine/lowres

See docs/contourcraft_demo.md for full setup, options, playback, and troubleshooting.

License

Our code is released under the MIT License. Vendored components under third_party/ (SiTH, UP2You, ContourCraft), external models, and optional demo tools/data (e.g. SMPL-X, AMASS) remain under their respective licenses.

References

Our implementation builds on SiTH, UP2You, InstantAvatar, smplfitter, ContourCraft, and AITViewer.

If you find this code useful in your research, please consider citing:

@inproceedings{vuran2025remu,
  title   = {ReMu: Reconstructing Multi-layer 3D Clothed Human from Images},
  author  = {Vuran, Onat and Ho, Hsuan-I},
  booktitle = {British Machine Vision Conference (BMVC)},
  year    = {2025}
}

@inproceedings{ho2024sith,
    title={SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion},
    author={Ho, Hsuan-I and Song, Jie and Hilliges, Otmar},
    booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2024}
  }

@article{jiang2022instantavatar,
  author    = {Jiang, Tianjian and Chen, Xu and Song, Jie and Hilliges, Otmar},
  title     = {InstantAvatar: Learning Avatars from Monocular Video in 60 Seconds},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2023},
}