Approach

We prompt Stable Diffusion for virtual views of an object category that are then used to train a monocular articulated object reconstruction model that factorises the input image of an object instance into articulated shape, appearance (albedo and diffuse and ambient intensities), viewpoint, and light direction. During training, we also sample synthetic instance views that are then "critiqued" by Stable Diffusion to guide the learning.

Single-view 3D Reconstruction

Controllable 3D Shape Synthesis

Our method enables the generation of controllable 3D assets from either a real image or an image synthesised using Stable Diffusion. Once generated, we have the ability to adjust lighting, swap textures between models of the same category, and even animate the shape.

Animation

Relighting

Texture Swapping

Animodel - 3D Articulated Animals Dataset and Benchmark

We introduce a new 3D Articulated Animals Dataset to directly evaluate the quality of single-view 3D reconstruction of articulated animals. This dataset includes textured 3D meshes of articulated animals, including horses, cows, and sheep, crafted by a professional 3D artist, and is accompanied by realistic articulated poses.

Paper

arXiv Preprint

BibTeX

@Article{jakab2023farm3d,
    title={{Farm3D}: Learning Articulated 3D Animals by Distilling 2D Diffusion},
    author={Jakab, Tomas and Li, Ruining and Wu, Shangzhe and Rupprecht, Christian and Vedaldi, Andrea},
    journal={arXiv preprint arXiv:2304.10535},
    year={2023},
}