Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

Zibo Zhao1,2       Wen Liu2      Xin Chen2      Xianfang Zeng2     
Rui Wang2      Pei Cheng2      Bin Fu2      Tao Chen3
Gang Yu2      Shenghua Gao1,4,5
1ShanghaiTech University      2Tencent PCG, China     
3School of Information Science and Technology, Fudan University, China     
4Shanghai Engineering Research Center of Intelligent Vision and Imaging
5Shanghai Engineering Research Center of Energy Efficient and Custom AI IC


We present a novel alignment-before-generation approach to tackle the challenging task of generating general 3D shapes based on 2D images or texts. Directly learning a conditional generative model from images or texts to 3D shapes is prone to producing inconsistent results with the conditions because 3D shapes have an additional dimension whose distribution significantly differs from that of 2D images and texts. To bridge the domain gap among the three modalities and facilitate multi-modal-conditioned 3D shape generation, we explore representing 3D shapes in a shape-image-text-aligned space. Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model (ASLDM). The former model encodes the 3D shapes into the shape latent space aligned to the image and text and reconstructs the fine-grained 3D neural fields corresponding to given shape embeddings via the transformer-based decoder. The latter model learns a probabilistic mapping function from the image or text space to the latent shape space. Our extensive experiments demonstrate that our proposed approach can generate higher-quality and more diverse 3D shapes that better semantically conform to the visual or textural conditional inputs, validating the effectiveness of the shape-image-text-aligned space for cross-modality 3D shape generation.

Network Architecture

Qualitative Comparison

Click for More Image-conditioned Comparisons

Click for More Text-conditioned Comparisons

Test in the wild

Click for More Image-conditioned Cases

Click for More Text-conditioned Cases


title={Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation},
author={Zibo Zhao and Wen Liu and Xin Chen and Xianfang Zeng and Rui Wang and Pei Cheng and BIN FU and Tao Chen and Gang YU and Shenghua Gao},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems},