UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation

Arxiv

Zexiang Liu1* Yangguang Li1* Youtian Lin1* Xin Yu2 Sida Peng3 Yan-Pei Cao1 Xiaojuan Qi2 Xiaoshui Huang4 Ding Liang1 Wanli Ouyang4

1VAST    2The University of Hong Kong     3Zhejiang University      4Shanghai AI Laboratory

* Equal contributions

Paper

Code

Abstract

Recent advancements in text-to-3D generation technology have significantly advanced the conversion of textual descriptions into imaginative well-geometrical and finely textured 3D objects. Despite these developments, a prevalent limitation arises from the use of RGB data in diffusion or reconstruction models, which often results in models with inherent lighting and shadows effects that detract from their realism, thereby limiting their usability in applications that demand accurate relighting capabilities. To bridge this gap, we present UniDream, a text-to-3D generation framework by incorporating unified diffusion priors. Our approach consists of three main components: (1) a dual-phase training process to get albedo-normal aligned multi-view diffusion and reconstruction models, (2) a progressive generation procedure for geometry and albedo-textures based on Score Distillation Sample (SDS) using the trained reconstruction and diffusion models, and (3) an innovative application of SDS for finalizing PBR generation while keeping a fixed albedo based on Stable Diffusion model. Extensive evaluations demonstrate that UniDream surpasses existing methods in generating 3D objects with clearer albedo textures, smoother surfaces, enhanced realism, and superior relighting capabilities.


Method

ant

Overview of UniDream. Left: the multi-view diffusion model generates multi-view images based on input text. Middle: first, four view albedo maps obtain 3D prior by the reconstruction model, and then the multi-view diffusion model performs SDS optimization based on the 3D prior to generate a 3D object with albedo texture. Right: using Stable Diffusion model to generate PBR material..


Example Generated Results

PBR->Normal->Albedo->Roughness->Metallic for reference.

Methods Comparison Results

Each case has 7 results, which are the results comparison of DreamFusion, Magic3D, ProlificDreamer, MVDream, UniDream-PBR, UniDream-Relighting-I, and UniDream-Relighting-II. UniDream-PBR is PBR->Normal->Albedo->Roughness->Metallic from left to right. UniDream-Relighting is the result of replacing the environment map for relighting, which is PBR->Diffuse Color->Specular Color->Diffuse Light->Specular Light from left to right.

PBR Comparison Results

BR->Normal->Albedo->Roughness->Metallic for reference.

Citation

@misc{liu2023unidream,
      title={UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation}, 
      author={Zexiang Liu and Yangguang Li and Youtian Lin and Xin Yu and Sida Peng and Yan-Pei Cao and Xiaojuan Qi and Xiaoshui Huang and Ding Liang and Wanli Ouyang},
      year={2023},
      eprint={2312.08754},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}