Generative Multiplane Images: Making a 2D GAN 3D-Aware

Xiaoming Zhao; Fangchang Ma; David Güera; Zhile Ren; Alexander G. Schwing; Alex Colburn

doi:10.1007/978-3-031-20065-6_2

Generative Multiplane Images: Making a 2D GAN 3D-Aware

Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G. Schwing, Alex Colburn

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

What is really needed to make an existing 2D GAN 3D-aware? To answer this question, we modify a classical GAN, i.e., StyleGANv2, as little as possible. We find that only two modifications are absolutely necessary: 1) a multiplane image style generator branch which produces a set of alpha maps conditioned on their depth; 2) a pose-conditioned discriminator. We refer to the generated output as a ‘generative multiplane image’ (GMPI) and emphasize that its renderings are not only high-quality but also guaranteed to be view-consistent, which makes GMPIs different from many prior works. Importantly, the number of alpha maps can be dynamically adjusted and can differ between training and inference, alleviating memory concerns and enabling fast training of GMPIs in less than half a day at a resolution of 1024 ². Our findings are consistent across three challenging and common high-resolution datasets, including FFHQ, AFHQv2 and MetFaces.

Original language	English (US)
Title of host publication	Computer Vision – ECCV 2022 - 17th European Conference, Proceedings
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
Publisher	Springer
Pages	18-35
Number of pages	18
ISBN (Print)	9783031200649
DOIs	https://doi.org/10.1007/978-3-031-20065-6_2
State	Published - 2022
Event	17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel Duration: Oct 23 2022 → Oct 27 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13665 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th European Conference on Computer Vision, ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	10/23/22 → 10/27/22

Keywords

3D-aware generation
GANs
Multiplane images

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Online availability

10.1007/978-3-031-20065-6_2

Library availability

Discover UIUC Full Text

Cite this

Zhao, X., Ma, F., Güera, D., Ren, Z., Schwing, A. G., & Colburn, A. (2022). Generative Multiplane Images: Making a 2D GAN 3D-Aware. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022 - 17th European Conference, Proceedings (pp. 18-35). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13665 LNCS). Springer. https://doi.org/10.1007/978-3-031-20065-6_2

Generative Multiplane Images: Making a 2D GAN 3D-Aware. / Zhao, Xiaoming; Ma, Fangchang; Güera, David et al.
Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. ed. / Shai Avidan; Gabriel Brostow; Moustapha Cissé; Giovanni Maria Farinella; Tal Hassner. Springer, 2022. p. 18-35 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13665 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Zhao, X, Ma, F, Güera, D, Ren, Z, Schwing, AG & Colburn, A 2022, Generative Multiplane Images: Making a 2D GAN 3D-Aware. in S Avidan, G Brostow, M Cissé, GM Farinella & T Hassner (eds), Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13665 LNCS, Springer, pp. 18-35, 17th European Conference on Computer Vision, ECCV 2022, Tel Aviv, Israel, 10/23/22. https://doi.org/10.1007/978-3-031-20065-6_2

Zhao X, Ma F, Güera D, Ren Z, Schwing AG, Colburn A. Generative Multiplane Images: Making a 2D GAN 3D-Aware. In Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors, Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. Springer. 2022. p. 18-35. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-20065-6_2

Zhao, Xiaoming ; Ma, Fangchang ; Güera, David et al. / Generative Multiplane Images : Making a 2D GAN 3D-Aware. Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. editor / Shai Avidan ; Gabriel Brostow ; Moustapha Cissé ; Giovanni Maria Farinella ; Tal Hassner. Springer, 2022. pp. 18-35 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{c6f949908e844a87bdced089256ccb57,

title = "Generative Multiplane Images: Making a 2D GAN 3D-Aware",

abstract = "What is really needed to make an existing 2D GAN 3D-aware? To answer this question, we modify a classical GAN, i.e., StyleGANv2, as little as possible. We find that only two modifications are absolutely necessary: 1) a multiplane image style generator branch which produces a set of alpha maps conditioned on their depth; 2) a pose-conditioned discriminator. We refer to the generated output as a {\textquoteleft}generative multiplane image{\textquoteright} (GMPI) and emphasize that its renderings are not only high-quality but also guaranteed to be view-consistent, which makes GMPIs different from many prior works. Importantly, the number of alpha maps can be dynamically adjusted and can differ between training and inference, alleviating memory concerns and enabling fast training of GMPIs in less than half a day at a resolution of 1024 2. Our findings are consistent across three challenging and common high-resolution datasets, including FFHQ, AFHQv2 and MetFaces.",

keywords = "3D-aware generation, GANs, Multiplane images",

author = "Xiaoming Zhao and Fangchang Ma and David G{\"u}era and Zhile Ren and Schwing, {Alexander G.} and Alex Colburn",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 17th European Conference on Computer Vision, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2022",

doi = "10.1007/978-3-031-20065-6_2",

language = "English (US)",

isbn = "9783031200649",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "18--35",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e} and Farinella, {Giovanni Maria} and Tal Hassner",

booktitle = "Computer Vision – ECCV 2022 - 17th European Conference, Proceedings",

address = "Germany",

}

TY - GEN

T1 - Generative Multiplane Images

T2 - 17th European Conference on Computer Vision, ECCV 2022

AU - Zhao, Xiaoming

AU - Ma, Fangchang

AU - Güera, David

AU - Ren, Zhile

AU - Schwing, Alexander G.

AU - Colburn, Alex

PY - 2022

Y1 - 2022

N2 - What is really needed to make an existing 2D GAN 3D-aware? To answer this question, we modify a classical GAN, i.e., StyleGANv2, as little as possible. We find that only two modifications are absolutely necessary: 1) a multiplane image style generator branch which produces a set of alpha maps conditioned on their depth; 2) a pose-conditioned discriminator. We refer to the generated output as a ‘generative multiplane image’ (GMPI) and emphasize that its renderings are not only high-quality but also guaranteed to be view-consistent, which makes GMPIs different from many prior works. Importantly, the number of alpha maps can be dynamically adjusted and can differ between training and inference, alleviating memory concerns and enabling fast training of GMPIs in less than half a day at a resolution of 1024 2. Our findings are consistent across three challenging and common high-resolution datasets, including FFHQ, AFHQv2 and MetFaces.

AB - What is really needed to make an existing 2D GAN 3D-aware? To answer this question, we modify a classical GAN, i.e., StyleGANv2, as little as possible. We find that only two modifications are absolutely necessary: 1) a multiplane image style generator branch which produces a set of alpha maps conditioned on their depth; 2) a pose-conditioned discriminator. We refer to the generated output as a ‘generative multiplane image’ (GMPI) and emphasize that its renderings are not only high-quality but also guaranteed to be view-consistent, which makes GMPIs different from many prior works. Importantly, the number of alpha maps can be dynamically adjusted and can differ between training and inference, alleviating memory concerns and enabling fast training of GMPIs in less than half a day at a resolution of 1024 2. Our findings are consistent across three challenging and common high-resolution datasets, including FFHQ, AFHQv2 and MetFaces.

KW - 3D-aware generation

KW - GANs

KW - Multiplane images

UR - http://www.scopus.com/inward/record.url?scp=85144554359&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85144554359&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-20065-6_2

DO - 10.1007/978-3-031-20065-6_2

M3 - Conference contribution

AN - SCOPUS:85144554359

SN - 9783031200649

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 18

EP - 35

BT - Computer Vision – ECCV 2022 - 17th European Conference, Proceedings

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

A2 - Farinella, Giovanni Maria

A2 - Hassner, Tal

PB - Springer

Y2 - 23 October 2022 through 27 October 2022

ER -