Sem2NeRF: converting single-view semantic masks to neural radiance fields

Yuedong Chen; Qianyi Wu; Chuanxia Zheng; Tat Jen Cham; Jianfei Cai

doi:10.1007/978-3-031-19781-9_42

Sem2NeRF: converting single-view semantic masks to neural radiance fields

Yuedong Chen, Qianyi Wu, Chuanxia Zheng, Tat Jen Cham, Jianfei Cai

Department of Data Science & AI

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

11 Citations (Scopus)

Abstract

Image translation and manipulation have gain increasing attention along with the rapid development of deep generative models. Although existing approaches have brought impressive results, they mainly operated in 2D space. In light of recent advances in NeRF-based 3D-aware generative models, we introduce a new task, Semantic-to-NeRF translation, that aims to reconstruct a 3D scene modelled by NeRF, conditioned on one single-view semantic mask as input. To kick-off this novel task, we propose the Sem2NeRF framework. In particular, Sem2NeRF addresses the highly challenging task by encoding the semantic mask into the latent code that controls the 3D scene representation of a pre-trained decoder. To further improve the accuracy of the mapping, we integrate a new region-aware learning strategy into the design of both the encoder and the decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate that it outperforms several strong baselines on two benchmark datasets. Code and video are available at https://donydchen.github.io/sem2nerf/.

Original language	English
Title of host publication	Computer Vision – ECCV 2022 - 17th European Conference Tel Aviv, Israel, October 23–27, 2022 Proceedings, Part XIV
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
Place of Publication	Cham Switzerland
Publisher	Springer
Pages	730-748
Number of pages	19
ISBN (Electronic)	9783031197819
ISBN (Print)	9783031197802
DOIs	https://doi.org/10.1007/978-3-031-19781-9_42
Publication status	Published - 2022
Event	European Conference on Computer Vision 2022 - Tel Aviv, Israel Duration: 23 Oct 2022 → 27 Oct 2022 Conference number: 17th https://link.springer.com/book/10.1007/978-3-031-19830-4 (Proceedings) https://eccv2022.ecva.net (Website)

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	13674
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Computer Vision 2022
Abbreviated title	ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	23/10/22 → 27/10/22
Internet address	https://link.springer.com/book/10.1007/978-3-031-19830-4 (Proceedings) https://eccv2022.ecva.net (Website)

Keywords

3D deep learning
Conditional generative model
Image-to-image translation
NeRF-based generation
Neural radiance fields

Access to Document

10.1007/978-3-031-19781-9_42

Cite this

Chen, Y., Wu, Q., Zheng, C., Cham, T. J., & Cai, J. (2022). Sem2NeRF: converting single-view semantic masks to neural radiance fields. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022 - 17th European Conference Tel Aviv, Israel, October 23–27, 2022 Proceedings, Part XIV (pp. 730-748). (Lecture Notes in Computer Science; Vol. 13674). Springer. https://doi.org/10.1007/978-3-031-19781-9_42

Chen, Yuedong ; Wu, Qianyi ; Zheng, Chuanxia et al. / Sem2NeRF : converting single-view semantic masks to neural radiance fields. Computer Vision – ECCV 2022 - 17th European Conference Tel Aviv, Israel, October 23–27, 2022 Proceedings, Part XIV. editor / Shai Avidan ; Gabriel Brostow ; Moustapha Cissé ; Giovanni Maria Farinella ; Tal Hassner. Cham Switzerland : Springer, 2022. pp. 730-748 (Lecture Notes in Computer Science).

@inproceedings{f7a954fc85ac4f0b86c2e93ee767e694,

title = "Sem2NeRF: converting single-view semantic masks to neural radiance fields",

abstract = "Image translation and manipulation have gain increasing attention along with the rapid development of deep generative models. Although existing approaches have brought impressive results, they mainly operated in 2D space. In light of recent advances in NeRF-based 3D-aware generative models, we introduce a new task, Semantic-to-NeRF translation, that aims to reconstruct a 3D scene modelled by NeRF, conditioned on one single-view semantic mask as input. To kick-off this novel task, we propose the Sem2NeRF framework. In particular, Sem2NeRF addresses the highly challenging task by encoding the semantic mask into the latent code that controls the 3D scene representation of a pre-trained decoder. To further improve the accuracy of the mapping, we integrate a new region-aware learning strategy into the design of both the encoder and the decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate that it outperforms several strong baselines on two benchmark datasets. Code and video are available at https://donydchen.github.io/sem2nerf/.",

keywords = "3D deep learning, Conditional generative model, Image-to-image translation, NeRF-based generation, Neural radiance fields",

author = "Yuedong Chen and Qianyi Wu and Chuanxia Zheng and Cham, {Tat Jen} and Jianfei Cai",

note = "Funding Information: This research is partially supported by CRC Building 4.0 Project #44. Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; European Conference on Computer Vision 2022, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2022",

doi = "10.1007/978-3-031-19781-9_42",

language = "English",

isbn = "9783031197802",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "730--748",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e} and Farinella, {Giovanni Maria} and Tal Hassner",

booktitle = "Computer Vision – ECCV 2022 - 17th European Conference Tel Aviv, Israel, October 23–27, 2022 Proceedings, Part XIV",

url = "https://link.springer.com/book/10.1007/978-3-031-19830-4, https://eccv2022.ecva.net",

}

Chen, Y, Wu, Q, Zheng, C, Cham, TJ & Cai, J 2022, Sem2NeRF: converting single-view semantic masks to neural radiance fields. in S Avidan, G Brostow, M Cissé, GM Farinella & T Hassner (eds), Computer Vision – ECCV 2022 - 17th European Conference Tel Aviv, Israel, October 23–27, 2022 Proceedings, Part XIV. Lecture Notes in Computer Science, vol. 13674, Springer, Cham Switzerland , pp. 730-748, European Conference on Computer Vision 2022, Tel Aviv, Israel, 23/10/22. https://doi.org/10.1007/978-3-031-19781-9_42

Sem2NeRF: converting single-view semantic masks to neural radiance fields. / Chen, Yuedong; Wu, Qianyi; Zheng, Chuanxia et al.
Computer Vision – ECCV 2022 - 17th European Conference Tel Aviv, Israel, October 23–27, 2022 Proceedings, Part XIV. ed. / Shai Avidan; Gabriel Brostow; Moustapha Cissé; Giovanni Maria Farinella; Tal Hassner. Cham Switzerland : Springer, 2022. p. 730-748 (Lecture Notes in Computer Science; Vol. 13674).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Sem2NeRF

T2 - European Conference on Computer Vision 2022

AU - Chen, Yuedong

AU - Wu, Qianyi

AU - Zheng, Chuanxia

AU - Cham, Tat Jen

AU - Cai, Jianfei

N1 - Conference code: 17th

PY - 2022

Y1 - 2022

N2 - Image translation and manipulation have gain increasing attention along with the rapid development of deep generative models. Although existing approaches have brought impressive results, they mainly operated in 2D space. In light of recent advances in NeRF-based 3D-aware generative models, we introduce a new task, Semantic-to-NeRF translation, that aims to reconstruct a 3D scene modelled by NeRF, conditioned on one single-view semantic mask as input. To kick-off this novel task, we propose the Sem2NeRF framework. In particular, Sem2NeRF addresses the highly challenging task by encoding the semantic mask into the latent code that controls the 3D scene representation of a pre-trained decoder. To further improve the accuracy of the mapping, we integrate a new region-aware learning strategy into the design of both the encoder and the decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate that it outperforms several strong baselines on two benchmark datasets. Code and video are available at https://donydchen.github.io/sem2nerf/.

AB - Image translation and manipulation have gain increasing attention along with the rapid development of deep generative models. Although existing approaches have brought impressive results, they mainly operated in 2D space. In light of recent advances in NeRF-based 3D-aware generative models, we introduce a new task, Semantic-to-NeRF translation, that aims to reconstruct a 3D scene modelled by NeRF, conditioned on one single-view semantic mask as input. To kick-off this novel task, we propose the Sem2NeRF framework. In particular, Sem2NeRF addresses the highly challenging task by encoding the semantic mask into the latent code that controls the 3D scene representation of a pre-trained decoder. To further improve the accuracy of the mapping, we integrate a new region-aware learning strategy into the design of both the encoder and the decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate that it outperforms several strong baselines on two benchmark datasets. Code and video are available at https://donydchen.github.io/sem2nerf/.

KW - 3D deep learning

KW - Conditional generative model

KW - Image-to-image translation

KW - NeRF-based generation

KW - Neural radiance fields

UR - http://www.scopus.com/inward/record.url?scp=85142676989&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-19781-9_42

DO - 10.1007/978-3-031-19781-9_42

M3 - Conference Paper

AN - SCOPUS:85142676989

SN - 9783031197802

T3 - Lecture Notes in Computer Science

SP - 730

EP - 748

BT - Computer Vision – ECCV 2022 - 17th European Conference Tel Aviv, Israel, October 23–27, 2022 Proceedings, Part XIV

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

A2 - Farinella, Giovanni Maria

A2 - Hassner, Tal

PB - Springer

CY - Cham Switzerland

Y2 - 23 October 2022 through 27 October 2022

ER -

Chen Y, Wu Q, Zheng C, Cham TJ, Cai J. Sem2NeRF: converting single-view semantic masks to neural radiance fields. In Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors, Computer Vision – ECCV 2022 - 17th European Conference Tel Aviv, Israel, October 23–27, 2022 Proceedings, Part XIV. Cham Switzerland : Springer. 2022. p. 730-748. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-19781-9_42