Structural consistency and controllability for diverse colorization

Safa Messaoud; David Forsyth; Alexander G. Schwing

doi:10.1007/978-3-030-01231-1_37

Structural consistency and controllability for diverse colorization

Safa Messaoud, David Forsyth, Alexander G. Schwing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Colorizing a given gray-level image is an important task in the media and advertising industry. Due to the ambiguity inherent to colorization (many shades are often plausible), recent approaches started to explicitly model diversity. However, one of the most obvious artifacts, structural inconsistency, is rarely considered by existing methods which predict chrominance independently for every pixel. To address this issue, we develop a conditional random field based variational auto-encoder formulation which is able to achieve diversity while taking into account structural consistency. Moreover, we introduce a controllability mechanism that can incorporate external constraints from diverse sources including a user interface. Compared to existing baselines, we demonstrate that our method obtains more diverse and globally consistent colorizations on the LFW, LSUN-Church and ILSVRC-2015 datasets.

Original language	English (US)
Title of host publication	Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings
Editors	Martial Hebert, Yair Weiss, Vittorio Ferrari, Cristian Sminchisescu
Publisher	Springer-Verlag Berlin Heidelberg
Pages	603-619
Number of pages	17
ISBN (Print)	9783030012304
DOIs	https://doi.org/10.1007/978-3-030-01231-1_37
State	Published - 2018
Event	15th European Conference on Computer Vision, ECCV 2018 - Munich, Germany Duration: Sep 8 2018 → Sep 14 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11210 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	15th European Conference on Computer Vision, ECCV 2018
Country	Germany
City	Munich
Period	9/8/18 → 9/14/18

Keywords

Colorization
Gaussian-Conditional Random Field
VAE

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-030-01231-1_37

Fingerprint Dive into the research topics of 'Structural consistency and controllability for diverse colorization'. Together they form a unique fingerprint.

Cite this

Messaoud, S., Forsyth, D., & Schwing, A. G. (2018). Structural consistency and controllability for diverse colorization. In M. Hebert, Y. Weiss, V. Ferrari, & C. Sminchisescu (Eds.), Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings (pp. 603-619). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11210 LNCS). Springer-Verlag Berlin Heidelberg. https://doi.org/10.1007/978-3-030-01231-1_37

Structural consistency and controllability for diverse colorization. / Messaoud, Safa; Forsyth, David ; Schwing, Alexander G.

Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. ed. / Martial Hebert; Yair Weiss; Vittorio Ferrari; Cristian Sminchisescu. Springer-Verlag Berlin Heidelberg, 2018. p. 603-619 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11210 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Messaoud, S, Forsyth, D & Schwing, AG 2018, Structural consistency and controllability for diverse colorization. in M Hebert, Y Weiss, V Ferrari & C Sminchisescu (eds), Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11210 LNCS, Springer-Verlag Berlin Heidelberg, pp. 603-619, 15th European Conference on Computer Vision, ECCV 2018, Munich, Germany, 9/8/18. https://doi.org/10.1007/978-3-030-01231-1_37

Messaoud S, Forsyth D , Schwing AG. Structural consistency and controllability for diverse colorization. In Hebert M, Weiss Y, Ferrari V, Sminchisescu C, editors, Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. Springer-Verlag Berlin Heidelberg. 2018. p. 603-619. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-01231-1_37

Messaoud, Safa ; Forsyth, David ; Schwing, Alexander G. / Structural consistency and controllability for diverse colorization. Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. editor / Martial Hebert ; Yair Weiss ; Vittorio Ferrari ; Cristian Sminchisescu. Springer-Verlag Berlin Heidelberg, 2018. pp. 603-619 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{5fb40f3cf0934d49af0fe0050c891ffc,

title = "Structural consistency and controllability for diverse colorization",

abstract = "Colorizing a given gray-level image is an important task in the media and advertising industry. Due to the ambiguity inherent to colorization (many shades are often plausible), recent approaches started to explicitly model diversity. However, one of the most obvious artifacts, structural inconsistency, is rarely considered by existing methods which predict chrominance independently for every pixel. To address this issue, we develop a conditional random field based variational auto-encoder formulation which is able to achieve diversity while taking into account structural consistency. Moreover, we introduce a controllability mechanism that can incorporate external constraints from diverse sources including a user interface. Compared to existing baselines, we demonstrate that our method obtains more diverse and globally consistent colorizations on the LFW, LSUN-Church and ILSVRC-2015 datasets.",

keywords = "Colorization, Gaussian-Conditional Random Field, VAE",

author = "Safa Messaoud and David Forsyth and Schwing, {Alexander G.}",

note = "Funding Information: Acknowledgments. This material is based upon work supported in part by the National Science Foundation under Grant No. 1718221, Samsung, and 3M. We thank NVIDIA for providing the GPUs used for this research.; 15th European Conference on Computer Vision, ECCV 2018 ; Conference date: 08-09-2018 Through 14-09-2018",

year = "2018",

doi = "10.1007/978-3-030-01231-1_37",

language = "English (US)",

isbn = "9783030012304",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer-Verlag Berlin Heidelberg",

pages = "603--619",

editor = "Martial Hebert and Yair Weiss and Vittorio Ferrari and Cristian Sminchisescu",

booktitle = "Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings",

}

TY - GEN

T1 - Structural consistency and controllability for diverse colorization

AU - Messaoud, Safa

AU - Forsyth, David

AU - Schwing, Alexander G.

N1 - Funding Information: Acknowledgments. This material is based upon work supported in part by the National Science Foundation under Grant No. 1718221, Samsung, and 3M. We thank NVIDIA for providing the GPUs used for this research.

PY - 2018

Y1 - 2018

N2 - Colorizing a given gray-level image is an important task in the media and advertising industry. Due to the ambiguity inherent to colorization (many shades are often plausible), recent approaches started to explicitly model diversity. However, one of the most obvious artifacts, structural inconsistency, is rarely considered by existing methods which predict chrominance independently for every pixel. To address this issue, we develop a conditional random field based variational auto-encoder formulation which is able to achieve diversity while taking into account structural consistency. Moreover, we introduce a controllability mechanism that can incorporate external constraints from diverse sources including a user interface. Compared to existing baselines, we demonstrate that our method obtains more diverse and globally consistent colorizations on the LFW, LSUN-Church and ILSVRC-2015 datasets.

AB - Colorizing a given gray-level image is an important task in the media and advertising industry. Due to the ambiguity inherent to colorization (many shades are often plausible), recent approaches started to explicitly model diversity. However, one of the most obvious artifacts, structural inconsistency, is rarely considered by existing methods which predict chrominance independently for every pixel. To address this issue, we develop a conditional random field based variational auto-encoder formulation which is able to achieve diversity while taking into account structural consistency. Moreover, we introduce a controllability mechanism that can incorporate external constraints from diverse sources including a user interface. Compared to existing baselines, we demonstrate that our method obtains more diverse and globally consistent colorizations on the LFW, LSUN-Church and ILSVRC-2015 datasets.

KW - Colorization

KW - Gaussian-Conditional Random Field

KW - VAE

UR - http://www.scopus.com/inward/record.url?scp=85055088417&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055088417&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01231-1_37

DO - 10.1007/978-3-030-01231-1_37

M3 - Conference contribution

AN - SCOPUS:85055088417

SN - 9783030012304

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 603

EP - 619

BT - Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings

A2 - Hebert, Martial

A2 - Weiss, Yair

A2 - Ferrari, Vittorio

A2 - Sminchisescu, Cristian

PB - Springer-Verlag Berlin Heidelberg

T2 - 15th European Conference on Computer Vision, ECCV 2018

Y2 - 8 September 2018 through 14 September 2018

ER -