Structural consistency and controllability for diverse colorization

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Colorizing a given gray-level image is an important task in the media and advertising industry. Due to the ambiguity inherent to colorization (many shades are often plausible), recent approaches started to explicitly model diversity. However, one of the most obvious artifacts, structural inconsistency, is rarely considered by existing methods which predict chrominance independently for every pixel. To address this issue, we develop a conditional random field based variational auto-encoder formulation which is able to achieve diversity while taking into account structural consistency. Moreover, we introduce a controllability mechanism that can incorporate external constraints from diverse sources including a user interface. Compared to existing baselines, we demonstrate that our method obtains more diverse and globally consistent colorizations on the LFW, LSUN-Church and ILSVRC-2015 datasets.

Original languageEnglish (US)
Title of host publicationComputer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings
EditorsMartial Hebert, Yair Weiss, Vittorio Ferrari, Cristian Sminchisescu
PublisherSpringer-Verlag
Pages603-619
Number of pages17
ISBN (Print)9783030012304
DOIs
StatePublished - Jan 1 2018
Event15th European Conference on Computer Vision, ECCV 2018 - Munich, Germany
Duration: Sep 8 2018Sep 14 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11210 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other15th European Conference on Computer Vision, ECCV 2018
CountryGermany
CityMunich
Period9/8/189/14/18

Fingerprint

Religious buildings
Controllability
User interfaces
Marketing
Pixels
Conditional Random Fields
Encoder
Inconsistency
User Interface
Baseline
Industry
Pixel
Predict
Formulation
Demonstrate
Model
Advertising
Ambiguity

Keywords

  • Colorization
  • Gaussian-Conditional Random Field
  • VAE

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Messaoud, S., Forsyth, D. A., & Schwing, A. G. (2018). Structural consistency and controllability for diverse colorization. In M. Hebert, Y. Weiss, V. Ferrari, & C. Sminchisescu (Eds.), Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings (pp. 603-619). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11210 LNCS). Springer-Verlag. https://doi.org/10.1007/978-3-030-01231-1_37

Structural consistency and controllability for diverse colorization. / Messaoud, Safa; Forsyth, David Alexander; Schwing, Alexander Gerhard.

Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. ed. / Martial Hebert; Yair Weiss; Vittorio Ferrari; Cristian Sminchisescu. Springer-Verlag, 2018. p. 603-619 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11210 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Messaoud, S, Forsyth, DA & Schwing, AG 2018, Structural consistency and controllability for diverse colorization. in M Hebert, Y Weiss, V Ferrari & C Sminchisescu (eds), Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11210 LNCS, Springer-Verlag, pp. 603-619, 15th European Conference on Computer Vision, ECCV 2018, Munich, Germany, 9/8/18. https://doi.org/10.1007/978-3-030-01231-1_37
Messaoud S, Forsyth DA, Schwing AG. Structural consistency and controllability for diverse colorization. In Hebert M, Weiss Y, Ferrari V, Sminchisescu C, editors, Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. Springer-Verlag. 2018. p. 603-619. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-01231-1_37
Messaoud, Safa ; Forsyth, David Alexander ; Schwing, Alexander Gerhard. / Structural consistency and controllability for diverse colorization. Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. editor / Martial Hebert ; Yair Weiss ; Vittorio Ferrari ; Cristian Sminchisescu. Springer-Verlag, 2018. pp. 603-619 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{5fb40f3cf0934d49af0fe0050c891ffc,
title = "Structural consistency and controllability for diverse colorization",
abstract = "Colorizing a given gray-level image is an important task in the media and advertising industry. Due to the ambiguity inherent to colorization (many shades are often plausible), recent approaches started to explicitly model diversity. However, one of the most obvious artifacts, structural inconsistency, is rarely considered by existing methods which predict chrominance independently for every pixel. To address this issue, we develop a conditional random field based variational auto-encoder formulation which is able to achieve diversity while taking into account structural consistency. Moreover, we introduce a controllability mechanism that can incorporate external constraints from diverse sources including a user interface. Compared to existing baselines, we demonstrate that our method obtains more diverse and globally consistent colorizations on the LFW, LSUN-Church and ILSVRC-2015 datasets.",
keywords = "Colorization, Gaussian-Conditional Random Field, VAE",
author = "Safa Messaoud and Forsyth, {David Alexander} and Schwing, {Alexander Gerhard}",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-030-01231-1_37",
language = "English (US)",
isbn = "9783030012304",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer-Verlag",
pages = "603--619",
editor = "Martial Hebert and Yair Weiss and Vittorio Ferrari and Cristian Sminchisescu",
booktitle = "Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings",

}

TY - GEN

T1 - Structural consistency and controllability for diverse colorization

AU - Messaoud, Safa

AU - Forsyth, David Alexander

AU - Schwing, Alexander Gerhard

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Colorizing a given gray-level image is an important task in the media and advertising industry. Due to the ambiguity inherent to colorization (many shades are often plausible), recent approaches started to explicitly model diversity. However, one of the most obvious artifacts, structural inconsistency, is rarely considered by existing methods which predict chrominance independently for every pixel. To address this issue, we develop a conditional random field based variational auto-encoder formulation which is able to achieve diversity while taking into account structural consistency. Moreover, we introduce a controllability mechanism that can incorporate external constraints from diverse sources including a user interface. Compared to existing baselines, we demonstrate that our method obtains more diverse and globally consistent colorizations on the LFW, LSUN-Church and ILSVRC-2015 datasets.

AB - Colorizing a given gray-level image is an important task in the media and advertising industry. Due to the ambiguity inherent to colorization (many shades are often plausible), recent approaches started to explicitly model diversity. However, one of the most obvious artifacts, structural inconsistency, is rarely considered by existing methods which predict chrominance independently for every pixel. To address this issue, we develop a conditional random field based variational auto-encoder formulation which is able to achieve diversity while taking into account structural consistency. Moreover, we introduce a controllability mechanism that can incorporate external constraints from diverse sources including a user interface. Compared to existing baselines, we demonstrate that our method obtains more diverse and globally consistent colorizations on the LFW, LSUN-Church and ILSVRC-2015 datasets.

KW - Colorization

KW - Gaussian-Conditional Random Field

KW - VAE

UR - http://www.scopus.com/inward/record.url?scp=85055088417&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055088417&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01231-1_37

DO - 10.1007/978-3-030-01231-1_37

M3 - Conference contribution

AN - SCOPUS:85055088417

SN - 9783030012304

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 603

EP - 619

BT - Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings

A2 - Hebert, Martial

A2 - Weiss, Yair

A2 - Ferrari, Vittorio

A2 - Sminchisescu, Cristian

PB - Springer-Verlag

ER -