TY - JOUR
T1 - Structure-Texture Image Decomposition Using Deep Variational Priors
AU - Kim, Youngjung
AU - Ham, Bumsub
AU - Do, Minh N.
AU - Sohn, Kwanghoon
N1 - Funding Information:
Manuscript received June 7, 2018; revised November 19, 2018 and December 16, 2018; accepted December 17, 2018. Date of publication December 24, 2018; date of current version March 21, 2019. This work was supported by the Multi-Ministry Collaborative R&D Program (R&D program for complex cognitive technology) through the National Research Foundation of Korea (NRF) funded by MSIT, MOTIE, and KNPA, under Grant NRF-2018M3E3A1057289. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Peter Tay. (Corresponding author: Kwanghoon Sohn.) Y. Kim is with the Institute of Defense Advanced Technology Research, Agency for Defense Development, Daejeon 340-60, South Korea (e-mail: read12300@add.re.kr).
Publisher Copyright:
© 1992-2012 IEEE.
PY - 2019/6
Y1 - 2019/6
N2 - Most variational formulations for structure-Texture image decomposition force the structure images to have small norm in some functional spaces and to share a common notion of edges, i.e., large-gradients or large-intensity differences. However, such a definition makes it difficult to distinguish structure edges from oscillations that have fine spatial scale but high contrast. In this paper, we introduce a new model by learning deep variational priors for structure images without explicit training data. An alternating direction method of a multiplier algorithm and its modular structure are adopted to plug deep variational priors into an iterative smoothing process. The central observations are that convolution neural networks (CNNs) can replace the total variation prior, and are indeed powerful to capture the natures of structure and texture. We show that our learned priors using CNNs successfully differentiate high-Amplitude details from structure edges, and avoid halo artifacts. Different from previous data-driven smoothing schemes, our formulation provides another degree of freedom to produce continuous smoothing effects. Experimental results demonstrate the effectiveness of our approach on various computational photography and image processing applications, including texture removal, detail manipulation, HDR tone-mapping, and non-photorealistic abstraction.
AB - Most variational formulations for structure-Texture image decomposition force the structure images to have small norm in some functional spaces and to share a common notion of edges, i.e., large-gradients or large-intensity differences. However, such a definition makes it difficult to distinguish structure edges from oscillations that have fine spatial scale but high contrast. In this paper, we introduce a new model by learning deep variational priors for structure images without explicit training data. An alternating direction method of a multiplier algorithm and its modular structure are adopted to plug deep variational priors into an iterative smoothing process. The central observations are that convolution neural networks (CNNs) can replace the total variation prior, and are indeed powerful to capture the natures of structure and texture. We show that our learned priors using CNNs successfully differentiate high-Amplitude details from structure edges, and avoid halo artifacts. Different from previous data-driven smoothing schemes, our formulation provides another degree of freedom to produce continuous smoothing effects. Experimental results demonstrate the effectiveness of our approach on various computational photography and image processing applications, including texture removal, detail manipulation, HDR tone-mapping, and non-photorealistic abstraction.
KW - Structure-Texture image decomposition
KW - adaptive neighborhood filtering
KW - alternating direction method of multiplier algorithm
KW - texture filtering
KW - total variation
UR - http://www.scopus.com/inward/record.url?scp=85058993180&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85058993180&partnerID=8YFLogxK
U2 - 10.1109/TIP.2018.2889531
DO - 10.1109/TIP.2018.2889531
M3 - Article
C2 - 30582541
AN - SCOPUS:85058993180
SN - 1057-7149
VL - 28
SP - 2692
EP - 2704
JO - IEEE Transactions on Image Processing
JF - IEEE Transactions on Image Processing
IS - 6
M1 - 8586974
ER -