PhotoStylist: Altering the Style of Photos Based on the Connotations of Texts

Siamul Karim Khan, Daniel (Yue) Zhang, Ziyi Kou, Yang Zhang, Dong Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The need to modify a photo to reflect the connotations of a text can arise due to multifarious reasons (e.g., a musician might modify a photo in the album cover to better reflect the connotations in her song lyrics). An interesting observation is that different styles of photos convey different feelings. In this paper, we propose the PhotoStylist scheme to effectively modify the style of an input photo to represent the connotations in an input text. Existing methods that aim to transfer emotions into photos rely on an emotion class being provided as input and modify the overall color of photos based on the input emotion class, generating unrealistic colors for many objects in the image. To address these limitations, we design PhotoStylist, a novel deep-learning-based approach, to alter the individual style of each object in the photo in a way that the connotations of the input text are naturally and effectively embedded into the modified photos. Evaluation results on the Amazon Mechanical Turk (MTurk) show that our scheme can achieve output photos significantly closer to the connotations of the input text than the output photos from the state-of-the-art baselines.

Original languageEnglish (US)
Title of host publicationAdvances in Knowledge Discovery and Data Mining - 25th Pacific-Asia Conference, PAKDD 2021, Proceedings
EditorsKamal Karlapalem, Hong Cheng, Naren Ramakrishnan, R. K. Agrawal, P. Krishna Reddy, Jaideep Srivastava, Tanmoy Chakraborty
PublisherSpringer
Pages642-654
Number of pages13
ISBN (Print)9783030757618
DOIs
StatePublished - 2021
Externally publishedYes
Event25th Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2021 - Virtual, Online
Duration: May 11 2021May 14 2021

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12712 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference25th Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2021
CityVirtual, Online
Period5/11/215/14/21

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'PhotoStylist: Altering the Style of Photos Based on the Connotations of Texts'. Together they form a unique fingerprint.

Cite this