Through the Eyes of a Poet: Classical Poetry Recommendation with Visual Input on Social Media

Daniel Yue Zhang, Bo Ni, Qiyu Zhi, Thomas Plummer, Qi Li, Hao Zheng, Qingkai Zeng, Yang Zhang, Dong Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the increasing popularity of portable devices with cameras (e.g., smartphones and tablets) and ubiquitous Internet connectivity, travelers can share their instant experience during the travel by posting photos they took to social media platforms. In this paper, we present a new image-driven poetry recommender system that takes a traveler’s photo as input and recommends classical poems that can enrich the photo with aesthetically pleasing quotes from the poems. Three critical challenges exist to solve this new problem: i) how to extract the implicit artistic conception embedded in both poems and images? ii) How to identify the salient objects in the image without knowing the creator’s intent? iii) How to accommodate the diverse user perceptions of the image and make a diversified poetry recommendation? The proposed iPoemRec system jointly addresses the above challenges by developing heterogeneous information network and neural embedding techniques. Evaluation results from real-world datasets and a user study demonstrate that our system can recommend highly relevant classical poems for a given photo and receive significantly higher user ratings compared to the state-of-the-art baselines.

Original languageEnglish (US)
Title of host publicationProceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2019
EditorsFrancesca Spezzano, Wei Chen, Xiaokui Xiao
PublisherAssociation for Computing Machinery, Inc
Pages333-340
Number of pages8
ISBN (Electronic)9781450368681
DOIs
StatePublished - Aug 27 2019
Externally publishedYes
Event11th IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2019 - Vancouver, Canada
Duration: Aug 27 2019Aug 30 2019

Publication series

NameProceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2019

Conference

Conference11th IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2019
Country/TerritoryCanada
CityVancouver
Period8/27/198/30/19

ASJC Scopus subject areas

  • Communication
  • Computer Networks and Communications
  • Information Systems and Management
  • Sociology and Political Science

Fingerprint

Dive into the research topics of 'Through the Eyes of a Poet: Classical Poetry Recommendation with Visual Input on Social Media'. Together they form a unique fingerprint.

Cite this