Improving event extraction via multimodal integration

Tongtao Zhang; Spencer Whitehead; Hanwang Zhang; Hongzhi Li; Joseph Ellis; Lifu Huang; Wei Liu; Heng Ji; Shih Fu Chang

doi:10.1145/3123266.3123294

Improving event extraction via multimodal integration

Tongtao Zhang, Spencer Whitehead, Hanwang Zhang, Hongzhi Li, Joseph Ellis, Lifu Huang, Wei Liu, Heng Ji, Shih Fu Chang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we focus on improving Event Extraction (EE) by incorporating visual knowledge with words and phrases from text documents. We first discover visual patterns from large-scale textimage pairs in a weakly-supervised manner and then propose a multimodal event extraction algorithm where the event extractor is jointly trained with textual features and visual patterns. Extensive experimental results on benchmark data sets demonstrate that the proposed multimodal EE method can achieve significantly better performance on event extraction: absolute 7.1% F-score gain on event trigger labeling and 8.5% F-score gain on event argument labeling.

Original language	English (US)
Title of host publication	MM 2017 - Proceedings of the 2017 ACM Multimedia Conference
Publisher	Association for Computing Machinery, Inc
Pages	270-278
Number of pages	9
ISBN (Electronic)	9781450349062
DOIs	https://doi.org/10.1145/3123266.3123294
State	Published - Oct 23 2017
Externally published	Yes
Event	25th ACM International Conference on Multimedia, MM 2017 - Mountain View, United States Duration: Oct 23 2017 → Oct 27 2017

Publication series

Name	MM 2017 - Proceedings of the 2017 ACM Multimedia Conference

Other

Other	25th ACM International Conference on Multimedia, MM 2017
Country/Territory	United States
City	Mountain View
Period	10/23/17 → 10/27/17

Keywords

Event extraction
Multimodal approach
Natural language processing
Visual pattern discovery

ASJC Scopus subject areas

Computer Graphics and Computer-Aided Design
Media Technology
Computer Vision and Pattern Recognition
Software

Online availability

10.1145/3123266.3123294

Library availability

Discover UIUC Full Text

Cite this

Zhang, T., Whitehead, S., Zhang, H., Li, H., Ellis, J., Huang, L., Liu, W., Ji, H., & Chang, S. F. (2017). Improving event extraction via multimodal integration. In MM 2017 - Proceedings of the 2017 ACM Multimedia Conference (pp. 270-278). (MM 2017 - Proceedings of the 2017 ACM Multimedia Conference). Association for Computing Machinery, Inc. https://doi.org/10.1145/3123266.3123294

Improving event extraction via multimodal integration. / Zhang, Tongtao; Whitehead, Spencer; Zhang, Hanwang et al.
MM 2017 - Proceedings of the 2017 ACM Multimedia Conference. Association for Computing Machinery, Inc, 2017. p. 270-278 (MM 2017 - Proceedings of the 2017 ACM Multimedia Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Zhang, T, Whitehead, S, Zhang, H, Li, H, Ellis, J, Huang, L, Liu, W, Ji, H & Chang, SF 2017, Improving event extraction via multimodal integration. in MM 2017 - Proceedings of the 2017 ACM Multimedia Conference. MM 2017 - Proceedings of the 2017 ACM Multimedia Conference, Association for Computing Machinery, Inc, pp. 270-278, 25th ACM International Conference on Multimedia, MM 2017, Mountain View, United States, 10/23/17. https://doi.org/10.1145/3123266.3123294

@inproceedings{2d860429d2db4b8691b4c61b82dee95f,

title = "Improving event extraction via multimodal integration",

abstract = "In this paper, we focus on improving Event Extraction (EE) by incorporating visual knowledge with words and phrases from text documents. We first discover visual patterns from large-scale textimage pairs in a weakly-supervised manner and then propose a multimodal event extraction algorithm where the event extractor is jointly trained with textual features and visual patterns. Extensive experimental results on benchmark data sets demonstrate that the proposed multimodal EE method can achieve significantly better performance on event extraction: absolute 7.1% F-score gain on event trigger labeling and 8.5% F-score gain on event argument labeling.",

keywords = "Event extraction, Multimodal approach, Natural language processing, Visual pattern discovery",

author = "Tongtao Zhang and Spencer Whitehead and Hanwang Zhang and Hongzhi Li and Joseph Ellis and Lifu Huang and Wei Liu and Heng Ji and Chang, {Shih Fu}",

note = "Funding Information: This work was supported by the U.S. DARPA Multimedia Seedling grant, DARPA DEFT No. FA8750-13-2-0041 and FA8750-13-2-0045. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on. Publisher Copyright: {\textcopyright} 2017 Association for Computing Machinery.; 25th ACM International Conference on Multimedia, MM 2017 ; Conference date: 23-10-2017 Through 27-10-2017",

year = "2017",

month = oct,

day = "23",

doi = "10.1145/3123266.3123294",

language = "English (US)",

series = "MM 2017 - Proceedings of the 2017 ACM Multimedia Conference",

publisher = "Association for Computing Machinery, Inc",

pages = "270--278",

booktitle = "MM 2017 - Proceedings of the 2017 ACM Multimedia Conference",

}

TY - GEN

T1 - Improving event extraction via multimodal integration

AU - Zhang, Tongtao

AU - Whitehead, Spencer

AU - Zhang, Hanwang

AU - Li, Hongzhi

AU - Ellis, Joseph

AU - Huang, Lifu

AU - Liu, Wei

AU - Ji, Heng

AU - Chang, Shih Fu

N1 - Funding Information: This work was supported by the U.S. DARPA Multimedia Seedling grant, DARPA DEFT No. FA8750-13-2-0041 and FA8750-13-2-0045. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on. Publisher Copyright: © 2017 Association for Computing Machinery.

PY - 2017/10/23

Y1 - 2017/10/23

N2 - In this paper, we focus on improving Event Extraction (EE) by incorporating visual knowledge with words and phrases from text documents. We first discover visual patterns from large-scale textimage pairs in a weakly-supervised manner and then propose a multimodal event extraction algorithm where the event extractor is jointly trained with textual features and visual patterns. Extensive experimental results on benchmark data sets demonstrate that the proposed multimodal EE method can achieve significantly better performance on event extraction: absolute 7.1% F-score gain on event trigger labeling and 8.5% F-score gain on event argument labeling.

AB - In this paper, we focus on improving Event Extraction (EE) by incorporating visual knowledge with words and phrases from text documents. We first discover visual patterns from large-scale textimage pairs in a weakly-supervised manner and then propose a multimodal event extraction algorithm where the event extractor is jointly trained with textual features and visual patterns. Extensive experimental results on benchmark data sets demonstrate that the proposed multimodal EE method can achieve significantly better performance on event extraction: absolute 7.1% F-score gain on event trigger labeling and 8.5% F-score gain on event argument labeling.

KW - Event extraction

KW - Multimodal approach

KW - Natural language processing

KW - Visual pattern discovery

UR - http://www.scopus.com/inward/record.url?scp=85035213240&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85035213240&partnerID=8YFLogxK

U2 - 10.1145/3123266.3123294

DO - 10.1145/3123266.3123294

M3 - Conference contribution

AN - SCOPUS:85035213240

T3 - MM 2017 - Proceedings of the 2017 ACM Multimedia Conference

SP - 270

EP - 278

BT - MM 2017 - Proceedings of the 2017 ACM Multimedia Conference

PB - Association for Computing Machinery, Inc

T2 - 25th ACM International Conference on Multimedia, MM 2017

Y2 - 23 October 2017 through 27 October 2017

ER -

Improving event extraction via multimodal integration

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this