Adding smarter systems instead of human annotators: Re-ranking for system combination

Suzanne Tamang, Heng Ji

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Using a Knowledge Base Population (KBP) slot filling task as a case study, we describe a re-ranking framework in the context of two experimental settings: (1) high transparency; a few pipelines share similar resources that can be used to provide the developer detailed intermediate answer results; (2) low transparency; many systems use diverse resources, and serve as black boxes, absent of any intermediate system results. In both settings, our results show that statistical re-ranking can effectively combine automated systems, achieving better performance than the best state-of-the-art individual system (6.6% absolute improvement in F-score) and alternative combination methods. Furthermore, to create labeled data for system development and assessment, information extraction tasks often require expensive human annotators to struggle with the vast amounts of information contained within a large scale corpus. In this paper, we demonstrate the impact of our learning-to-rank framework to combine output from multiple slot filling systems to populate entity-attribute facts in a knowledge base. We show that our approach can be used to create answer keys more efficiently and at a lower cost (63.5% reduction) than laborious human annotation.

Original languageEnglish (US)
Title of host publicationCIKM 2011 Glasgow
Subtitle of host publicationSMER'11 - Proceedings of the 1st International Workshop on Search and Mining Entity-Relationship Data
Pages3-8
Number of pages6
DOIs
StatePublished - Dec 15 2011
Externally publishedYes
Event1st International Workshop on Search and Mining Entity-Relationship Data, SMER'11, Held at 20th ACM Conference on Information and Knowledge Management, CIKM 2011 - Glasgow, United Kingdom
Duration: Oct 28 2011Oct 28 2011

Publication series

NameInternational Conference on Information and Knowledge Management, Proceedings

Conference

Conference1st International Workshop on Search and Mining Entity-Relationship Data, SMER'11, Held at 20th ACM Conference on Information and Knowledge Management, CIKM 2011
Country/TerritoryUnited Kingdom
CityGlasgow
Period10/28/1110/28/11

Keywords

  • information extraction
  • knowledge base population
  • supervised re-ranking
  • text analysis

ASJC Scopus subject areas

  • Decision Sciences(all)
  • Business, Management and Accounting(all)

Fingerprint

Dive into the research topics of 'Adding smarter systems instead of human annotators: Re-ranking for system combination'. Together they form a unique fingerprint.

Cite this