Building topic models in a federated digital library through selective document exclusion

Miles Efron, Peter Organisciak, Katrina Fenlon

Research output: Contribution to journalArticlepeer-review

Abstract

Building topic models in federated digital collections presents numerous challenges due to metadata inconsistencies. The quality of topical metadata is difficult to ascertain and is interspersed with often irrelevant administrative metadata. In this study, we propose a way to improve topic modeling in large collections by identifying documents that convey only weak topical information. These documents are ignored when training topic models. Their topical associations are instead inferred model training. A method is outlined for identifying weakly topical documents by defining runs of similar documents in a collection. In preliminary evaluation using a corpus from the Institute of Museum and Library Services Digital Collections and Content aggregation, results show an increase in coherence among words in topics. In showing this, we demonstrate that it may be beneficial to induce topic models using less, higher-quality data.

Original languageEnglish (US)
JournalProceedings of the ASIST Annual Meeting
Volume48
DOIs
StatePublished - 2011

Keywords

  • Digital libraries
  • Latent topic models, document representation

ASJC Scopus subject areas

  • Information Systems
  • Library and Information Sciences

Fingerprint Dive into the research topics of 'Building topic models in a federated digital library through selective document exclusion'. Together they form a unique fingerprint.

Cite this