A bayesian approach to discovering truth from conflicting sources for data integration

Bo Zhao, Benjamin I.P. Rubinstein, Jim Gemmell, Jiawei Han

Research output: Contribution to journalArticle

Abstract

In practical data integration systems, it is common for the data sources being integrated to provide conflicting information about the same entity. Consequently, a major challenge for data integration is to derive the most complete and accurate integrated records from diverse and sometimes conflicting sources. We term this challenge the truth finding problem. We observe that some sources are generally more reliable than others, and therefore a good model of source quality is the key to solving the truth finding problem. In this work, we propose a probabilistic graphical model that can automatically infer true records and source quality without any supervision. In contrast to previous methods, our principled approach leverages a generative process of two types of errors (false positive and false negative) by modeling two different aspects of source quality. In so doing, ours is also the first approach designed to merge multi-valued attribute types. Our method is scalable, due to an efficient sampling-based inference algorithm that needs very few iterations in practice and enjoys linear time complexity, with an even faster incremental variant. Experiments on two real world datasets show that our new method outperforms existing state-of the-art approaches to the truth finding problem.

Original languageEnglish (US)
Pages (from-to)550-561
Number of pages12
JournalProceedings of the VLDB Endowment
Volume5
Issue number6
DOIs
StatePublished - Feb 2012

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Computer Science(all)

Fingerprint Dive into the research topics of 'A bayesian approach to discovering truth from conflicting sources for data integration'. Together they form a unique fingerprint.

  • Cite this