TY - JOUR
T1 - Conflicts to Harmony
T2 - A Framework for Resolving Conflicts in Heterogeneous Data by Truth Discovery
AU - Li, Yaliang
AU - Li, Qi
AU - Gao, Jing
AU - Su, Lu
AU - Zhao, Bo
AU - Fan, Wei
AU - Han, Jiawei
N1 - Publisher Copyright:
© 2016 IEEE.
Copyright:
Copyright 2016 Elsevier B.V., All rights reserved.
PY - 2016/8/1
Y1 - 2016/8/1
N2 - In many applications, one can obtain descriptions about the same objects or events from a variety of sources. As a result, this will inevitably lead to data or information conflicts. One important problem is to identify the true information (i.e., the truths) among conflicting sources of data. It is intuitive to trust reliable sources more when deriving the truths, but it is usually unknown which one is more reliable a priori. Moreover, each source possesses a variety of properties with different data types. An accurate estimation of source reliability has to be made by modeling multiple properties in a unified model. Existing conflict resolution work either does not conduct source reliability estimation, or models multiple properties separately. In this paper, we propose to resolve conflicts among multiple sources of heterogeneous data types. We model the problem using an optimization framework where truths and source reliability are defined as two sets of unknown variables. The objective is to minimize the overall weighted deviation between the truths and the multi-source observations where each source is weighted by its reliability. Different loss functions can be incorporated into this framework to recognize the characteristics of various data types, and efficient computation approaches are developed. The proposed framework is further adapted to deal with streaming data in an incremental fashion and large-scale data in MapReduce model. Experiments on real-world weather, stock, and flight data as well as simulated multi-source data demonstrate the advantage of jointly modeling different data types in the proposed framework.
AB - In many applications, one can obtain descriptions about the same objects or events from a variety of sources. As a result, this will inevitably lead to data or information conflicts. One important problem is to identify the true information (i.e., the truths) among conflicting sources of data. It is intuitive to trust reliable sources more when deriving the truths, but it is usually unknown which one is more reliable a priori. Moreover, each source possesses a variety of properties with different data types. An accurate estimation of source reliability has to be made by modeling multiple properties in a unified model. Existing conflict resolution work either does not conduct source reliability estimation, or models multiple properties separately. In this paper, we propose to resolve conflicts among multiple sources of heterogeneous data types. We model the problem using an optimization framework where truths and source reliability are defined as two sets of unknown variables. The objective is to minimize the overall weighted deviation between the truths and the multi-source observations where each source is weighted by its reliability. Different loss functions can be incorporated into this framework to recognize the characteristics of various data types, and efficient computation approaches are developed. The proposed framework is further adapted to deal with streaming data in an incremental fashion and large-scale data in MapReduce model. Experiments on real-world weather, stock, and flight data as well as simulated multi-source data demonstrate the advantage of jointly modeling different data types in the proposed framework.
KW - Data fusion
KW - heterogeneous data
KW - truth discovery
UR - http://www.scopus.com/inward/record.url?scp=84978127645&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84978127645&partnerID=8YFLogxK
U2 - 10.1109/TKDE.2016.2559481
DO - 10.1109/TKDE.2016.2559481
M3 - Article
AN - SCOPUS:84978127645
VL - 28
SP - 1986
EP - 1999
JO - IEEE Transactions on Knowledge and Data Engineering
JF - IEEE Transactions on Knowledge and Data Engineering
SN - 1041-4347
IS - 8
M1 - 7460925
ER -