An important task of modeling complex social behaviors is to observe and understand individual/group beliefs and attitudes. These beliefs, however, are not stable and may change multiple times as people gain additional information/perceptions from various external sources, which in turn, may affect their subsequent behavior. To detect and track such influential sources is challenging, as they are often invisible to the public due to a variety of reasons - private communications, what one randomly reads or hears, and implicit social hierarchies, to name a few. Existing approaches usually focus on detecting distribution variations in behavioral data, but overlook the underlying reason for the variation. In this paper, we present a novel approach that models the belief change over time caused by hidden sources, taking into consideration the evolution of their impact patterns. Specifically, a finite fusion model is defined to encode the latent parameters that characterize the distribution of the hidden sources and their impact weights. We compare our work with two general mixture models, namely Gaussian Mixture Model and Mixture Bayesian Network. Experiments on both synthetic data and a real-world scenario show that our approach is effective on detecting and tracking hidden sources and outperforms existing methods.