Social influence, the phenomenon that the actions of a user can induce her/his friends to behave in a similar way, plays a key role in many (online) social systems. For example, a company wants to market a new product through the effect of "word of mouth" in the social network. It wishes to find and convince a small number of influential users to adopt the product, and the goal is to trigger a large cascade of further adoptions. Fundamentally, we need to answer the following question: how to quantify the influence between two users in a large social network? To address this question, we propose a pairwise factor graph (PFG) model to model the social influence in social networks. An efficient algorithm is designed to learn the model and make inference. We further propose a dynamic factor graph (DFG) model to incorporate the time information. Experimental results on three different genres of data sets show that the proposed approaches can efficiently infer the dynamic social influence. The results are applied to the influence maximization problem, which aims to find a small subset of nodes (users) in a social network that could maximize the spread of influence. Experiments show that the proposed approach can facilitate the application.