This paper presents a probabilistic framework for segmenting and tracking multiple non rigid foreground objects for video surveillance, using a static monocular camera. The algorithm combines information in a probabilistic sense and poses the problem of matching the segmented foreground objects with blobs in the next frame as a non bipartite matching problem. To solve this problem, probability is calculated for each possible matching. Initialization of new objects is also treated in a probabilistic manner. The new framework is shown to be able to handle a greater set of difficult situations and to improve performance significantly.