The framework of support vector machines (SVM) is becoming extremely popular in the field of statistical pattern classification. In this paper we investigate a technique which couples Kalman filter closely with the SVM. The problem of object tracking can be seen as a pattern recognition problem. However, because of the dynamics, this pattern might experience some changes over time. In order to keep track of the position of the pattern and to make out the desired pattern from the background, we must have some strong continuous time model. We propose an algorithm which combines the Markov property of the Kalman filter with the strong classification capability of SVM. The whole system has been tested on real life problems and we found that with this framework we could track a particular object even in a frame which contains identical objects. The results were compared to that of obtained by color blob tracking which showed the strength of the approach.