Automatic video annotation using multimodal Dirichlet process mixture model

In this paper we infer a multimodal Dirichlet process mixture model from video data, the mixture components in this model follow a Gaussian-Multinomial distribution. The multimodal Dirichlet process mixture model clusters freely available multimodal data in videos i.e., the combination of visual track and the corresponding keywords extracted from speech transcripts obtained from the audio track of videos, using the parameters of the model we build a predictive model that can output keyword annotations given video shots. In the multimodal Dirichlet process mixture model the keywords follow a multinomial distribution while the features used to represent the video shot follow a Gaussian distribution. We infer the multimodal Dirichlet process mixture model by collecting samples from the corresponding markov chain using a Blocked Gibbs sampling algorithm, and use the inferred parameters to predict video shot annotations that can be used to perform text based retrieval of shots. We compare the performance of our proposed model with other baseline models that use predicted annotations for retrieval.