Probabilistic multimedia objects (multijects): a novel approach to video indexing and retrieval in multimedia systems

This paper proposes a novel scheme for bridging the gap between low level media features and high level semantics using a probabilistic framework. We propose a framework, in which scenes can be indexed at a semantic level. The fundamental components of the framework are sites, objects and events. Detection of presence of an instance of one of these influences the probability of the presence of instances within other classes. Detection of instances is done using probabilistic multimedia objects: Multijects. Indexing using Multijects can handle queries posed at semantic level. Multijects are built in a Markovian framework. Two ways of building the Multijects from low level features fusing features from multiple modalities are presented. A probabilistic framework is also envisioned to encode the higher level relationship between Multijects, which enhances or reduces the probabilities of concurrent existence of various Multijects. An actual implementation is presented by developing Multijects representing higher level concept of 'Explosion' and 'Waterfall'. The models are evaluated by using the Multijects to detect explosions and waterfalls in movies. Results reveal, that the Multijects detect the aforementioned events with greater accuracy and are able to segment the video into scenes which have explosions and waterfalls.