Vision-based construction activity analysis in long video sequences via hidden markov models: Experiments on earthmoving operations

This paper presents a new method for detailed activity analysis of dynamic construction resources in highly varying videos obtained from construction site cameras. Toward this goal, we propose a Hidden Markov Model (HMM) that is able to automatically discover and assign sequences of activities that are most discriminative for an observed construction operation. To do so, the algorithm leverages dense trajectory features from a detected dynamic resource (e.g., excavator) in a video. Using these dense trajectory features, we train a Gaussian mixture model (GMM) to estimate the probability density function of each activity with multiple one-versus-all support vector machine classifiers. The proposed HMM also models duration of each activity, and the transition between activities (e.g., "swing bucket loaded" after "load bucket" for earth-moving activities of an excavator). As a proof-of-concept, we train and test our HMM+GMM model on an unprecedented dataset of 10 real-world long video sequences of interacting pairs of excavators and dumptrucks. Our preliminary experimental results on long-sequence activity recognition in presence of noise, occlusions, and scene clutter demonstrate the effectiveness of our method.