This paper presents an automated and real-time algorithm for recognition and 2D tracking of construction workers and equipment from site video streams. In recent years, several research studies have proposed semi-automated vision-based methods for tracking of construction workers and equipment. Nonetheless, there is still a need for automated initial recognition and real-time tracking of these resources in video streams. To address these limitations, a new algorithm based on histograms of Oriented Gradients (HOG) is proposed. The method uses HOG features with a new multiple binary Support Vector Machine (SVM) classifier to automatically recognize and differentiate workers and equipment. These resources are tracked in real-time using a new GPU-based implementation of the detector and classifier. Experimental results are presented on a comprehensive set of video streams on excavators, trucks, and workers collected from different projects. Our preliminary results indicate the applicability of the proposed approach for automated recognition and real-time 2D tracking of workers and equipment from a single video camera. Unlike other methods, our algorithm can enable automated and real-time construction performance assessment (including detection of idle resources) and does not need manual or semi-automated initialization of the resources in 2D video frames. The preliminary experimental results and perceived benefits of the proposed method are discussed in detail.