TY - GEN
T1 - Recurrent Models for Situation Recognition
AU - Mallya, Arun
AU - Lazebnik, Svetlana
N1 - Publisher Copyright:
© 2017 IEEE.
PY - 2017/12/22
Y1 - 2017/12/22
N2 - This work proposes Recurrent Neural Network (RNN) models to predict structured 'image situations' - actions and noun entities fulfilling semantic roles related to the action. In contrast to prior work relying on Conditional Random Fields (CRFs), we use a specialized action prediction network followed by an RNN for noun prediction. Our system obtains state-of-the-art accuracy on the challenging recent imSitu dataset, beating CRF-based models, including ones trained with additional data. Further, we show that specialized features learned from situation prediction can be transferred to the task of image captioning to more accurately describe human-object interactions.
AB - This work proposes Recurrent Neural Network (RNN) models to predict structured 'image situations' - actions and noun entities fulfilling semantic roles related to the action. In contrast to prior work relying on Conditional Random Fields (CRFs), we use a specialized action prediction network followed by an RNN for noun prediction. Our system obtains state-of-the-art accuracy on the challenging recent imSitu dataset, beating CRF-based models, including ones trained with additional data. Further, we show that specialized features learned from situation prediction can be transferred to the task of image captioning to more accurately describe human-object interactions.
UR - http://www.scopus.com/inward/record.url?scp=85041905928&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85041905928&partnerID=8YFLogxK
U2 - 10.1109/ICCV.2017.57
DO - 10.1109/ICCV.2017.57
M3 - Conference contribution
AN - SCOPUS:85041905928
T3 - Proceedings of the IEEE International Conference on Computer Vision
SP - 455
EP - 463
BT - Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 16th IEEE International Conference on Computer Vision, ICCV 2017
Y2 - 22 October 2017 through 29 October 2017
ER -