The three R's of computer vision: Recognition, reconstruction and reorganization

Jitendra Malik, Pablo Arbeláez, João Carreira, Katerina Fragkiadaki, Ross Girshick, Georgia Gkioxari, Saurabh Gupta, Bharath Hariharan, Abhishek Kar, Shubham Tulsiani

Research output: Contribution to journalArticle


We argue for the importance of the interaction between recognition, reconstruction and re-organization, and propose that as a unifying framework for computer vision. In this view, recognition of objects is reciprocally linked to re-organization, with bottom-up grouping processes generating candidates, which can be classified using top down knowledge, following which the segmentations can be refined again. Recognition of 3D objects could benefit from a reconstruction of 3D structure, and 3D reconstruction can benefit from object category-specific priors. We also show that reconstruction of 3D structure from video data goes hand in hand with the reorganization of the scene. We demonstrate pipelined versions of two systems, one for RGB-D images, and another for RGB images, which produce rich 3D scene interpretations in this framework.

Original languageEnglish (US)
Pages (from-to)4-14
Number of pages11
JournalPattern Recognition Letters
StatePublished - Mar 1 2016
Externally publishedYes


  • 3D models
  • Action recognition: grouping
  • Object recognition
  • Segmentation
  • Shape reconstruction

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'The three R's of computer vision: Recognition, reconstruction and reorganization'. Together they form a unique fingerprint.

  • Cite this

    Malik, J., Arbeláez, P., Carreira, J., Fragkiadaki, K., Girshick, R., Gkioxari, G., Gupta, S., Hariharan, B., Kar, A., & Tulsiani, S. (2016). The three R's of computer vision: Recognition, reconstruction and reorganization. Pattern Recognition Letters, 72, 4-14.