Video segmentation by tracking many figure-ground segments

Fuxin Li, Taeyoung Kim, Ahmad Humayun, David Tsai, James M. Rehg

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We propose an unsupervised video segmentation approach by simultaneously tracking multiple holistic figure-ground segments. Segment tracks are initialized from a pool of segment proposals generated from a figure-ground segmentation algorithm. Then, online non-local appearance models are trained incrementally for each track using a multi-output regularized least squares formulation. By using the same set of training examples for all segment tracks, a computational trick allows us to track hundreds of segment tracks efficiently, as well as perform optimal online updates in closed-form. Besides, a new composite statistical inference approach is proposed for refining the obtained segment tracks, which breaks down the initial segment proposals and recombines for better ones by utilizing high-order statistic estimates from the appearance model and enforcing temporal consistency. For evaluating the algorithm, a dataset, SegTrack v2, is collected with about 1,000 frames with pixel-level annotations. The proposed framework outperforms state-of-the-art approaches in the dataset, showing its efficiency and robustness to challenges in different video sequences.

Original languageEnglish (US)
Title of host publicationProceedings - 2013 IEEE International Conference on Computer Vision, ICCV 2013
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages8
ISBN (Print)9781479928392
StatePublished - 2013
Externally publishedYes
Event2013 14th IEEE International Conference on Computer Vision, ICCV 2013 - Sydney, NSW, Australia
Duration: Dec 1 2013Dec 8 2013

Publication series

NameProceedings of the IEEE International Conference on Computer Vision


Other2013 14th IEEE International Conference on Computer Vision, ICCV 2013
CitySydney, NSW


  • appearance model
  • composite statistical inference
  • CPMC
  • CSI
  • tracking segments
  • Video Segmentation

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Video segmentation by tracking many figure-ground segments'. Together they form a unique fingerprint.

Cite this