TY - CONF
T1 - An analysis of unsupervised pre-training in light of recent advances
AU - Le Paine, Tom
AU - Khorrami, Pooya
AU - Han, Wei
AU - Huang, Thomas S.
N1 - Funding Information:
This material is based upon work supported by the National Science Foundation under Grant No. 392 NSF IIS13-18971. The two Tesla K40 GPUs used for this research were donated by the NVIDIA Corporation. We would like to acknowledge Theano (Bergstra et al. (2010)) and Pylearn2 (Goodfellow et al. (2013a)), on which our code is based. Also, we would like to thank Shiyu Chang for many helpful discussions and suggestions.
Funding Information:
This material is based upon work supported by the National Science Foundation under Grant No. 392 NSF IIS13-18971. The two Tesla K40 GPUs used for this research were donated by the NVIDIA Corporation. We would like to acknowledge Theano (Bergstra et al. (2010)) and Pylearn2 (Good-fellow et al. (2013a)), on which our code is based. Also, we would like to thank Shiyu Chang for many helpful discussions and suggestions.
PY - 2015
Y1 - 2015
N2 - Convolutional neural networks perform well on object recognition because of a number of recent advances: rectified linear units (ReLUs), data augmentation, dropout, and large labelled datasets. Unsupervised data has been proposed as another way to improve performance. Unfortunately, unsupervised pre-training is not used by state-of-the-art methods leading to the following question: Is unsupervised pre-training still useful given recent advances? If so, when? We answer this in three parts: we 1) develop an unsupervised method that incorporates ReLUs and recent unsupervised regularization techniques, 2) analyze the benefits of unsupervised pre-training compared to data augmentation and dropout on CIFAR-10 while varying the ratio of unsupervised to supervised samples, 3) verify our findings on STL-10. We discover unsupervised pre-training, as expected, helps when the ratio of unsupervised to supervised samples is high, and surprisingly, hurts when the ratio is low. We also use unsupervised pre-training with additional color augmentation to achieve near state-of-the-art performance on STL-10.
AB - Convolutional neural networks perform well on object recognition because of a number of recent advances: rectified linear units (ReLUs), data augmentation, dropout, and large labelled datasets. Unsupervised data has been proposed as another way to improve performance. Unfortunately, unsupervised pre-training is not used by state-of-the-art methods leading to the following question: Is unsupervised pre-training still useful given recent advances? If so, when? We answer this in three parts: we 1) develop an unsupervised method that incorporates ReLUs and recent unsupervised regularization techniques, 2) analyze the benefits of unsupervised pre-training compared to data augmentation and dropout on CIFAR-10 while varying the ratio of unsupervised to supervised samples, 3) verify our findings on STL-10. We discover unsupervised pre-training, as expected, helps when the ratio of unsupervised to supervised samples is high, and surprisingly, hurts when the ratio is low. We also use unsupervised pre-training with additional color augmentation to achieve near state-of-the-art performance on STL-10.
UR - http://www.scopus.com/inward/record.url?scp=85083952515&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85083952515&partnerID=8YFLogxK
M3 - Paper
AN - SCOPUS:85083952515
T2 - 3rd International Conference on Learning Representations, ICLR 2015
Y2 - 7 May 2015 through 9 May 2015
ER -