Learning without Forgetting

Zhizhong Li, Derek Hoiem

Research output: Contribution to journalArticlepeer-review


When building a unified vision system or gradually adding new apabilities to a system, the usual assumption is that training data for all tasks is always available. However, as the number of tasks grows, storing and retraining on such data becomes infeasible. A new problem arises where we add new capabilities to a Convolutional Neural Network (CNN), but the training data for its existing capabilities are unavailable. We propose our Learning without Forgetting method, which uses only new task data to train the network while preserving the original capabilities. Our method performs favorably compared to commonly used feature extraction and fine-tuning adaption techniques and performs similarly to multitask learning that uses original task data we assume unavailable. A more surprising observation is that Learning without Forgetting may be able to replace fine-tuning with similar old and new task datasets for improved new task performance.

Original languageEnglish (US)
Article number8107520
Pages (from-to)2935-2947
Number of pages13
JournalIEEE transactions on pattern analysis and machine intelligence
Issue number12
StatePublished - Dec 1 2018


  • Convolutional neural networks
  • deep learning
  • multi-task learning
  • transfer learning
  • visual recognition

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics


Dive into the research topics of 'Learning without Forgetting'. Together they form a unique fingerprint.

Cite this