Knowledge flow: Improve upon your teachers

Iou Jen Liu, Jian Peng, Alexander G. Schwing

Research output: Contribution to conferencePaper

Abstract

A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves 'knowledge' from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other 'knowledge exchange' methods.

Original languageEnglish (US)
StatePublished - 2019
Event7th International Conference on Learning Representations, ICLR 2019 - New Orleans, United States
Duration: May 6 2019May 9 2019

Conference

Conference7th International Conference on Learning Representations, ICLR 2019
CountryUnited States
CityNew Orleans
Period5/6/195/9/19

ASJC Scopus subject areas

  • Education
  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Fingerprint Dive into the research topics of 'Knowledge flow: Improve upon your teachers'. Together they form a unique fingerprint.

  • Cite this

    Liu, I. J., Peng, J., & Schwing, A. G. (2019). Knowledge flow: Improve upon your teachers. Paper presented at 7th International Conference on Learning Representations, ICLR 2019, New Orleans, United States.