Abstract

Deep neural networks have shown incredible performance for inference tasks in a variety of domains, but require significant storage space, which limits scaling and use for on-device intelligence. This paper is concerned with finding universal lossless compressed representations of deep feedforward networks with synaptic weights drawn from discrete sets, and directly performing inference without full decompression. The basic insight that allows less rate than naïve approaches is recognizing that the bipartite graph layers of feedforward networks have a kind of permutation invariance to the labeling of nodes, in terms of inferential operation. We provide efficient algorithms to dissipate this irrelevant uncertainty and then use arithmetic coding to nearly achieve the entropy bound in a universal manner. We also provide experimental results of our approach on several standard datasets.

Original languageEnglish (US)
Pages (from-to)732-745
Number of pages14
JournalIEEE Journal on Selected Areas in Information Theory
Volume3
Issue number4
DOIs
StatePublished - Dec 1 2022

Keywords

  • Neural network compression
  • artificial neural networks
  • entropy coding
  • source coding

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Media Technology
  • Artificial Intelligence
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'Universal and Succinct Source Coding of Deep Neural Networks'. Together they form a unique fingerprint.

Cite this