Towards optimal quantization of neural networks

Avhishek Chatterjee, Lav R Varshney

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Due to the unprecedented success of deep neural networks in inference tasks like speech and image recognition, there has been increasing interest in using them in mobile and in-sensor applications. As most current deep neural networks are very large in size, a major challenge lies in storing the network in devices with limited memory. Consequently there is growing interest in compressing deep networks by quantizing synaptic weights, but most prior work is heuristic and lacking theoretical foundations. Here we develop an approach to quantizing deep networks using functional high-rate quantization theory. Under certain technical conditions, this approach leads to an optimal quantizer that is computed using the celebrated backpropagation algorithm. In all other cases, a heuristic quantizer with certain regularization guarantees can be computed.

Original languageEnglish (US)
Title of host publication2017 IEEE International Symposium on Information Theory, ISIT 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1162-1166
Number of pages5
ISBN (Electronic)9781509040964
DOIs
StatePublished - Aug 9 2017
Event2017 IEEE International Symposium on Information Theory, ISIT 2017 - Aachen, Germany
Duration: Jun 25 2017Jun 30 2017

Publication series

NameIEEE International Symposium on Information Theory - Proceedings
ISSN (Print)2157-8095

Other

Other2017 IEEE International Symposium on Information Theory, ISIT 2017
CountryGermany
CityAachen
Period6/25/176/30/17

Fingerprint

Quantization
Neural Networks
Neural networks
Image recognition
Backpropagation algorithms
Speech recognition
Heuristics
Image Recognition
Back-propagation Algorithm
Speech Recognition
Data storage equipment
Regularization
Sensors
Sensor
Deep neural networks

Keywords

  • Deep neural network
  • Quantization theory

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Information Systems
  • Modeling and Simulation
  • Applied Mathematics

Cite this

Chatterjee, A., & Varshney, L. R. (2017). Towards optimal quantization of neural networks. In 2017 IEEE International Symposium on Information Theory, ISIT 2017 (pp. 1162-1166). [8006711] (IEEE International Symposium on Information Theory - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISIT.2017.8006711

Towards optimal quantization of neural networks. / Chatterjee, Avhishek; Varshney, Lav R.

2017 IEEE International Symposium on Information Theory, ISIT 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 1162-1166 8006711 (IEEE International Symposium on Information Theory - Proceedings).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Chatterjee, A & Varshney, LR 2017, Towards optimal quantization of neural networks. in 2017 IEEE International Symposium on Information Theory, ISIT 2017., 8006711, IEEE International Symposium on Information Theory - Proceedings, Institute of Electrical and Electronics Engineers Inc., pp. 1162-1166, 2017 IEEE International Symposium on Information Theory, ISIT 2017, Aachen, Germany, 6/25/17. https://doi.org/10.1109/ISIT.2017.8006711
Chatterjee A, Varshney LR. Towards optimal quantization of neural networks. In 2017 IEEE International Symposium on Information Theory, ISIT 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 1162-1166. 8006711. (IEEE International Symposium on Information Theory - Proceedings). https://doi.org/10.1109/ISIT.2017.8006711
Chatterjee, Avhishek ; Varshney, Lav R. / Towards optimal quantization of neural networks. 2017 IEEE International Symposium on Information Theory, ISIT 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 1162-1166 (IEEE International Symposium on Information Theory - Proceedings).
@inproceedings{36783c638b66419fb1627043057835d0,
title = "Towards optimal quantization of neural networks",
abstract = "Due to the unprecedented success of deep neural networks in inference tasks like speech and image recognition, there has been increasing interest in using them in mobile and in-sensor applications. As most current deep neural networks are very large in size, a major challenge lies in storing the network in devices with limited memory. Consequently there is growing interest in compressing deep networks by quantizing synaptic weights, but most prior work is heuristic and lacking theoretical foundations. Here we develop an approach to quantizing deep networks using functional high-rate quantization theory. Under certain technical conditions, this approach leads to an optimal quantizer that is computed using the celebrated backpropagation algorithm. In all other cases, a heuristic quantizer with certain regularization guarantees can be computed.",
keywords = "Deep neural network, Quantization theory",
author = "Avhishek Chatterjee and Varshney, {Lav R}",
year = "2017",
month = "8",
day = "9",
doi = "10.1109/ISIT.2017.8006711",
language = "English (US)",
series = "IEEE International Symposium on Information Theory - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "1162--1166",
booktitle = "2017 IEEE International Symposium on Information Theory, ISIT 2017",
address = "United States",

}

TY - GEN

T1 - Towards optimal quantization of neural networks

AU - Chatterjee, Avhishek

AU - Varshney, Lav R

PY - 2017/8/9

Y1 - 2017/8/9

N2 - Due to the unprecedented success of deep neural networks in inference tasks like speech and image recognition, there has been increasing interest in using them in mobile and in-sensor applications. As most current deep neural networks are very large in size, a major challenge lies in storing the network in devices with limited memory. Consequently there is growing interest in compressing deep networks by quantizing synaptic weights, but most prior work is heuristic and lacking theoretical foundations. Here we develop an approach to quantizing deep networks using functional high-rate quantization theory. Under certain technical conditions, this approach leads to an optimal quantizer that is computed using the celebrated backpropagation algorithm. In all other cases, a heuristic quantizer with certain regularization guarantees can be computed.

AB - Due to the unprecedented success of deep neural networks in inference tasks like speech and image recognition, there has been increasing interest in using them in mobile and in-sensor applications. As most current deep neural networks are very large in size, a major challenge lies in storing the network in devices with limited memory. Consequently there is growing interest in compressing deep networks by quantizing synaptic weights, but most prior work is heuristic and lacking theoretical foundations. Here we develop an approach to quantizing deep networks using functional high-rate quantization theory. Under certain technical conditions, this approach leads to an optimal quantizer that is computed using the celebrated backpropagation algorithm. In all other cases, a heuristic quantizer with certain regularization guarantees can be computed.

KW - Deep neural network

KW - Quantization theory

UR - http://www.scopus.com/inward/record.url?scp=85034099972&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85034099972&partnerID=8YFLogxK

U2 - 10.1109/ISIT.2017.8006711

DO - 10.1109/ISIT.2017.8006711

M3 - Conference contribution

AN - SCOPUS:85034099972

T3 - IEEE International Symposium on Information Theory - Proceedings

SP - 1162

EP - 1166

BT - 2017 IEEE International Symposium on Information Theory, ISIT 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -