Population Risk Improvement with Model Compression: An Information-Theoretic Approach

Yuheng Bu, Weihao Gao, Shaofeng Zou, Venugopal V. Veeravalli

Research output: Contribution to journalArticlepeer-review

Abstract

It has been reported in many recent works on deep model compression that the population risk of a compressed model can be even better than that of the original model. In this paper, an information-theoretic explanation for this population risk improvement phenomenon is provided by jointly studying the decrease in the generalization error and the increase in the empirical risk that results from model compression. It is first shown that model compression reduces an information-theoretic bound on the generalization error, which suggests that model compression can be interpreted as a regularization technique to avoid overfitting. The increase in empirical risk caused by model compression is then characterized using rate distortion theory. These results imply that the overall population risk could be improved by model compression if the decrease in generalization error exceeds the increase in empirical risk. A linear regression example is presented to demonstrate that such a decrease in population risk due to model compression is indeed possible. Our theoretical results further suggest a way to improve a widely used model compression algorithm, i.e., Hessian-weighted K-means clustering, by regularizing the distance between the clustering centers. Experiments with neural networks are provided to validate our theoretical assertions.
Original languageEnglish (US)
JournalEntropy
Volume23
Issue number10
DOIs
StatePublished - 2021

Keywords

  • empirical risk
  • generalization error
  • K-means clustering
  • model compression
  • population risk
  • rate distortion theory
  • vector quantization

Fingerprint

Dive into the research topics of 'Population Risk Improvement with Model Compression: An Information-Theoretic Approach'. Together they form a unique fingerprint.

Cite this