DZip: Improved general-purpose lossless compression based on novel neural network modeling

Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, Idoia Ochoa

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We consider lossless compression based on statistical data modeling followed by prediction-based encoding, where an accurate statistical model for the input data leads to substantial improvements in compression. We propose DZip, a general-purpose compressor for sequential data that exploits the well-known modeling capabilities of neural networks (NNs) for prediction, followed by arithmetic coding. DZip uses a novel hybrid architecture based on adaptive and semi-adaptive training. Unlike most NN based compressors, DZip does not require additional training data and is not restricted to specific data types, only needing the alphabet size of the input data. The proposed compressor outperforms general-purpose compressors such as Gzip (on average 26% reduction) on a variety of real datasets, achieves near-optimal compression on synthetic datasets, and performs close to specialized compressors for large sequence lengths, without any human input. The main limitation of DZip in its current implementation is the encoding/decoding time, which limits its practicality. Nevertheless, the results showcase the potential of developing improved general-purpose compressors based on neural networks and hybrid modeling.

Original languageEnglish (US)
Title of host publicationProceedings - DCC 2020
Subtitle of host publicationData Compression Conference
EditorsAli Bilgin, Michael W. Marcellin, Joan Serra-Sagrista, James A. Storer
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages372
Number of pages1
ISBN (Electronic)9781728164571
DOIs
StatePublished - Mar 2020
Event2020 Data Compression Conference, DCC 2020 - Snowbird, United States
Duration: Mar 24 2020Mar 27 2020

Publication series

NameData Compression Conference Proceedings
Volume2020-March
ISSN (Print)1068-0314

Conference

Conference2020 Data Compression Conference, DCC 2020
CountryUnited States
CitySnowbird
Period3/24/203/27/20

ASJC Scopus subject areas

  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'DZip: Improved general-purpose lossless compression based on novel neural network modeling'. Together they form a unique fingerprint.

Cite this