DLbricks: Composable benchmark generation to reduce deep learning benchmarking effort on CPUs

Cheng Li, Abdul Dakkak, Jinjun Xiong, Wen Mei Hwu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The past few years have seen a surge of applying Deep Learning (DL) models for a wide array of tasks such as image classification, object detection, machine translation, etc. While DL models provide an opportunity to solve otherwise intractable tasks, their adoption relies on them being optimized to meet target latency and resource requirements. Benchmarking is a key step in this process but has been hampered in part due to the lack of representative and up-to-date benchmarking suites. This paper proposes DLBricks, a composable benchmark generation design that reduces the effort of developing, maintaining, and running DL benchmarks. DLBricks decomposes DL models into a set of unique runnable networks and constructs the original model's performance using the performance of the generated benchmarks. Since benchmarks are generated automatically and the benchmarking time is minimized, DLBricks can keep up-to-date with the latest proposed models, relieving the pressure of selecting representative DL models. We evaluate DLBricks using 50 MXNet models spanning 5 DL tasks on 4 representative CPU systems. We show that DLBricks provides an accurate performance estimate for the DL models and reduces the benchmarking time across systems (e.g. within 95% accuracy and up to 4.4× benchmarking time speedup on Amazon EC2 c5.xlarge).

Original languageEnglish (US)
Title of host publicationICPE 2020 - Proceedings of the ACM/SPEC International Conference on Performance Engineering
PublisherAssociation for Computing Machinery, Inc
Pages202-209
Number of pages8
ISBN (Electronic)9781450369916
DOIs
StatePublished - Apr 20 2020
Event11th ACM/SPEC International Conference on Performance Engineering, ICPE 2020 - Edmonton, Canada
Duration: Apr 20 2020Apr 24 2020

Publication series

NameICPE 2020 - Proceedings of the ACM/SPEC International Conference on Performance Engineering

Conference

Conference11th ACM/SPEC International Conference on Performance Engineering, ICPE 2020
CountryCanada
CityEdmonton
Period4/20/204/24/20

Keywords

  • Benchmarking
  • Deep learning
  • Performance measurement

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'DLbricks: Composable benchmark generation to reduce deep learning benchmarking effort on CPUs'. Together they form a unique fingerprint.

  • Cite this

    Li, C., Dakkak, A., Xiong, J., & Hwu, W. M. (2020). DLbricks: Composable benchmark generation to reduce deep learning benchmarking effort on CPUs. In ICPE 2020 - Proceedings of the ACM/SPEC International Conference on Performance Engineering (pp. 202-209). (ICPE 2020 - Proceedings of the ACM/SPEC International Conference on Performance Engineering). Association for Computing Machinery, Inc. https://doi.org/10.1145/3358960.3379143