A performance projection of mini-applications onto benchmarks toward the performance projection of real-applications

Miwako Tsuji, William T Kramer, Mitsuhisa Sato

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Widely used benchmarks, such as High Performance Linpack (HPL), do not always provide direct insights are notoriously poor indicators of into the actual application performance of systems. When real applications are used, and there have been are criticisms indicating that the performance of simplified benchmarks such as HPL no longer strongly correlate to real application performance. In contrast, performance evaluations based on real or mini applications may give a direct estimation into application performance. The Sustained System Performance (SSP) metric, which is used to evaluate systems based on the performance at scale of various applications, has been successfully adopted to procure systems at the National Energy Research Scientific Computing Center (NERSC), the National Center for Supercomputing Applications (NCSA) and other facilities. However, significant effort is required to tune and optimize several mini applications for each of systems. In this paper, we propose a new performance metric-the Simplified Sustained System Performance (SSSP) metric-based on a suite of simple benchmarks, which enables performance projection that correlates with full applications, but use onto a suite of mini applications. While the SSP metric is calculated over a set of applications, the SSSP metric applies its methodology to a set of benchmarks. Preliminary weighting factors for benchmarks are introduced to approximate the original SSP metric more accurately by the SSSP metric. To define the weighting factors, we perform a simple learning algorithm. Our preliminary experiments show that even though our metric is still easy to measure because it is based on a combination of simple benchmarks, it can provide projections of the performance of applications.

Original languageEnglish (US)
Title of host publicationProceedings - 2017 IEEE International Conference on Cluster Computing, CLUSTER 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages826-833
Number of pages8
ISBN (Electronic)9781538623268
DOIs
StatePublished - Sep 22 2017
Event2017 IEEE International Conference on Cluster Computing, CLUSTER 2017 - Honolulu, United States
Duration: Sep 5 2017Sep 8 2017

Publication series

NameProceedings - IEEE International Conference on Cluster Computing, ICCC
Volume2017-September
ISSN (Print)1552-5244

Other

Other2017 IEEE International Conference on Cluster Computing, CLUSTER 2017
CountryUnited States
CityHonolulu
Period9/5/179/8/17

    Fingerprint

Keywords

  • Benchmarks
  • Mini-Applications
  • Performance projection

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Signal Processing

Cite this

Tsuji, M., Kramer, W. T., & Sato, M. (2017). A performance projection of mini-applications onto benchmarks toward the performance projection of real-applications. In Proceedings - 2017 IEEE International Conference on Cluster Computing, CLUSTER 2017 (pp. 826-833). [8049025] (Proceedings - IEEE International Conference on Cluster Computing, ICCC; Vol. 2017-September). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CLUSTER.2017.123