Advances in Benchmarking Techniques: New Standards and Quantitative Metrics

Thomas M. Conte, Wen-Mei W Hwu

Research output: Contribution to journalArticle

Abstract

Comparing computer system performance is difficult. Techniques for benchmarking computers have evolved from small test programs into extensive specifications of workloads. Over the last several years, competing vendors in several major computer markets have banded together to form consortia to develop standard benchmark sets. In addition to system evaluation, benchmarking is increasingly being used for system design. Characterizing the benchmarks in terms of system design requirements aids both of these activities. This chapter discusses new advances in benchmarking, including quantitative benchmark characterization. An overview of several popular, contemporary benchmark suites is presented, along with a discussion of the different philosophies of each suite. A detailed characterization is presented for a popular workstation benchmark suite—SPECint92. These characteristics focus on memory system and processor microarchitecture design, since these two subsystems largely determine the performance for workstation class machines. Although comprehensive in its hardware considerations, software interactions are not covered by these characteristics. Suggestions for software characteristics are discussed at the close of the chapter.

Original languageEnglish (US)
Pages (from-to)231-253
Number of pages23
JournalAdvances in Computers
Volume41
Issue numberC
DOIs
StatePublished - Jan 1 1995

Fingerprint

Benchmarking
Systems analysis
Computer workstations
Computer systems
Specifications
Hardware
Data storage equipment

ASJC Scopus subject areas

  • Computer Science(all)

Cite this

Advances in Benchmarking Techniques : New Standards and Quantitative Metrics. / Conte, Thomas M.; Hwu, Wen-Mei W.

In: Advances in Computers, Vol. 41, No. C, 01.01.1995, p. 231-253.

Research output: Contribution to journalArticle

@article{65cd6ffffafb4eff85772eda87ff3873,
title = "Advances in Benchmarking Techniques: New Standards and Quantitative Metrics",
abstract = "Comparing computer system performance is difficult. Techniques for benchmarking computers have evolved from small test programs into extensive specifications of workloads. Over the last several years, competing vendors in several major computer markets have banded together to form consortia to develop standard benchmark sets. In addition to system evaluation, benchmarking is increasingly being used for system design. Characterizing the benchmarks in terms of system design requirements aids both of these activities. This chapter discusses new advances in benchmarking, including quantitative benchmark characterization. An overview of several popular, contemporary benchmark suites is presented, along with a discussion of the different philosophies of each suite. A detailed characterization is presented for a popular workstation benchmark suite—SPECint92. These characteristics focus on memory system and processor microarchitecture design, since these two subsystems largely determine the performance for workstation class machines. Although comprehensive in its hardware considerations, software interactions are not covered by these characteristics. Suggestions for software characteristics are discussed at the close of the chapter.",
author = "Conte, {Thomas M.} and Hwu, {Wen-Mei W}",
year = "1995",
month = "1",
day = "1",
doi = "10.1016/S0065-2458(08)60235-1",
language = "English (US)",
volume = "41",
pages = "231--253",
journal = "Advances in Computers",
issn = "0065-2458",
publisher = "Academic Press Inc.",
number = "C",

}

TY - JOUR

T1 - Advances in Benchmarking Techniques

T2 - New Standards and Quantitative Metrics

AU - Conte, Thomas M.

AU - Hwu, Wen-Mei W

PY - 1995/1/1

Y1 - 1995/1/1

N2 - Comparing computer system performance is difficult. Techniques for benchmarking computers have evolved from small test programs into extensive specifications of workloads. Over the last several years, competing vendors in several major computer markets have banded together to form consortia to develop standard benchmark sets. In addition to system evaluation, benchmarking is increasingly being used for system design. Characterizing the benchmarks in terms of system design requirements aids both of these activities. This chapter discusses new advances in benchmarking, including quantitative benchmark characterization. An overview of several popular, contemporary benchmark suites is presented, along with a discussion of the different philosophies of each suite. A detailed characterization is presented for a popular workstation benchmark suite—SPECint92. These characteristics focus on memory system and processor microarchitecture design, since these two subsystems largely determine the performance for workstation class machines. Although comprehensive in its hardware considerations, software interactions are not covered by these characteristics. Suggestions for software characteristics are discussed at the close of the chapter.

AB - Comparing computer system performance is difficult. Techniques for benchmarking computers have evolved from small test programs into extensive specifications of workloads. Over the last several years, competing vendors in several major computer markets have banded together to form consortia to develop standard benchmark sets. In addition to system evaluation, benchmarking is increasingly being used for system design. Characterizing the benchmarks in terms of system design requirements aids both of these activities. This chapter discusses new advances in benchmarking, including quantitative benchmark characterization. An overview of several popular, contemporary benchmark suites is presented, along with a discussion of the different philosophies of each suite. A detailed characterization is presented for a popular workstation benchmark suite—SPECint92. These characteristics focus on memory system and processor microarchitecture design, since these two subsystems largely determine the performance for workstation class machines. Although comprehensive in its hardware considerations, software interactions are not covered by these characteristics. Suggestions for software characteristics are discussed at the close of the chapter.

UR - http://www.scopus.com/inward/record.url?scp=77956857701&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77956857701&partnerID=8YFLogxK

U2 - 10.1016/S0065-2458(08)60235-1

DO - 10.1016/S0065-2458(08)60235-1

M3 - Article

AN - SCOPUS:77956857701

VL - 41

SP - 231

EP - 253

JO - Advances in Computers

JF - Advances in Computers

SN - 0065-2458

IS - C

ER -