Advances in Benchmarking Techniques: New Standards and Quantitative Metrics

Comparing computer system performance is difficult. Techniques for benchmarking computers have evolved from small test programs into extensive specifications of workloads. Over the last several years, competing vendors in several major computer markets have banded together to form consortia to develop standard benchmark sets. In addition to system evaluation, benchmarking is increasingly being used for system design. Characterizing the benchmarks in terms of system design requirements aids both of these activities. This chapter discusses new advances in benchmarking, including quantitative benchmark characterization. An overview of several popular, contemporary benchmark suites is presented, along with a discussion of the different philosophies of each suite. A detailed characterization is presented for a popular workstation benchmark suite—SPECint92. These characteristics focus on memory system and processor microarchitecture design, since these two subsystems largely determine the performance for workstation class machines. Although comprehensive in its hardware considerations, software interactions are not covered by these characteristics. Suggestions for software characteristics are discussed at the close of the chapter.