Code coverage and input variability: Effects on architecture and compiler research

Meaningful application benchmarking is crucial to processor design space exploration and compiler development. Many recent studies in the embedded processor domain have used reference telecommunications C programs which were originally intended to verify product compliance with a standard. In this paper, we demonstrate that telecommunications reference applications include a significant amount of superfluous code that skews a wide variety of experiments related to code size. A categorization of extra code is presented, and it is also demonstrated that benchmark inputs do not test broad usage patterns that might exist in a real system. It is shown that care must be taken in future studies to properly construct benchmarks which match the intended purpose of the target system and to provide inputs that more thoroughly exercise paths through telecommunications applications.