Ctest4J: A Practical Configuration Testing Framework for Java

We present Ctest4J, a practical configuration testing framework for Java projects. Configuration testing is a recently proposed approach for finding both misconfigurations and code bugs. Ctest4J addresses the limitations of configuration testing scripts from prior work, including lack of parallel test execution, poor maintainability due to external dependencies, limited integration with modern build systems, and the need for manual instrumentation of configuration API. Ctest4J is a unified framework to write, maintain, and execute configuration tests (Ctests) and integrates with multiple testing frameworks (JUnit4, JUnit5, and TestNG) and build systems (Maven and Gradle). With Ctest4J, Ctests can be maintained similarly to regular unit tests. Ctest4J also provides a utility for automated code instrumentation for common configuration API. We evaluate Ctest4J on 12 open-source projects. We show that Ctest4J effectively enables configuration testing for these projects and speeds up Ctest execution by 3.4X compared to prior scripts. Ctest4J can be found at https://github.com/xlab-uiuc/ctest4j.