Quickly Producing "Isomorphic" Exercises: Quantifying the Impact of Programming Question Permutations

Small, auto-gradable programming exercises provide a useful tool with which to assess students' programming skills in introductory computer science. To reduce the time needed to produce programming exercises of similar difficulty, previous research has applied a permutation strategy to existing questions. Prior work has left several open questions: is prior exposure to a question typically indicative of higher student performance? Are observed changes in difficulty due to the specific surface feature permutations applied? How is student performance impacted by the first version of a question to which they may be exposed? In this work, we pursue this permutation strategy in multiple semesters of an introductory Python course to investigate these open questions. We use linear regression models to tease out the impacts of different surface feature changes and usage conditions. Our analysis finds similar tendencies as in prior work: question versions available in study materials tend to have 5 - 11 percentage point higher scores than novel permutations and more ''substantial'' surface feature changes tend to produce harder questions. Our results suggest this last finding is sensitive to how evenly permutations are applied across existing questions, as the precise impact of individual permutations changes between semesters.