How productive are homework and elective practice? Applying a post hoc modeling of student knowledge in a large, introductory computing course

In this paper, we attempt to estimate how much learning happens in required practice activities (homework) relative to elective practice activities (studying). This analysis is done in the context of a large enrollment (N = 601) introductory programming course that made heavy use of auto-grading randomizing question (item) generators. Because these item generators (and other problems) were used as homework, on practice exams, and as part of exams, a given student may have encountered the same generator multiple times during the class, providing snapshots of the evolution of the student's ability to complete that problem correctly. We use a post hoc model of “this-item-correct” prediction to estimate individual student knowledge on each attempt of a given question. Across five exams, correctness tracing attributes 57-65% of the learning that occurs to the homework period and the remainder to elective practice (the study period).