Opportunities for Gaming the System: How Data Collection Design Influences Program Assessment Outcomes

Opportunities for college students to develop their leadership capacity through participation in formal leadership programs have grown tremendously. As the assessment of such programs rises, educators must become knowledgeable about the potentially significant design effects on assessment results. Using a national leadership program as a case study, we quasi-experimentally examined the effects of data collection design on measured outcomes, and discovered surprisingly noteworthy differences in results. Our work suggests that collection design may be even more important to whether leadership programs seem to make a difference than the curriculum itself.