Evaluating the developmental stages of digital educational programs: Challenges and opportunities

The US Institute of Education Sciences (IES) currently provides a progressive series of education research grants that can support the development of innovative and creative educational programs, starting with exploratory grants, and progressing through developmental grants, efficacy trials, and finally, effectiveness studies. The IES further requires an evaluation component for each of these program developmental stages. In this article, the authors report on their experiences as evaluators of the developmental stage of a digital educational program called Scholar. Their reflections concentrate on what might be uniquely relevant to digital educational programs and to programs at the IES developmental stage. The key issues discussed include: (a) the challenge of assessing implementation fidelity and variability for a developmental project; (b) the challenge of assessing the program's reach to diverse program participants, especially those least well served in a given context; and (c) the challenges of securing sufficient and relevant implementation data, and of addressing pressures for gathering preliminary efficacy data. Interwoven through these discussions are snapshots of selected program participants' voices describing their own program experiences and views. The article concludes with the authors' reflections and commentary on future directions for evaluating comparable digital educational programs at the IES developmental stage.