Validity Study of the NAEP Mathematics Assessment: Grades 4 and 8: NAEP Validity Studies (NVS) Panel

In Spring 2006,. the NAEP Validity Studies (NVS) Panel was asked by the National Center for Education Statistics (NCES) to undertake a validity study to examine the quality of the NAEP Mathematics Assessments at grades 4 and 8. Specifically, NCES asked the NVS Panel to address five questions: (1) Does the NAEP framework offer reasonable content and skill-based coverage compared to the assessments of states and other nations? (2) Does the NAEP item pool and assessment design accurately reflect the NAEP framework? (3) Is NAEP mathematically accurate and not unduly oriented to a particular curriculum, philosophy or pedagogy? (4) Does NAEP properly consider the spread of abilities in the assessable population? (5) Does NAEP provide information that is representative of all students, including students who are unable to demonstrate their achievements on the standard assessment? The Panel's central finding of the validity study is that the NAEP mathematics assessment is sufficiently robust to support the main conclusions that have been drawn about the U.S. and state progress in mathematics since 1990. NAEP results show achievement in mathematics rising steadily over the years for all subgroups, although gaps among subgroups persist. Other findings are that: The NAEP framework is reasonable; however, the NAEP framework and specifications do not provide as much guidance for test developers as they could. The NAEP item pool broadly aligns with the framework with some important exceptions. Item quality is typical of large-scale assessments but could be better. Measurement precision is good over a broad range of proficiency but could be better for lower-achieving students. Nine recommendations flow from this study. They are: (1) Sharpen the framework; (2) Provide detailed implementation plans; (3) Define a larger role for exemplar items; (4) Improve quality assurance for the overall item pool and for individual items; (5) Attend particularly to the various aspects of item quality; (6) Undertake a program of evidence-based research on item design; (7) Expand the range of item difficulty and curricular reach; (8) Manage changes in the item pool; and (9) Move NAEP in the direction of adaptive testing. These recommendations hold promise for improving the quality of assessment, not only within the NAEP program, but for U.S. education overall.