Cultivating assessment literacy: Standards evaluation through language-test specification reverse engineering

The promulgation of ESL learning standards by government agencies and professional organizations is intended to improve second-language (L2) proficiency and academic achievement. However, mandates have been criticized on various grounds, including difficulty of defining domains to be tested as well as vagueness of articulation. Arising from the last concern, this article reports on a study in which ESL instructors critiqued state standards and performance indicators by applying standards reverse engineering (SRE) to language-test specifications. The research was intended to determine the feasibility of employing SRE in standards/test evaluation and in facilitation of critical awareness of standards among ESL teachers, to determine the extent to which SRE could determine alignment between select items taken from state-mandated tests and state standards, and to investigate effects prior training in test-specification writing might have on SRE. The research design involved a workshop format, participants performing SRE on sample items of the New York State English as a Second Language Achievement Test. Results suggested that SRE is feasible for determining alignment of items with state standards and critiquing the standards themselves.