Investigating correspondence between language proficiency standards and academic content standards: A generalizability theory study

Research on the relationship between English language proficiency standards and academic content standards serves to provide information about the extent to which English language learners (ELLs) are expected to encounter academic language use that facilitates their content learning, such as in mathematics and science. Standards-to-standards correspondence thus contributes to validity evidence regarding ELL achievements in a standard-based assessment system. The current study aims to examine the reliability of reviewer judgments about language performance indicators associated with academic disciplines in standards-to-standards correspondence studies in the US K–12 settings. Ratings of cognitive complexity germane to the language performance indicators were collected from 20 correspondence studies with over 500 reviewers, consisting of content experts and ESL specialists. Using generalizability theory, we evaluate reviewer reliability and standard errors of measurement in their ratings with respect to the number of reviewers. Results show that depending on the particular grades and subject areas, 3–6 reviewers are needed to achieve acceptable reliability and to control for reasonable measurement errors in their judgments.