Does Standard Deviation Matter? Using "Standard Deviation" to Quantify Security of Multistage Testing

Chun Wang, Yi Zheng, Hua-hua Chang

Research output: Contribution to journalArticlepeer-review

Abstract

With the advent of web-based technology, online testing is becoming a mainstream mode in large-scale educational assessments. Most online tests are administered continuously in a testing window, which may post test security problems because examinees who take the test earlier may share information with those who take the test later. Researchers have proposed various statistical indices to assess the test security, and one most often used index is the average test-overlap rate, which was further generalized to the item pooling index (Chang & Zhang, 2002, 2003). These indices, however, are all defined as the means (that is, the expected proportion of common items among examinees) and they were originally proposed for computerized adaptive testing (CAT). Recently, multistage testing (MST) has become a popular alternative to CAT. The unique features of MST make it important to report not only the mean, but also the standard deviation (SD) of test overlap rate, as we advocate in this paper. The standard deviation of test overlap rate adds important information to the test security profile, because for the same mean, a large SD reflects that certain groups of examinees share more common items than other groups. In this study, we analytically derived the lower bounds of the SD under MST, with the results under CAT as a benchmark. It is shown that when the mean overlap rate is the same between MST and CAT, the SD of test overlap tends to be larger in MST. A simulation study was conducted to provide empirical evidence. We also compared the security of MST under the single-pool versus the multiple-pool designs; both analytical and simulation studies show that the non-overlapping multiple-pool design will slightly increase the security risk.

Original languageEnglish (US)
Pages (from-to)154-174
Number of pages21
JournalPsychometrika
Volume79
Issue number1
DOIs
StatePublished - Jan 2014

Keywords

  • computerized adaptive testing
  • item exposure
  • multistage testing
  • standard deviation
  • test overlap rate

ASJC Scopus subject areas

  • General Psychology
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'Does Standard Deviation Matter? Using "Standard Deviation" to Quantify Security of Multistage Testing'. Together they form a unique fingerprint.

Cite this