Utilizing Response Time in On-the-Fly Multistage Adaptive Testing

Yang Du; Anqi Li; Hua Hua Chang

doi:10.1007/978-3-030-01310-3_10

Utilizing Response Time in On-the-Fly Multistage Adaptive Testing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

On-the-fly multistage adaptive testing (OMST), which integrates computerized adaptive testing (CAT) and multistage testing (MST), has recently gained popularity. While CAT selects each item on-the-fly and MST bundles items to pre-assembled modules, OMST assembles modules on-the-fly after the first stage. Since item selection algorithms play a crucial role in latent trait estimation and test security in CAT designs, given the availability of response time (RT) in the current testing era, researchers have been actively striving to incorporate RT into item selection algorithms. However, most such algorithms were only applied to CAT whereas little is known about RT’s role in the domain of OMST. Building upon previous research on RT-oriented item selection procedures, this research intends to apply RT-oriented item selection algorithms to OMST. This study found that the relative performance of RT-oriented item selection methods in OMST was consistent with CAT. But the underlying item bank structure and test design features can make a huge difference with respect to estimation accuracy and test security.

Original language	English (US)
Title of host publication	Quantitative Psychology - 83rd Annual Meeting of the Psychometric Society, 2018
Editors	Marie Wiberg, Dylan Molenaar, Jorge González, Steven Culpepper, Rianne Janssen
Publisher	Springer
Pages	107-117
Number of pages	11
ISBN (Print)	9783030013097
DOIs	https://doi.org/10.1007/978-3-030-01310-3_10
State	Published - 2019
Externally published	Yes
Event	83rd Annual meeting of the Psychometric Society, 2018 - New York, United States Duration: Jul 9 2018 → Jul 13 2018

Publication series

Name	Springer Proceedings in Mathematics and Statistics
Volume	265
ISSN (Print)	2194-1009
ISSN (Electronic)	2194-1017

Conference

Conference	83rd Annual meeting of the Psychometric Society, 2018
Country/Territory	United States
City	New York
Period	7/9/18 → 7/13/18

Keywords

CAT
Item bank usage
On-the-fly multistage tests
Response time
Test security

ASJC Scopus subject areas

General Mathematics

Online availability

10.1007/978-3-030-01310-3_10

Library availability

Discover UIUC Full Text

Cite this

Du, Y., Li, A., & Chang, H. H. (2019). Utilizing Response Time in On-the-Fly Multistage Adaptive Testing. In M. Wiberg, D. Molenaar, J. González, S. Culpepper, & R. Janssen (Eds.), Quantitative Psychology - 83rd Annual Meeting of the Psychometric Society, 2018 (pp. 107-117). (Springer Proceedings in Mathematics and Statistics; Vol. 265). Springer. https://doi.org/10.1007/978-3-030-01310-3_10

Utilizing Response Time in On-the-Fly Multistage Adaptive Testing. / Du, Yang; Li, Anqi; Chang, Hua Hua.
Quantitative Psychology - 83rd Annual Meeting of the Psychometric Society, 2018. ed. / Marie Wiberg; Dylan Molenaar; Jorge González; Steven Culpepper; Rianne Janssen. Springer, 2019. p. 107-117 (Springer Proceedings in Mathematics and Statistics; Vol. 265).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Du, Y, Li, A & Chang, HH 2019, Utilizing Response Time in On-the-Fly Multistage Adaptive Testing. in M Wiberg, D Molenaar, J González, S Culpepper & R Janssen (eds), Quantitative Psychology - 83rd Annual Meeting of the Psychometric Society, 2018. Springer Proceedings in Mathematics and Statistics, vol. 265, Springer, pp. 107-117, 83rd Annual meeting of the Psychometric Society, 2018, New York, United States, 7/9/18. https://doi.org/10.1007/978-3-030-01310-3_10

@inproceedings{7fef19b4b3914c3991911adb6e02b962,

title = "Utilizing Response Time in On-the-Fly Multistage Adaptive Testing",

abstract = "On-the-fly multistage adaptive testing (OMST), which integrates computerized adaptive testing (CAT) and multistage testing (MST), has recently gained popularity. While CAT selects each item on-the-fly and MST bundles items to pre-assembled modules, OMST assembles modules on-the-fly after the first stage. Since item selection algorithms play a crucial role in latent trait estimation and test security in CAT designs, given the availability of response time (RT) in the current testing era, researchers have been actively striving to incorporate RT into item selection algorithms. However, most such algorithms were only applied to CAT whereas little is known about RT{\textquoteright}s role in the domain of OMST. Building upon previous research on RT-oriented item selection procedures, this research intends to apply RT-oriented item selection algorithms to OMST. This study found that the relative performance of RT-oriented item selection methods in OMST was consistent with CAT. But the underlying item bank structure and test design features can make a huge difference with respect to estimation accuracy and test security.",

keywords = "CAT, Item bank usage, On-the-fly multistage tests, Response time, Test security",

author = "Yang Du and Anqi Li and Chang, {Hua Hua}",

note = "Publisher Copyright: {\textcopyright} 2019, Springer Nature Switzerland AG.; 83rd Annual meeting of the Psychometric Society, 2018 ; Conference date: 09-07-2018 Through 13-07-2018",

year = "2019",

doi = "10.1007/978-3-030-01310-3_10",

language = "English (US)",

isbn = "9783030013097",

series = "Springer Proceedings in Mathematics and Statistics",

publisher = "Springer",

pages = "107--117",

editor = "Marie Wiberg and Dylan Molenaar and Jorge Gonz{\'a}lez and Steven Culpepper and Rianne Janssen",

booktitle = "Quantitative Psychology - 83rd Annual Meeting of the Psychometric Society, 2018",

address = "Germany",

}

TY - GEN

T1 - Utilizing Response Time in On-the-Fly Multistage Adaptive Testing

AU - Du, Yang

AU - Li, Anqi

AU - Chang, Hua Hua

PY - 2019

Y1 - 2019

N2 - On-the-fly multistage adaptive testing (OMST), which integrates computerized adaptive testing (CAT) and multistage testing (MST), has recently gained popularity. While CAT selects each item on-the-fly and MST bundles items to pre-assembled modules, OMST assembles modules on-the-fly after the first stage. Since item selection algorithms play a crucial role in latent trait estimation and test security in CAT designs, given the availability of response time (RT) in the current testing era, researchers have been actively striving to incorporate RT into item selection algorithms. However, most such algorithms were only applied to CAT whereas little is known about RT’s role in the domain of OMST. Building upon previous research on RT-oriented item selection procedures, this research intends to apply RT-oriented item selection algorithms to OMST. This study found that the relative performance of RT-oriented item selection methods in OMST was consistent with CAT. But the underlying item bank structure and test design features can make a huge difference with respect to estimation accuracy and test security.

AB - On-the-fly multistage adaptive testing (OMST), which integrates computerized adaptive testing (CAT) and multistage testing (MST), has recently gained popularity. While CAT selects each item on-the-fly and MST bundles items to pre-assembled modules, OMST assembles modules on-the-fly after the first stage. Since item selection algorithms play a crucial role in latent trait estimation and test security in CAT designs, given the availability of response time (RT) in the current testing era, researchers have been actively striving to incorporate RT into item selection algorithms. However, most such algorithms were only applied to CAT whereas little is known about RT’s role in the domain of OMST. Building upon previous research on RT-oriented item selection procedures, this research intends to apply RT-oriented item selection algorithms to OMST. This study found that the relative performance of RT-oriented item selection methods in OMST was consistent with CAT. But the underlying item bank structure and test design features can make a huge difference with respect to estimation accuracy and test security.

KW - CAT

KW - Item bank usage

KW - On-the-fly multistage tests

KW - Response time

KW - Test security

UR - http://www.scopus.com/inward/record.url?scp=85066095156&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066095156&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01310-3_10

DO - 10.1007/978-3-030-01310-3_10

M3 - Conference contribution

AN - SCOPUS:85066095156

SN - 9783030013097

T3 - Springer Proceedings in Mathematics and Statistics

SP - 107

EP - 117

BT - Quantitative Psychology - 83rd Annual Meeting of the Psychometric Society, 2018

A2 - Wiberg, Marie

A2 - Molenaar, Dylan

A2 - González, Jorge

A2 - Culpepper, Steven

A2 - Janssen, Rianne

PB - Springer

T2 - 83rd Annual meeting of the Psychometric Society, 2018

Y2 - 9 July 2018 through 13 July 2018

ER -

Utilizing Response Time in On-the-Fly Multistage Adaptive Testing

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Quantitative Psychology: 83rd Annual Meeting of the Psychometric Society, New York, NY 2018

Cite this