Advances in Psychometric Methods for Uncovering Latent Structure and Cognitive Processes

Diagnostic models are an important component of learning technology aimed at advancing a developmental framework for assessment. Current applications employ confirmatory methods which require detailed knowledge about the underlying structure. In this chapter, we review advances in exploratory diagnostic models for inferring the underlying structure and latent processes. We introduce a general diagnostic model, discuss Bayesian estimation of model parameters, and highlight important theoretical results concerning the identifiability of model parameters. We present an application of an exploratory diagnostic model and offer a detailed data analysis that covers issues of posterior convergence, relative model fit, and interpretation of parameter estimates. Interest in online learning technologies continues as a larger effort to support a developmental framework for assessment. New learning innovations are grounded in the notion of skill-based, formative assessment, which is a departure from summative assessments that rank test takers on one or a few continuous scales. Diagnostic models (DMs) are uniquely designed to support formative assessment. In fact, the ability of DMs to provide educators and decision-makers with fine-grained assessment information about student skill mastery makes it a popular psychometric framework for learning technology innovations. However, current research generally employs confirmatory DMs, which requires considerable domain-specific knowledge to accurately implement. Specifically, researchers must have detailed knowledge about the underlying skills needed to succeed on each item. Specific knowledge and cognitive theory may be available to support applications in some domains (e.g., mixed number fraction–subtraction; Tatsuoka, 1984), but the availability of theory may be the exception rather than the rule. Consequently, recent research (Chen, Culpepper, Chen, & Douglas, 2018; Chen, Liu, Xu, & Ying, 2015; Culpepper, 2019; Xu & Shang, 2017) developed exploratory DMs for binary data to provide researchers with tools to validate and develop cognitive theory that is needed to support learning technologies. In fact, a significant body of theoretical research established new identifiability conditions for exploratory DMs (e.g., see Chen et al., 2015; Xu, 2017; Xu & Shang, 2017). Additionally, the availability of identifiability conditions provided the foundation for several applications of exploratory DMs to fraction-subtraction (Chen et al., 2015; Chen et al., 2018; Culpepper, 2019) and items for the diagnosis of psychopathology (Chen et al., 2015; Jimenez & Culpepper, 2018). The purpose of this chapter is twofold. First, we review recent advances regarding exploratory DMs for validating and inferring the latent structure and response processes. In particular, we consider a general diagnostic modeling framework and discuss issues related to parameterization, Bayesian estimation, and identifiability of model parameters. Second, we present a new application of the exploratory DM and highlight issues such as model fit, convergence, and interpretation of the underlying structure and latent processes. Specifically, we present an application of the exploratory binary DM to 16 multiple choice items from the Synthetic Aperture Personality Assessment (SAPA; Revelle, 2018). The final section of this chapter offers discussion of the findings and provides concluding remarks.