Heuristic cognitive diagnosis when the Q-matrix is unknown

Hans Friedrich Koehn, Chia Yi Chiu, Michael J. Brusco

Research output: Contribution to journalArticle

Abstract

Cognitive diagnosis models of educational test performance rely on a binary Q-matrix that specifies the associations between individual test items and the cognitive attributes (skills) required to answer those items correctly. Current methods for fitting cognitive diagnosis models to educational test data and assigning examinees to proficiency classes are based on parametric estimation methods such as expectation maximization (EM) and Markov chain Monte Carlo (MCMC) that frequently encounter difficulties in practical applications. In response to these difficulties, non-parametric classification techniques (cluster analysis) have been proposed as heuristic alternatives to parametric procedures. These non-parametric classification techniques first aggregate each examinee's test item scores into a profile of attribute sum scores, which then serve as the basis for clustering examinees into proficiency classes. Like the parametric procedures, the non-parametric classification techniques require that the Q-matrix underlying a given test be known. Unfortunately, in practice, the Q-matrix for most tests is not known and must be estimated to specify the associations between items and attributes, risking a misspecified Q-matrix that may then result in the incorrect classification of examinees. This paper demonstrates that clustering examinees into proficiency classes based on their item scores rather than on their attribute sum-score profiles does not require knowledge of the Q-matrix, and results in a more accurate classification of examinees.

Original languageEnglish (US)
Pages (from-to)268-291
Number of pages24
JournalBritish Journal of Mathematical and Statistical Psychology
Volume68
Issue number2
DOIs
StatePublished - May 1 2015

Fingerprint

Q-matrix
Heuristics
Unknown
Attribute
Educational Models
Cluster Analysis
Clustering
Parametric Estimation
Markov Chains
Expectation Maximization
Performance Test
Markov Chain Monte Carlo
Binary
Alternatives
Model
Demonstrate
Class
Proficiency

Keywords

  • Asymptotic theory of cognitive diagnosis
  • Classification
  • Clustering
  • Cognitive diagnosis
  • Consistency
  • Heuristic

ASJC Scopus subject areas

  • Statistics and Probability
  • Arts and Humanities (miscellaneous)
  • Psychology(all)

Cite this

Heuristic cognitive diagnosis when the Q-matrix is unknown. / Koehn, Hans Friedrich; Chiu, Chia Yi; Brusco, Michael J.

In: British Journal of Mathematical and Statistical Psychology, Vol. 68, No. 2, 01.05.2015, p. 268-291.

Research output: Contribution to journalArticle

@article{76b37d4720094abb8a3e29df6e3421c9,
title = "Heuristic cognitive diagnosis when the Q-matrix is unknown",
abstract = "Cognitive diagnosis models of educational test performance rely on a binary Q-matrix that specifies the associations between individual test items and the cognitive attributes (skills) required to answer those items correctly. Current methods for fitting cognitive diagnosis models to educational test data and assigning examinees to proficiency classes are based on parametric estimation methods such as expectation maximization (EM) and Markov chain Monte Carlo (MCMC) that frequently encounter difficulties in practical applications. In response to these difficulties, non-parametric classification techniques (cluster analysis) have been proposed as heuristic alternatives to parametric procedures. These non-parametric classification techniques first aggregate each examinee's test item scores into a profile of attribute sum scores, which then serve as the basis for clustering examinees into proficiency classes. Like the parametric procedures, the non-parametric classification techniques require that the Q-matrix underlying a given test be known. Unfortunately, in practice, the Q-matrix for most tests is not known and must be estimated to specify the associations between items and attributes, risking a misspecified Q-matrix that may then result in the incorrect classification of examinees. This paper demonstrates that clustering examinees into proficiency classes based on their item scores rather than on their attribute sum-score profiles does not require knowledge of the Q-matrix, and results in a more accurate classification of examinees.",
keywords = "Asymptotic theory of cognitive diagnosis, Classification, Clustering, Cognitive diagnosis, Consistency, Heuristic",
author = "Koehn, {Hans Friedrich} and Chiu, {Chia Yi} and Brusco, {Michael J.}",
year = "2015",
month = "5",
day = "1",
doi = "10.1111/bmsp.12044",
language = "English (US)",
volume = "68",
pages = "268--291",
journal = "British Journal of Mathematical and Statistical Psychology",
issn = "0007-1102",
publisher = "Wiley-Blackwell",
number = "2",

}

TY - JOUR

T1 - Heuristic cognitive diagnosis when the Q-matrix is unknown

AU - Koehn, Hans Friedrich

AU - Chiu, Chia Yi

AU - Brusco, Michael J.

PY - 2015/5/1

Y1 - 2015/5/1

N2 - Cognitive diagnosis models of educational test performance rely on a binary Q-matrix that specifies the associations between individual test items and the cognitive attributes (skills) required to answer those items correctly. Current methods for fitting cognitive diagnosis models to educational test data and assigning examinees to proficiency classes are based on parametric estimation methods such as expectation maximization (EM) and Markov chain Monte Carlo (MCMC) that frequently encounter difficulties in practical applications. In response to these difficulties, non-parametric classification techniques (cluster analysis) have been proposed as heuristic alternatives to parametric procedures. These non-parametric classification techniques first aggregate each examinee's test item scores into a profile of attribute sum scores, which then serve as the basis for clustering examinees into proficiency classes. Like the parametric procedures, the non-parametric classification techniques require that the Q-matrix underlying a given test be known. Unfortunately, in practice, the Q-matrix for most tests is not known and must be estimated to specify the associations between items and attributes, risking a misspecified Q-matrix that may then result in the incorrect classification of examinees. This paper demonstrates that clustering examinees into proficiency classes based on their item scores rather than on their attribute sum-score profiles does not require knowledge of the Q-matrix, and results in a more accurate classification of examinees.

AB - Cognitive diagnosis models of educational test performance rely on a binary Q-matrix that specifies the associations between individual test items and the cognitive attributes (skills) required to answer those items correctly. Current methods for fitting cognitive diagnosis models to educational test data and assigning examinees to proficiency classes are based on parametric estimation methods such as expectation maximization (EM) and Markov chain Monte Carlo (MCMC) that frequently encounter difficulties in practical applications. In response to these difficulties, non-parametric classification techniques (cluster analysis) have been proposed as heuristic alternatives to parametric procedures. These non-parametric classification techniques first aggregate each examinee's test item scores into a profile of attribute sum scores, which then serve as the basis for clustering examinees into proficiency classes. Like the parametric procedures, the non-parametric classification techniques require that the Q-matrix underlying a given test be known. Unfortunately, in practice, the Q-matrix for most tests is not known and must be estimated to specify the associations between items and attributes, risking a misspecified Q-matrix that may then result in the incorrect classification of examinees. This paper demonstrates that clustering examinees into proficiency classes based on their item scores rather than on their attribute sum-score profiles does not require knowledge of the Q-matrix, and results in a more accurate classification of examinees.

KW - Asymptotic theory of cognitive diagnosis

KW - Classification

KW - Clustering

KW - Cognitive diagnosis

KW - Consistency

KW - Heuristic

UR - http://www.scopus.com/inward/record.url?scp=84927913441&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84927913441&partnerID=8YFLogxK

U2 - 10.1111/bmsp.12044

DO - 10.1111/bmsp.12044

M3 - Article

C2 - 25496248

AN - SCOPUS:84927913441

VL - 68

SP - 268

EP - 291

JO - British Journal of Mathematical and Statistical Psychology

JF - British Journal of Mathematical and Statistical Psychology

SN - 0007-1102

IS - 2

ER -