Combining computer adaptive testing technology with cognitively diagnostic assessment

A major advantage of computerized adaptive testing (CAT) is that it allows the test to home in on an examinee's ability level in an interactive manner. The aim of the new area of cognitive diagnosis is to provide information about specific content areas in which an examinee needs help. The goal of this study was to combine the benefit of specific feedback from cognitively diagnostic assessment with the advantages of CAT. In this study, three approaches to combining these were investigated: (1) item selection based on the traditional ability level estimate (0), (2) item selection based on the attribute mastery feedback provided by cognitively diagnostic assessment (a), and (3) item selection based on both the traditional ability level estimate (0) and the attribute mastery feedback provided by cognitively diagnostic assessment (a). The results from these three approaches were compared for θ estimation accuracy, attribute mastery estimation accuracy, and item exposure control. The θ- and α-based condition outperformed the α-based condition regarding θ estimation, attribute mastery pattern estimation, and item exposure control. Both the θ-based condition and the θ- and α-based condition performed similarly with regard to θ estimation, attribute mastery estimation, and item exposure control, but the θ- and α-based condition has an additional advantage in that it uses the shadow test method, which allows the administrator to incorporate additional constraints in the item selection process, such as content balancing, item type constraints, and so forth, and also to select items on the basis of both the current θ and α estimates, which can be built on top of existing 3PL testing programs.