An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales

Walter C. Borman, Daren E. Buck, Stephan J. Motowidlo, Mary Ann Hanson, Stephen Stark, Fritz Drasgow

Research output: Contribution to journalArticlepeer-review

Abstract

This laboratory research compared the reliability, validity, and accuracy of a computerized adaptive rating scale (CARS) format and 2 relatively common and representative rating formats. The CARS is a paired-comparison rating task that uses adaptive testing prirciples to present pairs of scaled behavioral statements to the rater to iteratively estimate a ratee's effectiveness on 3 dimensions of contextual performance. Videotaped vignettes of 6 office workers were prepared, depicting prescripted levels of contextual performance, and 112 subjects rated these vignettes using the CARS format and one or the other competing format. Results showed 23%-37% lower standard errors of measurement for the CARS format. In addition, validity was significantly higher foi the CARS format (d = .18) and Cronbach's accuracy coefficients showed significantly higher accuracy, with a median effect size of .08. The discussion focuses on possible reasons for the results.

Original languageEnglish (US)
Pages (from-to)965-973
Number of pages9
JournalJournal of Applied Psychology
Volume86
Issue number5
DOIs
StatePublished - Oct 2001

ASJC Scopus subject areas

  • Applied Psychology

Fingerprint Dive into the research topics of 'An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales'. Together they form a unique fingerprint.

Cite this