Scoring situational judgment tests: Once you get the data, your troubles begin

Although situational judgment tests (SJTs) have been in use for decades, consensus has not been reached on the best way to score these assessments or others (e.g., biodata) whose items do not have a single demonstrably correct answer. The purpose of this paper is to review and to demonstrate the scoring strategies that have been described in the literature. Implementation and relative merits of these strategies are described. Then, several of these methods are applied to create 11 different keys for a video-based SJT in order to demonstrate how to evaluate the quality of keys. Implications of scoring SJTs for theory and practice are discussed.