Within-group agreement for multi-item scales: Considering interitem correlations

Within-group agreement indexes support multilevel research on organizational climate, and are often used as a hurdle to justify aggregating individual-level responses to represent a group-level property. Such agreement indexes are derived from a theoretical model involving a group-level construct true score, with simplifying assumptions placed on the between-rater variance. The current paper comments on the role of these assumptions in traditional within-group agreement estimation, and focuses on one particular assumption made in the derivation of multi-item rwg(J): that internal consistency reliability (Cronbach's alpha) for climate scales is zero (LeBreton, James, & Lindell, 2005). We revisit LeBreton et al.'s derivation of the multi-item rwg agreement index, proposing an alternative set of derivations (and modified rwg agreement index) for use with reliable multi-item scales (where Cronbach's alpha > 0). Our goal is to highlight how the theoretical assumption of uncorrelated items on climate scales influences within-group agreement estimation.