How Big Are My Effects? Examining the Magnitude of Effect Sizes in Studies of Measurement Equivalence

Recently, an effect size measure, known as d_MACS, was developed for confirmatory factor analytic (CFA) studies of measurement equivalence. Although this index has several advantages over traditional methods of identifying nonequivalence, the scale and interpretation of this effect size are still unclear. As a result, the interpretation of the effect size is left to the subjective judgment of the researcher. To remedy this issue for other effect sizes, some have proposed guidelines for evaluating the magnitude of an effect based on the distribution of effect sizes in the literature. The goal of the current research was to develop similar guidelines for effect sizes of measurement nonequivalence and build on this work by also examining the practical importance of nonequivalence. Based on a review of past research, we conducted two simulation studies to generate distributions of effects sizes. Assuming the ideal scenario of invariant referent items, the results of these simulations were then used to develop empirical guidelines for interpreting nonequivalence and its effects on observed outcomes.