Survey Design, Order Effects, and Causal Mediation Analysis

Causal mediation analysis requires measurement of an outcome variable (O) with and without treatment, plus a set of mediator variables (M) that constitute possible pathways for the treatment effect. There is no consensus on whether surveys should measure potentially mediating variables before or after the outcome variables—MO or OM. We use a replication exercise to demonstrate how the order of mediator and outcome items can be consequential for the results from causal mediation analysis. Order can affect mediation conclusions, even if the treatment effect is similar across designs. As such, randomizing order is usually prudent, although best practice depends on the researcher’s contextual knowledge about her particular application.