Improving Generalizability of Developmental Research Through Increased Use of Homogeneous Convenience Samples: A Monte Carlo Simulation

Due to its heavy reliance on convenience samples (CSs), developmental science has a generalizability problem that clouds its broader applicability and frustrates replicability. The surest solution to this problem is to make better use, where feasible, of probability samples, which afford clear generalizability. Because CSs that are homogeneous on one or more sociodemographic factor may afford a clearer generalizability than heterogeneous CSs, the use of homogeneous CSs instead of heterogeneous CSs may also help mitigate this generalizability problem. In this article, we argue why homogeneous CSs afford clearer generalizability, and we formally test this argument via Monte Carlo simulations. For illustration, our simulations focused on sampling bias in the sociodemographic factors of ethnicity and socioeconomic status and on the outcome of adolescent academic achievement. Monte Carlo simulations indicated that homogeneous CSs (particularly those homogeneous on multiple sociodemographic factors) reliably produce estimates that are appreciably less biased than heterogeneous CSs. Sensitivity analyses indicated that these reductions in estimate bias generalize to estimates of means and estimates of association (e.g., correlations) although reductions in estimate bias were more muted for associations. The increased employment of homogeneous CSs (particularly those homogeneous on multiple sociodemographic factors) instead of heterogeneous CSs would appreciably improve the generalizability of developmental research. Broader implications for replicability and the study of minoritized populations, considerations for application, and suggestions for sampling best practices are discussed.