TY - JOUR
T1 - Rejective Sampling, Rerandomization, and Regression Adjustment in Survey Experiments
AU - Yang, Zihao
AU - Qu, Tianyi
AU - Li, Xinran
N1 - Funding Information:
We thank the associate editor and two reviewers for insightful and constructive comments.
Publisher Copyright:
© 2021 American Statistical Association.
PY - 2023
Y1 - 2023
N2 - Classical randomized experiments, equipped with randomization-based inference, provide assumption-free inference for treatment effects. They have been the gold standard for drawing causal inference and provide excellent internal validity. However, they have also been criticized for questionable external validity, in the sense that the conclusion may not generalize well to a larger population. The randomized survey experiment is a design tool that can help mitigate this concern, by randomly selecting the experimental units from the target population of interest. However, as pointed out by Morgan and Rubin, chance imbalances often exist in covariate distributions between different treatment groups even under completely randomized experiments. Not surprisingly, such covariate imbalances also occur in randomized survey experiments. Furthermore, the covariate imbalances happen not only between different treatment groups, but also between the sampled experimental units and the overall population of interest. In this article, we propose a two-stage rerandomization design that can actively avoid undesirable covariate imbalances at both the sampling and treatment assignment stages. We further develop asymptotic theory for rerandomized survey experiments, demonstrating that rerandomization provides better covariate balance, more precise treatment effect estimators, and shorter large-sample confidence intervals. We also propose covariate adjustment to deal with remaining covariate imbalances after rerandomization, showing that it can further improve both the sampling and estimated precision. Our work allows general relationship among covariates at the sampling, treatment assignment and analysis stages, and generalizes both rerandomization in classical randomized experiments and rejective sampling in survey sampling.
AB - Classical randomized experiments, equipped with randomization-based inference, provide assumption-free inference for treatment effects. They have been the gold standard for drawing causal inference and provide excellent internal validity. However, they have also been criticized for questionable external validity, in the sense that the conclusion may not generalize well to a larger population. The randomized survey experiment is a design tool that can help mitigate this concern, by randomly selecting the experimental units from the target population of interest. However, as pointed out by Morgan and Rubin, chance imbalances often exist in covariate distributions between different treatment groups even under completely randomized experiments. Not surprisingly, such covariate imbalances also occur in randomized survey experiments. Furthermore, the covariate imbalances happen not only between different treatment groups, but also between the sampled experimental units and the overall population of interest. In this article, we propose a two-stage rerandomization design that can actively avoid undesirable covariate imbalances at both the sampling and treatment assignment stages. We further develop asymptotic theory for rerandomized survey experiments, demonstrating that rerandomization provides better covariate balance, more precise treatment effect estimators, and shorter large-sample confidence intervals. We also propose covariate adjustment to deal with remaining covariate imbalances after rerandomization, showing that it can further improve both the sampling and estimated precision. Our work allows general relationship among covariates at the sampling, treatment assignment and analysis stages, and generalizes both rerandomization in classical randomized experiments and rejective sampling in survey sampling.
KW - Causal inference
KW - Covariate imbalance
KW - Mahalanobis distance
KW - Potential outcome
KW - Randomization-based inference
UR - http://www.scopus.com/inward/record.url?scp=85120612968&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85120612968&partnerID=8YFLogxK
U2 - 10.1080/01621459.2021.1984926
DO - 10.1080/01621459.2021.1984926
M3 - Article
AN - SCOPUS:85120612968
SN - 0162-1459
VL - 118
SP - 1207
EP - 1221
JO - Journal of the American Statistical Association
JF - Journal of the American Statistical Association
IS - 542
ER -