Is Your Multi-Stage Model Selection Strategy Affecting Your Inferences?

Ecologists routinely fit complex models with multiple parameters of interest, where hundreds or more competing models are plausible. To limit the number of fitted models, ecologists often employ model selection strategies with a series of stages, in which alternatives for certain features of a model are compared while other features are held constant. These multi-stage strategies operate via a series of decisions that may impact inferences, but which have not been critically evaluated. We tested the performance of different model selection strategies using four datasets and three model types. For each dataset, we determined the “true” distribution of AIC weights by fitting all plausible models. Then, we calculated the number of models that would have been fitted and the portion of “true” AIC weight we recovered under different model selection strategies. The sequential-by-submodel strategy compares alternatives for one sub-model (parameter) at a time, and fitting of subsequent submodels depends on the structures selected in previous stages; this strategy often performed poorly and was very sensitive to the order of stages. Two alternative strategies were more reliable: build-up (stages defined by increasing detail or complexity across submodels) and secondary candidate sets (submodel structures evaluated independently, and top set of model structures combined for selection in a final stage). Within-stage model selection criteria were also important, and strategies using thresholds of ΔAIC ≥ 5 to select structures to carry forward to subsequent stages performed better than those with stricter thresholds. Our results indicate that not all multi-stage model selection strategies are created equal, and the resulting inference may depend greatly on the strategy chosen. Multi-stage approaches cannot compensate for a lack of critical thought in selecting covariates and building models to represent competing a priori hypotheses. However, careful consideration of multi-stage strategies will improve consistency in inference over time and across studies.