Reply: Birnbaum's (2012) statistical tests of independence have unknown Type-I error rates and do not replicate within participant

Yun shil Cha, Michelle Choi, Ying Guo, Michel Regenwetter, Chris Zwilling

Research output: Contribution to journalArticlepeer-review


Birnbaum (2011, 2012) questioned the iid (independent and identically distributed) sampling assumptions used by state-of-the-art statistical tests in Regenwetter, Dana and Davis-Stober's (2010, 2011) analysis of the "linear order model". Birnbaum (2012) cited, but did not use, a test of iid by Smith and Batchelder (2008) with analytically known properties. Instead, he created two new test statistics with unknown sampling distributions. Our rebuttal has five components: 1) We demonstrate that the Regenwetter et al. data pass Smith and Batchelder's test of iid with flying colors. 2) We provide evidence from Monte Carlo simulations that Birnbaum's (2012) proposed tests have unknown Type-I error rates, which depend on the actual choice probabilities and on how data are coded as well as on the null hypothesis of iid sampling. 3) Birnbaum analyzed only a third of Regenwetter et al.'s data. We show that his two new tests fail to replicate on the other two-thirds of the data, within participants. 4) Birnbaum selectively picked data of one respondent to suggest that choice probabilities may have changed partway into the experiment. Such nonstationarity could potentially cause a seemingly good fit to be a Type-II error. We show that the linear order model fits equally well if we allow for warm-up effects. 5) Using hypothetical data, Birnbaum (2012) claimed to show that "true- and-error" models for binary pattern probabilities overcome the alleged short-comings of Regenwetter et al.'s approach. We disprove this claim on the same data.

Original languageEnglish (US)
Pages (from-to)55-73
Number of pages19
JournalJudgment and Decision Making
Issue number1
StatePublished - Jan 2013


  • Binary choice models
  • Iid sampling
  • Statistical testing
  • True-and-error models

ASJC Scopus subject areas

  • General Decision Sciences
  • Applied Psychology
  • Economics and Econometrics


Dive into the research topics of 'Reply: Birnbaum's (2012) statistical tests of independence have unknown Type-I error rates and do not replicate within participant'. Together they form a unique fingerprint.

Cite this