Parsimonious Testing of Transitive or Intransitive Preferences: Reply to Birnbaum (2011)

Birnbaum (2011) raised important challenges to testing transitivity. We summarize why an approach based on counting response patterns does not solve these challenges. Foremost, we show why parsimonious tests of transitivity require at least 5 choice alternatives. While the approach of Regenwetter, Dana, and Davis-Stober (2011) achieves high power with modest sample sizes for 5 alternatives, pattern-counting approaches face the difficulty of combinatoric explosion in permissible response patterns. Even for fewer than 5 alternatives, if the choice of how to " block" individual responses into response patterns is slightly mistaken, intransitive preferences can mimic transitive ones. At the same time, statistical tests on proportions of response patterns rely on similar " independent and identically distributed" sampling assumptions as tests based on response proportions. For example, the hypothetical data of Birnbaum (2011, Tables 2 and 3) hinge on the assumption that response patterns are properly blocked, as well as sampled independently and with a stationary distribution. We test an intransitive lexicographic semiorder model on Tversky's (1969) and Regenwetter et al.'s data and, consistent with Birnbaum's (2011) concern, we find evidence for model mimicry in some cases.