TY - JOUR

T1 - What's next? sequence length and impossible loops in state transition measurement

AU - Bosch, Nigel

AU - Paquette, Luc

N1 - Publisher Copyright:
© 2021. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

PY - 2021/6/30

Y1 - 2021/6/30

N2 - Transition metrics, which quantify the propensity for one event to follow another, are often utilized to study sequential patterns of behaviors, emotions, actions, and other states. However, little is known about the conditions in which application of transition metrics is appropriate. We report on two experiments in which we simulated sequences of states to explore the properties of common transition metrics (conditional probability, D'Mello's L, lag sequential analysis, and Yule's Q) where results should be null (i.e., random sequences). In experiment 1, we found that transition metrics produced statistically significant results with non-null effect sizes (e.g., Q > 0.2) when sequences of states were short. In experiment 2, we explored situations where consecutively repeated states (i.e., loops, or self-transitions) are impossible - e.g., in digital learning environments where actions such as hint requests cannot be made twice in a row. We found that impossible loops affected all transition metrics (e.g., Q = .646). Based on simulations, we recommend sequences of length 50 or more for transition metric analyses. Our software for calculating transition metrics and running simulated experiments is publicly available.

AB - Transition metrics, which quantify the propensity for one event to follow another, are often utilized to study sequential patterns of behaviors, emotions, actions, and other states. However, little is known about the conditions in which application of transition metrics is appropriate. We report on two experiments in which we simulated sequences of states to explore the properties of common transition metrics (conditional probability, D'Mello's L, lag sequential analysis, and Yule's Q) where results should be null (i.e., random sequences). In experiment 1, we found that transition metrics produced statistically significant results with non-null effect sizes (e.g., Q > 0.2) when sequences of states were short. In experiment 2, we explored situations where consecutively repeated states (i.e., loops, or self-transitions) are impossible - e.g., in digital learning environments where actions such as hint requests cannot be made twice in a row. We found that impossible loops affected all transition metrics (e.g., Q = .646). Based on simulations, we recommend sequences of length 50 or more for transition metric analyses. Our software for calculating transition metrics and running simulated experiments is publicly available.

KW - Sequential analysis

KW - Simulated data

KW - Transition metrics

UR - http://www.scopus.com/inward/record.url?scp=85109181836&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85109181836&partnerID=8YFLogxK

U2 - 10.5281/zenodo.5048423

DO - 10.5281/zenodo.5048423

M3 - Article

AN - SCOPUS:85109181836

VL - 13

SP - 1

EP - 23

JO - Journal of Educational Data Mining

JF - Journal of Educational Data Mining

SN - 2157-2100

IS - 1

ER -