TY - GEN
T1 - I'm Sure! Automatic Detection of Metacognition in Online Course Discussion Forums
AU - Huang, Eddie
AU - Valdiviejas, Hannah
AU - Bosch, Nigel
N1 - Publisher Copyright:
© 2019 IEEE.
PY - 2019/9
Y1 - 2019/9
N2 - Metacognition is a valuable tool for learning, since it is closely related to self-regulation and awareness of one's own affect. However, methods for automatically detecting and studying metacognition are scarce. Thus, in this paper we describe an algorithm for automatic detection of metacognitive language in writing. We analyzed text from the forums of two online, university-level science courses, which revealed common patterns of phrases that we used for automatic metacognition detection. The algorithm we developed exhibited high accuracy on expert-labeled metacognitive phrases (Spearman's rho = 0.878 and Cohen's kappa = 0.792), and provides a reliable, fast method for automatically annotating text corpora that are too large for manual annotation. We applied this algorithm to analyze relationships between students' metacognitive language and their academic performance, finding small correlations with course grade and medium-sized differences in metacognition across courses. We discuss how our algorithm can be used to advance metacognitive studies and online educational systems.
AB - Metacognition is a valuable tool for learning, since it is closely related to self-regulation and awareness of one's own affect. However, methods for automatically detecting and studying metacognition are scarce. Thus, in this paper we describe an algorithm for automatic detection of metacognitive language in writing. We analyzed text from the forums of two online, university-level science courses, which revealed common patterns of phrases that we used for automatic metacognition detection. The algorithm we developed exhibited high accuracy on expert-labeled metacognitive phrases (Spearman's rho = 0.878 and Cohen's kappa = 0.792), and provides a reliable, fast method for automatically annotating text corpora that are too large for manual annotation. We applied this algorithm to analyze relationships between students' metacognitive language and their academic performance, finding small correlations with course grade and medium-sized differences in metacognition across courses. We discuss how our algorithm can be used to advance metacognitive studies and online educational systems.
KW - Learning analytics
KW - Metacognition
KW - Natural language processing
UR - http://www.scopus.com/inward/record.url?scp=85077798723&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85077798723&partnerID=8YFLogxK
U2 - 10.1109/ACII.2019.8925506
DO - 10.1109/ACII.2019.8925506
M3 - Conference contribution
AN - SCOPUS:85077798723
T3 - 2019 8th International Conference on Affective Computing and Intelligent Interaction, ACII 2019
SP - 241
EP - 247
BT - 2019 8th International Conference on Affective Computing and Intelligent Interaction, ACII 2019
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 8th International Conference on Affective Computing and Intelligent Interaction, ACII 2019
Y2 - 3 September 2019 through 6 September 2019
ER -