Automatic assessment of complex assignments using topic models

Saar Kuzi, William Cope, Duncan Ferguson, Chase Geigle, Cheng Xiang Zhai

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Automated assessment of complex assignments is crucial for scaling up learning of complex skills such as critical thinking. To address this challenge, one previous work has applied supervised machine learning to automate the assessment by learning from examples of graded assignments by humans. However, in the previous work, only simple lexical features, such as words or n-grams, have been used. In this paper, we propose to use topics as features for this task, which are more interpretable than those simple lexical features and can also address polysemy and synonymy of lexical semantics. The topics can be learned automatically from the student assignment data by using a probabilistic topic model. We propose and study multiple approaches to construct topical features and to combine topical features with simple lexical features. We evaluate the proposed methods using clinical case assignments performed by veterinary medicine students. The experimental results show that topical features are generally very effective and can substantially improve performance when added on top of the lexical features. However, their effectiveness is highly sensitive to how the topics are constructed and a combination of topics constructed using multiple views of the text data works the best. Our results also show that combining the prediction results of using different types of topical features and of topical and lexical features is more effective than pooling all features together to form a larger feature space.

Original languageEnglish (US)
Title of host publicationProceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450368049
DOIs
StatePublished - Jun 24 2019
Event6th ACM Conference on Learning at Scale, L@S 2019 - Chicago, United States
Duration: Jun 24 2019Jun 25 2019

Publication series

NameProceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019

Conference

Conference6th ACM Conference on Learning at Scale, L@S 2019
CountryUnited States
CityChicago
Period6/24/196/25/19

Fingerprint

Veterinary medicine
Students
learning
veterinary medicine
Learning systems
Semantics
scaling
student
semantics
performance
Statistical Models

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Education

Cite this

Kuzi, S., Cope, W., Ferguson, D., Geigle, C., & Zhai, C. X. (2019). Automatic assessment of complex assignments using topic models. In Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019 (Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019). Association for Computing Machinery, Inc. https://doi.org/10.1145/3330430.3333615

Automatic assessment of complex assignments using topic models. / Kuzi, Saar; Cope, William; Ferguson, Duncan; Geigle, Chase; Zhai, Cheng Xiang.

Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019. Association for Computing Machinery, Inc, 2019. (Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kuzi, S, Cope, W, Ferguson, D, Geigle, C & Zhai, CX 2019, Automatic assessment of complex assignments using topic models. in Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019. Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019, Association for Computing Machinery, Inc, 6th ACM Conference on Learning at Scale, L@S 2019, Chicago, United States, 6/24/19. https://doi.org/10.1145/3330430.3333615
Kuzi S, Cope W, Ferguson D, Geigle C, Zhai CX. Automatic assessment of complex assignments using topic models. In Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019. Association for Computing Machinery, Inc. 2019. (Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019). https://doi.org/10.1145/3330430.3333615
Kuzi, Saar ; Cope, William ; Ferguson, Duncan ; Geigle, Chase ; Zhai, Cheng Xiang. / Automatic assessment of complex assignments using topic models. Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019. Association for Computing Machinery, Inc, 2019. (Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019).
@inproceedings{9d8a70e0269d46b49ef81c54b727977b,
title = "Automatic assessment of complex assignments using topic models",
abstract = "Automated assessment of complex assignments is crucial for scaling up learning of complex skills such as critical thinking. To address this challenge, one previous work has applied supervised machine learning to automate the assessment by learning from examples of graded assignments by humans. However, in the previous work, only simple lexical features, such as words or n-grams, have been used. In this paper, we propose to use topics as features for this task, which are more interpretable than those simple lexical features and can also address polysemy and synonymy of lexical semantics. The topics can be learned automatically from the student assignment data by using a probabilistic topic model. We propose and study multiple approaches to construct topical features and to combine topical features with simple lexical features. We evaluate the proposed methods using clinical case assignments performed by veterinary medicine students. The experimental results show that topical features are generally very effective and can substantially improve performance when added on top of the lexical features. However, their effectiveness is highly sensitive to how the topics are constructed and a combination of topics constructed using multiple views of the text data works the best. Our results also show that combining the prediction results of using different types of topical features and of topical and lexical features is more effective than pooling all features together to form a larger feature space.",
author = "Saar Kuzi and William Cope and Duncan Ferguson and Chase Geigle and Zhai, {Cheng Xiang}",
year = "2019",
month = "6",
day = "24",
doi = "10.1145/3330430.3333615",
language = "English (US)",
series = "Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019",
publisher = "Association for Computing Machinery, Inc",
booktitle = "Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019",

}

TY - GEN

T1 - Automatic assessment of complex assignments using topic models

AU - Kuzi, Saar

AU - Cope, William

AU - Ferguson, Duncan

AU - Geigle, Chase

AU - Zhai, Cheng Xiang

PY - 2019/6/24

Y1 - 2019/6/24

N2 - Automated assessment of complex assignments is crucial for scaling up learning of complex skills such as critical thinking. To address this challenge, one previous work has applied supervised machine learning to automate the assessment by learning from examples of graded assignments by humans. However, in the previous work, only simple lexical features, such as words or n-grams, have been used. In this paper, we propose to use topics as features for this task, which are more interpretable than those simple lexical features and can also address polysemy and synonymy of lexical semantics. The topics can be learned automatically from the student assignment data by using a probabilistic topic model. We propose and study multiple approaches to construct topical features and to combine topical features with simple lexical features. We evaluate the proposed methods using clinical case assignments performed by veterinary medicine students. The experimental results show that topical features are generally very effective and can substantially improve performance when added on top of the lexical features. However, their effectiveness is highly sensitive to how the topics are constructed and a combination of topics constructed using multiple views of the text data works the best. Our results also show that combining the prediction results of using different types of topical features and of topical and lexical features is more effective than pooling all features together to form a larger feature space.

AB - Automated assessment of complex assignments is crucial for scaling up learning of complex skills such as critical thinking. To address this challenge, one previous work has applied supervised machine learning to automate the assessment by learning from examples of graded assignments by humans. However, in the previous work, only simple lexical features, such as words or n-grams, have been used. In this paper, we propose to use topics as features for this task, which are more interpretable than those simple lexical features and can also address polysemy and synonymy of lexical semantics. The topics can be learned automatically from the student assignment data by using a probabilistic topic model. We propose and study multiple approaches to construct topical features and to combine topical features with simple lexical features. We evaluate the proposed methods using clinical case assignments performed by veterinary medicine students. The experimental results show that topical features are generally very effective and can substantially improve performance when added on top of the lexical features. However, their effectiveness is highly sensitive to how the topics are constructed and a combination of topics constructed using multiple views of the text data works the best. Our results also show that combining the prediction results of using different types of topical features and of topical and lexical features is more effective than pooling all features together to form a larger feature space.

UR - http://www.scopus.com/inward/record.url?scp=85071320908&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071320908&partnerID=8YFLogxK

U2 - 10.1145/3330430.3333615

DO - 10.1145/3330430.3333615

M3 - Conference contribution

AN - SCOPUS:85071320908

T3 - Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019

BT - Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019

PB - Association for Computing Machinery, Inc

ER -