Comparative evaluation of automated scoring of syntactic competence of non-native speakers

Klaus Zechner, Su Youn Yoon, Suma Bhat, Chee Wee Leong

Research output: Contribution to journalArticlepeer-review

Abstract

Syntactic competence, especially the ability to use a wide range of sophisticated grammatical expressions, represents an important aspect of communicative acumen. This paper explores the question of how to best evaluate the syntactic competence of non-native speakers in an automated way. Using spoken responses of test takers participating in an English practice assessment, three classes of grammatical features – features based on n-grams of part-of-speech tags (POS), features based on various clause types, and features based on various phrases – are compared in an end-to-end assessment system. Feature correlations with human proficiency scores show that POS features and phrase features exhibit the highest correlations with human scores. Including these three classes of grammar features in a baseline scoring model that measures various aspects of spoken proficiency excluding aspects of grammar, we find substantial increases in agreement between machine and human scores. Finally, we discuss the broader implications of our results on the design of automatic scoring systems for spoken language.

Original languageEnglish (US)
Pages (from-to)672-682
Number of pages11
JournalComputers in Human Behavior
Volume76
DOIs
StatePublished - Nov 2017

Keywords

  • Automated scoring
  • Automated speech recognition
  • English language assessment

ASJC Scopus subject areas

  • Arts and Humanities (miscellaneous)
  • Human-Computer Interaction
  • Psychology(all)

Fingerprint Dive into the research topics of 'Comparative evaluation of automated scoring of syntactic competence of non-native speakers'. Together they form a unique fingerprint.

Cite this