Cognitive state classification in a spoken tutorial dialogue system

Tong Zhang, Mark Hasegawa-Johnson, Stephen E. Levinson

Research output: Contribution to journalArticlepeer-review


This paper addresses the manual and automatic labelling, from spontaneous speech, of a particular type of user affect that we call the cognitive state in a tutorial dialogue system with students of primary and early middle school ages. Our definition of the cognitive state is based on analysis of children's spontaneous speech, which is acquired during Wizard-of-Oz simulations of an intelligent math and physics tutor. The cognitive states of children are categorized into three classes: confidence, puzzlement, and hesitation. The manual labelling of cognitive states had an inter-transcriber agreement of kappa score 0.93. The automatic cognitive state labels are generated by classifying prosodic features, text features, and spectral features. Text features are generated from an automatic speech recognition (ASR) system; features include indicator functions of keyword classes and part-of-speech sequences. Spectral features are created based on acoustic likelihood scores of a cognitive state-dependent ASR system, in which phoneme models are adapted to utterances labelled for a particular cognitive state. The effectiveness of the proposed method has been tested on both manually and automatically transcribed speech, and the test yielded very high correctness: 96.6% for manually transcribed speech and 95.7% for automatically recognized speech. Our study shows that the proposed spectral features greatly outperformed the other types of features in the cognitive state classification experiments. Our study also shows that the spectral and prosodic features derived directly from speech signals were very robust to speech recognition errors, much more than the lexical and part-of-speech based features.

Original languageEnglish (US)
Pages (from-to)616-632
Number of pages17
JournalSpeech Communication
Issue number6
StatePublished - Jun 2006


  • Intelligent tutoring system
  • Spoken language processing
  • User affect recognition

ASJC Scopus subject areas

  • Software
  • Modeling and Simulation
  • Communication
  • Language and Linguistics
  • Linguistics and Language
  • Computer Vision and Pattern Recognition
  • Computer Science Applications


Dive into the research topics of 'Cognitive state classification in a spoken tutorial dialogue system'. Together they form a unique fingerprint.

Cite this