Eyetracking in cognitive state detection for HCI

Darrell S. Rudmann, George W. McConkie, Xianjun Sam Zheng

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Past research in a number of fields confirms the existence of a link between cognition and eye movement control, beyond simply a pointing relationship. This being the case, it should be possible to use eye movement recording as a basis for detecting users' cognitive states in real time. Several examples of such cognitive state detectors have been reported in the literature. A multi-disciplinary project is described in which the goal is to provide the computer with as much real-time information about the human state (cognitive, affective and motivational state) as possible, and to base computer actions on this information. The application area in which this is being implemented is science education, learning about gears through exploration. Two studies are reported in which participants solve simple problems of pictured gear trains while their eye movements are recorded. The first study indicates that most eye movement sequences are compatible with predictions of a simple sequential cognitive model, and it is suggested that those sequences that do not fit the model may be of particular interest in the HCI context as indicating problems or alternative mental strategies. The mental rotation of gears sometimes produces sequences of short eye movements in the direction of motion; thus, such sequences may be useful as cognitive state detectors. The second study tested the hypothesis that participants are thinking about the object to which their eyes are directed. In this study, the display was turned off partway through the process of solving a problem, and the participants reported what they were thinking about at that time. While in most cases the participants reported cognitive activities involving the fixated object, this was not the case on a sizeable number of trials.

Original languageEnglish (US)
Title of host publicationICMI'03
Subtitle of host publicationFifth International Conference on Multimodal Interfaces
PublisherAssociation for Computing Machinery
Number of pages5
ISBN (Print)1581136218, 9781581136210
StatePublished - 2003
EventICMI'03: Fifth International Conference on Multimodal Interfaces - Vancouver, BC, Canada
Duration: Nov 5 2003Nov 7 2003

Publication series

NameICMI'03: Fifth International Conference on Multimodal Interfaces


ConferenceICMI'03: Fifth International Conference on Multimodal Interfaces
CityVancouver, BC


  • Cognitive State
  • Eye Tracking

ASJC Scopus subject areas

  • Engineering(all)


Dive into the research topics of 'Eyetracking in cognitive state detection for HCI'. Together they form a unique fingerprint.

Cite this