Strategies for deploying unreliable AI graders in high-transparency high-stakes exams

Sushmita Azad, Binglin Chen, Maxwell Fowler, Matthew West, Craig Zilles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We describe the deployment of an imperfect NLP-based automatic short answer grading system on an exam in a large-enrollment introductory college course. We characterize this deployment as both high stakes (the questions were on an mid-term exam worth 10% of students’ final grade) and high transparency (the question was graded interactively during the computer-based exam and correct solutions were shown to students that could be compared to their answer). We study two techniques designed to mitigate the potential student dissatisfaction resulting from students incorrectly not granted credit by the imperfect AI grader. We find (1) that providing multiple attempts can eliminate first-attempt false negatives at the cost of additional false positives, and (2) that students not granted credit from the algorithm cannot reliably determine if their answer was mis-scored.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education- 21st International Conference, AIED 2020, Proceedings, Part I
EditorsIg Ibert Bittencourt, Mutlu Cukurova, Rose Luckin, Kasia Muldner, Eva Millán
PublisherSpringer
Pages16-28
Number of pages13
ISBN (Print)9783030522360
DOIs
StatePublished - 2020
Event21st International Conference on Artificial Intelligence in Education, AIED 2020 - Ifrane, Morocco
Duration: Jul 6 2020Jul 10 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12163 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference21st International Conference on Artificial Intelligence in Education, AIED 2020
CountryMorocco
CityIfrane
Period7/6/207/10/20

Keywords

  • Automatic short answer grading
  • CS1
  • Code reading
  • Computer-based exams
  • EiPE
  • Transparency

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Strategies for deploying unreliable AI graders in high-transparency high-stakes exams'. Together they form a unique fingerprint.

Cite this