Code Generation Based Grading: Evaluating an Auto-grading Mechanism for "Explain-in-Plain-English" Questions

David H. Smith, Craig Zilles

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Comprehending and conveying the purpose of code is often cited as being a key learning objective within introductory programming courses. To address this objective, "Explain in Plain English'' questions, where students are shown a segment of code and asked to provide an abstract description of the code's purpose, have been adopted. However, given EiPE questions require a natural language response, they often require manual grading which is time-consuming for course staff and delays feedback for students. With the advent of large language models (LLMs) capable of generating code, responses to EiPE questions can be used to generate code segments, the correctness of which can then be easily verified using test cases. We refer to this approach as "Code Generation Based Grading'' (CGBG) and in this paper we explore its agreement with human graders using EiPE responses from past exams in an introductory programming course taught in Python. Overall, we find that all CGBG approaches achieve moderate agreement with human graders with the primary area of disagreement being its leniency with respect to low-level and line-by-line descriptions of code.

Original languageEnglish (US)
Title of host publicationITiCSE 2024 - Proceedings of the 2024 Conference Innovation and Technology in Computer Science Education
PublisherAssociation for Computing Machinery
Pages171-177
Number of pages7
ISBN (Electronic)9798400706004
DOIs
StatePublished - Jul 3 2024
Event29th Conference Innovation and Technology in Computer Science Education, ITiCSE 2024 - Milan, Italy
Duration: Jul 8 2024Jul 10 2024

Publication series

NameAnnual Conference on Innovation and Technology in Computer Science Education, ITiCSE
Volume1
ISSN (Print)1942-647X

Conference

Conference29th Conference Innovation and Technology in Computer Science Education, ITiCSE 2024
Country/TerritoryItaly
CityMilan
Period7/8/247/10/24

Keywords

  • auto-grading
  • eipe
  • gpt-4
  • large language models

ASJC Scopus subject areas

  • Management of Technology and Innovation
  • Education

Fingerprint

Dive into the research topics of 'Code Generation Based Grading: Evaluating an Auto-grading Mechanism for "Explain-in-Plain-English" Questions'. Together they form a unique fingerprint.

Cite this