Peer-grading "explain in Plain English" Questions: A Bayesian Calibration Method for Categorical Answers

"Explain in plain English'' (EipE) questions have been proposed as an important activity and assessment for studying novice programmers' grasp of programming knowledge and their ability to communicate their understanding. However, EipE questions aren't widely used in introductory programming courses in part because of the large grading effort required. In this paper, we present our experience of using peer grading for EipE questions in a large-enrollment introductory programming course, where students were asked to categorize other students' responses. We developed a novel Bayesian algorithm for performing calibrated peer grading on categorical data, and we used a heuristic grade assignment method based on the Bayesian estimates. The peer-grading exercises served both as a way to coach students on what is expected from EipE questions and as a way to alleviate the grading load for the course staff. Based on four rounds of peer-grading activities, we found that students are generally capable of categorizing responses to EiPE questions and that our proposed Bayesian method is more robust than unweighted voting.