PrairieLearn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning

PrairieLearn: Mastery-based Online Problem Solving with Adaptive Scoring and Recommendations Driven by Machine LearningWe present an online problem-solving system (PrairieLearn) that is designed to facilitatelearning to mastery. The objectives of this system are to: (1) enable students to practicesolving randomized problem variants repeatedly until mastery, (2) incentivize students torepeat questions until mastery is achieved, and (3) provide immediate feedback abouttheir current mastery level to the student.To achieve these objectives, we implemented an open-source web-based online systemcalled PrairieLearn, which consists of a Node.js server and a JavaScript web-app topresent randomized question variants to students. As students attempt questions, thesystem uses Bayesian estimation on a four-parameter item-response model to computethe real-time maximum-likelihood estimate of the student’s ability on the current topic.This estimate is shown to the student as a “mastery score”, which they can increase bysolving questions correctly. Because the mastery score is based on an estimate of studentproblem-solving ability, solving each question will result in a different change inmastery. For example, if a student has a high mastery, then successfully solving an easyquestion will only increase their estimated mastery by a small amount. These per-question mastery changes from answering questions are dynamically pre-calculated foreach question and shown to students as “question scores”, which thus adaptively changein response to the student’s performance. Additionally, the expected value of masteryincrease for each question is computed and reflected to the student as a recommendationfor which question they should attempt next. Finally, the system recoreds all questionattempts by students and processes this offline to learn improved models for predictingstudent mastery via maximum likelihood optimization.The results of using PrairieLearn over several semesters in a large engineering course(Introductory Dynamics) include: (1) significant gains in student mastery, as measured byexam results and concept inventory questions, (2) improved student satisfaction whencompared to existing online problem-solving systems, and (3) high instructor satisfaction.We present data derived from students’ usage of PrairieLearn, as well as from studentsurveys and focus groups.