Comparing machine learning to knowledge engineering for student behavior modeling: a case study in gaming the system

Learning analytics research has used both knowledge engineering and machine learning methods to model student behaviors within the context of digital learning environments. In this paper, we compare these two approaches, as well as a hybrid approach combining the two types of methods. We illustrate the strengths of each approach in the context of a case study in building models able to detect when students “game the system”, a behavior in which learners abuse the environment’s support functionalities in order to succeed by guessing or copying answers. We compare the predictive performance, interpretability and generalizability of models created using each approach, doing so across multiple intelligent tutoring systems. In our case study, we show that the machine-learned model required less resources to develop, but was less interpretable and general. In contrast, the knowledge engineering approach resulted in the most interpretable and general model. Combining both approaches in a hybrid model allowed us to create a model that performed best across the three dimensions, but requiring increased resources to develop.