Towards a model-free estimate of the limits to student modeling accuracy

Binglin Chen, Matthew West, Craig Zilles

Research output: Contribution to conferencePaper

Abstract

This paper attempts to quantify the accuracy limit of “next-item-correct” prediction by using numerical optimization to estimate the student’s probability of getting each question correct given a complete sequence of item responses. This optimization is performed without an explicit parameterized model of student behavior, but with the constraint that a student’s likelihood of getting a problem correct only increases or remains unchanged with additional practice (i.e., no forgetting). We present results for this method for the Assistments 2009–2010 data where it suggests that there is only modest opportunity for improvement beyond the state of the art predictors. Furthermore, we describe a framework for applying this method to datasets where problems can be tagged with multiple skills and problem difficulties. Lastly, we discuss the limitations of this method, specifically its inability to give tight bounds on short sequences.

Original languageEnglish (US)
StatePublished - Jan 1 2018
Event11th International Conference on Educational Data Mining, EDM 2018 - Buffalo, United States
Duration: Jul 15 2018Jul 18 2018

Conference

Conference11th International Conference on Educational Data Mining, EDM 2018
CountryUnited States
CityBuffalo
Period7/15/187/18/18

Fingerprint

Students

ASJC Scopus subject areas

  • Computer Science Applications
  • Information Systems

Cite this

Chen, B., West, M., & Zilles, C. (2018). Towards a model-free estimate of the limits to student modeling accuracy. Paper presented at 11th International Conference on Educational Data Mining, EDM 2018, Buffalo, United States.

Towards a model-free estimate of the limits to student modeling accuracy. / Chen, Binglin; West, Matthew; Zilles, Craig.

2018. Paper presented at 11th International Conference on Educational Data Mining, EDM 2018, Buffalo, United States.

Research output: Contribution to conferencePaper

Chen, B, West, M & Zilles, C 2018, 'Towards a model-free estimate of the limits to student modeling accuracy' Paper presented at 11th International Conference on Educational Data Mining, EDM 2018, Buffalo, United States, 7/15/18 - 7/18/18, .
Chen B, West M, Zilles C. Towards a model-free estimate of the limits to student modeling accuracy. 2018. Paper presented at 11th International Conference on Educational Data Mining, EDM 2018, Buffalo, United States.
Chen, Binglin ; West, Matthew ; Zilles, Craig. / Towards a model-free estimate of the limits to student modeling accuracy. Paper presented at 11th International Conference on Educational Data Mining, EDM 2018, Buffalo, United States.
@conference{7755eeea145e4a858453b615c0fcc0e5,
title = "Towards a model-free estimate of the limits to student modeling accuracy",
abstract = "This paper attempts to quantify the accuracy limit of “next-item-correct” prediction by using numerical optimization to estimate the student’s probability of getting each question correct given a complete sequence of item responses. This optimization is performed without an explicit parameterized model of student behavior, but with the constraint that a student’s likelihood of getting a problem correct only increases or remains unchanged with additional practice (i.e., no forgetting). We present results for this method for the Assistments 2009–2010 data where it suggests that there is only modest opportunity for improvement beyond the state of the art predictors. Furthermore, we describe a framework for applying this method to datasets where problems can be tagged with multiple skills and problem difficulties. Lastly, we discuss the limitations of this method, specifically its inability to give tight bounds on short sequences.",
author = "Binglin Chen and Matthew West and Craig Zilles",
year = "2018",
month = "1",
day = "1",
language = "English (US)",
note = "11th International Conference on Educational Data Mining, EDM 2018 ; Conference date: 15-07-2018 Through 18-07-2018",

}

TY - CONF

T1 - Towards a model-free estimate of the limits to student modeling accuracy

AU - Chen, Binglin

AU - West, Matthew

AU - Zilles, Craig

PY - 2018/1/1

Y1 - 2018/1/1

N2 - This paper attempts to quantify the accuracy limit of “next-item-correct” prediction by using numerical optimization to estimate the student’s probability of getting each question correct given a complete sequence of item responses. This optimization is performed without an explicit parameterized model of student behavior, but with the constraint that a student’s likelihood of getting a problem correct only increases or remains unchanged with additional practice (i.e., no forgetting). We present results for this method for the Assistments 2009–2010 data where it suggests that there is only modest opportunity for improvement beyond the state of the art predictors. Furthermore, we describe a framework for applying this method to datasets where problems can be tagged with multiple skills and problem difficulties. Lastly, we discuss the limitations of this method, specifically its inability to give tight bounds on short sequences.

AB - This paper attempts to quantify the accuracy limit of “next-item-correct” prediction by using numerical optimization to estimate the student’s probability of getting each question correct given a complete sequence of item responses. This optimization is performed without an explicit parameterized model of student behavior, but with the constraint that a student’s likelihood of getting a problem correct only increases or remains unchanged with additional practice (i.e., no forgetting). We present results for this method for the Assistments 2009–2010 data where it suggests that there is only modest opportunity for improvement beyond the state of the art predictors. Furthermore, we describe a framework for applying this method to datasets where problems can be tagged with multiple skills and problem difficulties. Lastly, we discuss the limitations of this method, specifically its inability to give tight bounds on short sequences.

UR - http://www.scopus.com/inward/record.url?scp=85072292658&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072292658&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85072292658

ER -