Predicting the difficulty of automatic item generators on exams from their difficulty on homeworks

Binglin Chen; Matthew West; Craig Zilles

doi:10.1145/3330430.3333647

Predicting the difficulty of automatic item generators on exams from their difficulty on homeworks

Binglin Chen, Matthew West, Craig Zilles

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

To design good assessments, it is useful to have an estimate of the difficulty of a novel exam question before running an exam. In this paper, we study a collection of a few hundred automatic item generators (short computer programs that generate a variety of unique item instances) and show that their exam difficulty can be roughly predicted from student performance on the same generator during pre-exam practice. Specifically, we show that the rate that students correctly respond to a generator on an exam is on average within 5% of the correct rate for those students on their last practice attempt. This study is conducted with data from introductory undergraduate Computer Science and Mechanical Engineering courses.

Original language	English (US)
Title of host publication	Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019
Publisher	Association for Computing Machinery
ISBN (Electronic)	9781450368049
DOIs	https://doi.org/10.1145/3330430.3333647
State	Published - Jun 24 2019
Event	6th ACM Conference on Learning at Scale, L@S 2019 - Chicago, United States Duration: Jun 24 2019 → Jun 25 2019

Publication series

Name	Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019

Conference

Conference	6th ACM Conference on Learning at Scale, L@S 2019
Country/Territory	United States
City	Chicago
Period	6/24/19 → 6/25/19

ASJC Scopus subject areas

Software
Computer Science Applications
Computer Networks and Communications
Education

Online availability

10.1145/3330430.3333647

Library availability

Discover UIUC Full Text

Cite this

Chen, B., West, M., & Zilles, C. (2019). Predicting the difficulty of automatic item generators on exams from their difficulty on homeworks. In Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019 (Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019). Association for Computing Machinery. https://doi.org/10.1145/3330430.3333647

Predicting the difficulty of automatic item generators on exams from their difficulty on homeworks. / Chen, Binglin; West, Matthew ; Zilles, Craig.
Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019. Association for Computing Machinery, 2019. (Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Chen, B, West, M & Zilles, C 2019, Predicting the difficulty of automatic item generators on exams from their difficulty on homeworks. in Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019. Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019, Association for Computing Machinery, 6th ACM Conference on Learning at Scale, L@S 2019, Chicago, United States, 6/24/19. https://doi.org/10.1145/3330430.3333647

@inproceedings{b77609c346eb413a970f3fe40412d5d2,

title = "Predicting the difficulty of automatic item generators on exams from their difficulty on homeworks",

abstract = "To design good assessments, it is useful to have an estimate of the difficulty of a novel exam question before running an exam. In this paper, we study a collection of a few hundred automatic item generators (short computer programs that generate a variety of unique item instances) and show that their exam difficulty can be roughly predicted from student performance on the same generator during pre-exam practice. Specifically, we show that the rate that students correctly respond to a generator on an exam is on average within 5% of the correct rate for those students on their last practice attempt. This study is conducted with data from introductory undergraduate Computer Science and Mechanical Engineering courses.",

author = "Binglin Chen and Matthew West and Craig Zilles",

note = "Publisher Copyright: {\textcopyright} 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM.; 6th ACM Conference on Learning at Scale, L@S 2019 ; Conference date: 24-06-2019 Through 25-06-2019",

year = "2019",

month = jun,

day = "24",

doi = "10.1145/3330430.3333647",

language = "English (US)",

series = "Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019",

publisher = "Association for Computing Machinery",

booktitle = "Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019",

address = "United States",

}

TY - GEN

T1 - Predicting the difficulty of automatic item generators on exams from their difficulty on homeworks

AU - Chen, Binglin

AU - West, Matthew

AU - Zilles, Craig

PY - 2019/6/24

Y1 - 2019/6/24

N2 - To design good assessments, it is useful to have an estimate of the difficulty of a novel exam question before running an exam. In this paper, we study a collection of a few hundred automatic item generators (short computer programs that generate a variety of unique item instances) and show that their exam difficulty can be roughly predicted from student performance on the same generator during pre-exam practice. Specifically, we show that the rate that students correctly respond to a generator on an exam is on average within 5% of the correct rate for those students on their last practice attempt. This study is conducted with data from introductory undergraduate Computer Science and Mechanical Engineering courses.

AB - To design good assessments, it is useful to have an estimate of the difficulty of a novel exam question before running an exam. In this paper, we study a collection of a few hundred automatic item generators (short computer programs that generate a variety of unique item instances) and show that their exam difficulty can be roughly predicted from student performance on the same generator during pre-exam practice. Specifically, we show that the rate that students correctly respond to a generator on an exam is on average within 5% of the correct rate for those students on their last practice attempt. This study is conducted with data from introductory undergraduate Computer Science and Mechanical Engineering courses.

UR - http://www.scopus.com/inward/record.url?scp=85083951163&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85083951163&partnerID=8YFLogxK

U2 - 10.1145/3330430.3333647

DO - 10.1145/3330430.3333647

M3 - Conference contribution

AN - SCOPUS:85083951163

T3 - Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019

BT - Proceedings of the 6th 2019 ACM Conference on Learning at Scale, L@S 2019

PB - Association for Computing Machinery

T2 - 6th ACM Conference on Learning at Scale, L@S 2019

Y2 - 24 June 2019 through 25 June 2019

ER -

Predicting the difficulty of automatic item generators on exams from their difficulty on homeworks

Abstract

Publication series

Conference

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this