Abstract

Automatic item generation enables a diverse array of questions to be generated through the use of question templates and randomly-selected parameters. Such automatic item generators are most useful if the generated item instances are of either equivalent or predictable difficulty. In this study, we analyzed student performance on over 300 item generators from four university-level STEM classes collected over a period of two years. In most cases, we find that the choice of parameters fails to significantly affect the problem difficulty. In our analysis, we found it useful to distinguish parameters that were drawn from a small number (<10) of values from those that are drawn from a large—often continuous—range of values. We observed that values from smaller ranges were more likely to significantly impact difficulty, as sometimes they represented different configurations of the problem (e.g., upward force vs. downward force). Through manual review of the problems with significant difficulty variance, we found it was, in general, easy to understand the source of the variance once the data were presented. These results suggest that the use of automatic item generation by college faculty is warranted, because most problems don’t exhibit significant difficulty variation, and the few that do can be detected through automatic means and addressed by the faculty member.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings
EditorsSeiji Isotani, Peter Hastings, Amy Ogan, Bruce McLaren, Eva Millán, Rose Luckin
PublisherSpringer-Verlag
Pages71-83
Number of pages13
ISBN (Print)9783030232030
DOIs
StatePublished - Jan 1 2019
Event20th International Conference on Artificial Intelligence in Education, AIED 2019 - Chicago, United States
Duration: Jun 25 2019Jun 29 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11625 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference20th International Conference on Artificial Intelligence in Education, AIED 2019
CountryUnited States
CityChicago
Period6/25/196/29/19

Fingerprint

Students
Generator
Template
Likely
Configuration
Range of data
Class
Review
Universities

Keywords

  • Automatic item generation
  • Item models

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Chen, B., Zilles, C., West, M., & Bretl, T. W. (2019). Effect of discrete and continuous parameter variation on difficulty in automatic item generation. In S. Isotani, P. Hastings, A. Ogan, B. McLaren, E. Millán, & R. Luckin (Eds.), Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings (pp. 71-83). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11625 LNAI). Springer-Verlag. https://doi.org/10.1007/978-3-030-23204-7_7

Effect of discrete and continuous parameter variation on difficulty in automatic item generation. / Chen, Binglin; Zilles, Craig; West, Matthew; Bretl, Timothy Wolfe.

Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings. ed. / Seiji Isotani; Peter Hastings; Amy Ogan; Bruce McLaren; Eva Millán; Rose Luckin. Springer-Verlag, 2019. p. 71-83 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11625 LNAI).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Chen, B, Zilles, C, West, M & Bretl, TW 2019, Effect of discrete and continuous parameter variation on difficulty in automatic item generation. in S Isotani, P Hastings, A Ogan, B McLaren, E Millán & R Luckin (eds), Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11625 LNAI, Springer-Verlag, pp. 71-83, 20th International Conference on Artificial Intelligence in Education, AIED 2019, Chicago, United States, 6/25/19. https://doi.org/10.1007/978-3-030-23204-7_7
Chen B, Zilles C, West M, Bretl TW. Effect of discrete and continuous parameter variation on difficulty in automatic item generation. In Isotani S, Hastings P, Ogan A, McLaren B, Millán E, Luckin R, editors, Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings. Springer-Verlag. 2019. p. 71-83. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-23204-7_7
Chen, Binglin ; Zilles, Craig ; West, Matthew ; Bretl, Timothy Wolfe. / Effect of discrete and continuous parameter variation on difficulty in automatic item generation. Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings. editor / Seiji Isotani ; Peter Hastings ; Amy Ogan ; Bruce McLaren ; Eva Millán ; Rose Luckin. Springer-Verlag, 2019. pp. 71-83 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{c791279ff9294bba98179dfc0619c922,
title = "Effect of discrete and continuous parameter variation on difficulty in automatic item generation",
abstract = "Automatic item generation enables a diverse array of questions to be generated through the use of question templates and randomly-selected parameters. Such automatic item generators are most useful if the generated item instances are of either equivalent or predictable difficulty. In this study, we analyzed student performance on over 300 item generators from four university-level STEM classes collected over a period of two years. In most cases, we find that the choice of parameters fails to significantly affect the problem difficulty. In our analysis, we found it useful to distinguish parameters that were drawn from a small number (<10) of values from those that are drawn from a large—often continuous—range of values. We observed that values from smaller ranges were more likely to significantly impact difficulty, as sometimes they represented different configurations of the problem (e.g., upward force vs. downward force). Through manual review of the problems with significant difficulty variance, we found it was, in general, easy to understand the source of the variance once the data were presented. These results suggest that the use of automatic item generation by college faculty is warranted, because most problems don’t exhibit significant difficulty variation, and the few that do can be detected through automatic means and addressed by the faculty member.",
keywords = "Automatic item generation, Item models",
author = "Binglin Chen and Craig Zilles and Matthew West and Bretl, {Timothy Wolfe}",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/978-3-030-23204-7_7",
language = "English (US)",
isbn = "9783030232030",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer-Verlag",
pages = "71--83",
editor = "Seiji Isotani and Peter Hastings and Amy Ogan and Bruce McLaren and Eva Mill{\'a}n and Rose Luckin",
booktitle = "Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings",

}

TY - GEN

T1 - Effect of discrete and continuous parameter variation on difficulty in automatic item generation

AU - Chen, Binglin

AU - Zilles, Craig

AU - West, Matthew

AU - Bretl, Timothy Wolfe

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Automatic item generation enables a diverse array of questions to be generated through the use of question templates and randomly-selected parameters. Such automatic item generators are most useful if the generated item instances are of either equivalent or predictable difficulty. In this study, we analyzed student performance on over 300 item generators from four university-level STEM classes collected over a period of two years. In most cases, we find that the choice of parameters fails to significantly affect the problem difficulty. In our analysis, we found it useful to distinguish parameters that were drawn from a small number (<10) of values from those that are drawn from a large—often continuous—range of values. We observed that values from smaller ranges were more likely to significantly impact difficulty, as sometimes they represented different configurations of the problem (e.g., upward force vs. downward force). Through manual review of the problems with significant difficulty variance, we found it was, in general, easy to understand the source of the variance once the data were presented. These results suggest that the use of automatic item generation by college faculty is warranted, because most problems don’t exhibit significant difficulty variation, and the few that do can be detected through automatic means and addressed by the faculty member.

AB - Automatic item generation enables a diverse array of questions to be generated through the use of question templates and randomly-selected parameters. Such automatic item generators are most useful if the generated item instances are of either equivalent or predictable difficulty. In this study, we analyzed student performance on over 300 item generators from four university-level STEM classes collected over a period of two years. In most cases, we find that the choice of parameters fails to significantly affect the problem difficulty. In our analysis, we found it useful to distinguish parameters that were drawn from a small number (<10) of values from those that are drawn from a large—often continuous—range of values. We observed that values from smaller ranges were more likely to significantly impact difficulty, as sometimes they represented different configurations of the problem (e.g., upward force vs. downward force). Through manual review of the problems with significant difficulty variance, we found it was, in general, easy to understand the source of the variance once the data were presented. These results suggest that the use of automatic item generation by college faculty is warranted, because most problems don’t exhibit significant difficulty variation, and the few that do can be detected through automatic means and addressed by the faculty member.

KW - Automatic item generation

KW - Item models

UR - http://www.scopus.com/inward/record.url?scp=85068322313&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068322313&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-23204-7_7

DO - 10.1007/978-3-030-23204-7_7

M3 - Conference contribution

AN - SCOPUS:85068322313

SN - 9783030232030

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 71

EP - 83

BT - Artificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings

A2 - Isotani, Seiji

A2 - Hastings, Peter

A2 - Ogan, Amy

A2 - McLaren, Bruce

A2 - Millán, Eva

A2 - Luckin, Rose

PB - Springer-Verlag

ER -