Automatic item generation enables a diverse array of questions to be generated through the use of question templates and randomly-selected parameters. Such automatic item generators are most useful if the generated item instances are of either equivalent or predictable difficulty. In this study, we analyzed student performance on over 300 item generators from four university-level STEM classes collected over a period of two years. In most cases, we find that the choice of parameters fails to significantly affect the problem difficulty. In our analysis, we found it useful to distinguish parameters that were drawn from a small number (<10) of values from those that are drawn from a large—often continuous—range of values. We observed that values from smaller ranges were more likely to significantly impact difficulty, as sometimes they represented different configurations of the problem (e.g., upward force vs. downward force). Through manual review of the problems with significant difficulty variance, we found it was, in general, easy to understand the source of the variance once the data were presented. These results suggest that the use of automatic item generation by college faculty is warranted, because most problems don’t exhibit significant difficulty variation, and the few that do can be detected through automatic means and addressed by the faculty member.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education - 20th International Conference, AIED 2019, Proceedings
EditorsSeiji Isotani, Eva Millán, Amy Ogan, Bruce McLaren, Peter Hastings, Rose Luckin
Number of pages13
ISBN (Print)9783030232030
StatePublished - 2019
Event20th International Conference on Artificial Intelligence in Education, AIED 2019 - Chicago, United States
Duration: Jun 25 2019Jun 29 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11625 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference20th International Conference on Artificial Intelligence in Education, AIED 2019
Country/TerritoryUnited States


  • Automatic item generation
  • Item models

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'Effect of discrete and continuous parameter variation on difficulty in automatic item generation'. Together they form a unique fingerprint.

Cite this