MineObserver 2.0: A Deep Learning & In-Game Framework for Assessing Natural Language Descriptions of Minecraft Imagery

Jay Mahajan; Samuel Hum; Jack Henhapl; Diya Yunus; Matthew Gadbury; Emi Brown; Jeff Ginger; H. Chad Lane

doi:10.1609/aaai.v38i21.30367

MineObserver 2.0: A Deep Learning & In-Game Framework for Assessing Natural Language Descriptions of Minecraft Imagery

Jay Mahajan, Samuel Hum, Jack Henhapl, Diya Yunus, Matthew Gadbury, Emi Brown, Jeff Ginger, H. Chad Lane

Research output: Contribution to journal › Conference article › peer-review

Abstract

MineObserver 2.0 is an AI framework that uses Computer Vision and Natural Language Processing for assessing the accuracy of learner-generated descriptions of Minecraft images that include some scientifically relevant content. The system automatically assesses the accuracy of participant observations, written in natural language, made during science learning activities that take place in Minecraft. We demonstrate our system working in real-time and describe a teacher support dashboard to showcase observations, both of which advance our previous work. We present the results of a study showing that MineObserver 2.0 improves over its predecessor both in perceived accuracy of the system's generated descriptions as well as in usefulness of the system's feedback. In future work we intend improve system-generated descriptions, give teachers more control and upgrade the system to perform continuous learning to more effectively and rapidly respond to novel observations made by learners.

Original language	English (US)
Pages (from-to)	23207-23214
Number of pages	8
Journal	Proceedings of the AAAI Conference on Artificial Intelligence
Volume	38
Issue number	21
Early online date	Mar 24 2024
DOIs	https://doi.org/10.1609/aaai.v38i21.30367
State	Published - Mar 25 2024
Event	38th AAAI Conference on Artificial Intelligence, AAAI 2024 - Vancouver, Canada Duration: Feb 20 2024 → Feb 27 2024

Keywords

Image Captioning
Minecraft
Natural Language Processing

ASJC Scopus subject areas

Artificial Intelligence

Online availability

10.1609/aaai.v38i21.30367

Library availability

Discover UIUC Full Text

Cite this

@article{13bd5a004be24d528ed6b43de84e9530,

title = "MineObserver 2.0: A Deep Learning & In-Game Framework for Assessing Natural Language Descriptions of Minecraft Imagery",

abstract = "MineObserver 2.0 is an AI framework that uses Computer Vision and Natural Language Processing for assessing the accuracy of learner-generated descriptions of Minecraft images that include some scientifically relevant content. The system automatically assesses the accuracy of participant observations, written in natural language, made during science learning activities that take place in Minecraft. We demonstrate our system working in real-time and describe a teacher support dashboard to showcase observations, both of which advance our previous work. We present the results of a study showing that MineObserver 2.0 improves over its predecessor both in perceived accuracy of the system's generated descriptions as well as in usefulness of the system's feedback. In future work we intend improve system-generated descriptions, give teachers more control and upgrade the system to perform continuous learning to more effectively and rapidly respond to novel observations made by learners.",

keywords = "Image Captioning, Minecraft, Natural Language Processing",

author = "Jay Mahajan and Samuel Hum and Jack Henhapl and Diya Yunus and Matthew Gadbury and Emi Brown and Jeff Ginger and Lane, {H. Chad}",

note = "The authors would like to thank the participants and instructors who provided support for the data collection and fellow researchers on the project for help with the coding and statistics. This material is based upon work supported by the National Science Foundation under Grants 1713609 and 1906873.; 38th AAAI Conference on Artificial Intelligence, AAAI 2024 ; Conference date: 20-02-2024 Through 27-02-2024",

year = "2024",

month = mar,

day = "25",

doi = "10.1609/aaai.v38i21.30367",

language = "English (US)",

volume = "38",

pages = "23207--23214",

journal = "Proceedings of the AAAI Conference on Artificial Intelligence",

issn = "2159-5399",

publisher = "Association for the Advancement of Artificial Intelligence",

number = "21",

}

TY - JOUR

T1 - MineObserver 2.0

T2 - 38th AAAI Conference on Artificial Intelligence, AAAI 2024

AU - Mahajan, Jay

AU - Hum, Samuel

AU - Henhapl, Jack

AU - Yunus, Diya

AU - Gadbury, Matthew

AU - Brown, Emi

AU - Ginger, Jeff

AU - Lane, H. Chad

N1 - The authors would like to thank the participants and instructors who provided support for the data collection and fellow researchers on the project for help with the coding and statistics. This material is based upon work supported by the National Science Foundation under Grants 1713609 and 1906873.

PY - 2024/3/25

Y1 - 2024/3/25

N2 - MineObserver 2.0 is an AI framework that uses Computer Vision and Natural Language Processing for assessing the accuracy of learner-generated descriptions of Minecraft images that include some scientifically relevant content. The system automatically assesses the accuracy of participant observations, written in natural language, made during science learning activities that take place in Minecraft. We demonstrate our system working in real-time and describe a teacher support dashboard to showcase observations, both of which advance our previous work. We present the results of a study showing that MineObserver 2.0 improves over its predecessor both in perceived accuracy of the system's generated descriptions as well as in usefulness of the system's feedback. In future work we intend improve system-generated descriptions, give teachers more control and upgrade the system to perform continuous learning to more effectively and rapidly respond to novel observations made by learners.

AB - MineObserver 2.0 is an AI framework that uses Computer Vision and Natural Language Processing for assessing the accuracy of learner-generated descriptions of Minecraft images that include some scientifically relevant content. The system automatically assesses the accuracy of participant observations, written in natural language, made during science learning activities that take place in Minecraft. We demonstrate our system working in real-time and describe a teacher support dashboard to showcase observations, both of which advance our previous work. We present the results of a study showing that MineObserver 2.0 improves over its predecessor both in perceived accuracy of the system's generated descriptions as well as in usefulness of the system's feedback. In future work we intend improve system-generated descriptions, give teachers more control and upgrade the system to perform continuous learning to more effectively and rapidly respond to novel observations made by learners.

KW - Image Captioning

KW - Minecraft

KW - Natural Language Processing

UR - http://www.scopus.com/inward/record.url?scp=85189648573&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85189648573&partnerID=8YFLogxK

U2 - 10.1609/aaai.v38i21.30367

DO - 10.1609/aaai.v38i21.30367

M3 - Conference article

AN - SCOPUS:85189648573

SN - 2159-5399

VL - 38

SP - 23207

EP - 23214

JO - Proceedings of the AAAI Conference on Artificial Intelligence

JF - Proceedings of the AAAI Conference on Artificial Intelligence

IS - 21

Y2 - 20 February 2024 through 27 February 2024

ER -

MineObserver 2.0: A Deep Learning & In-Game Framework for Assessing Natural Language Descriptions of Minecraft Imagery

Abstract

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this