TY - GEN
T1 - Work in Progress
T2 - 2022 IEEE Frontiers in Education Conference, FIE 2022
AU - Benson, Lisa
AU - Lichtenstein, Gary
AU - Ko, Evan
AU - Bates, Rebecca
AU - Watts, Kelsey
AU - Jensen, Karin
N1 - Funding Information:
ACKNOWLEDGMENT We thank our study participants for their time in completing reviews, surveys, and interviews. This material is based on work supported by the National Science Foundation under Grant #s EEC-2037807, 2037788, 2037797. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Funding Information:
This research was funded by the National Science Foundation.
Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - As a field, engineering education research (EER) draws from multiple disciplines and geographic regions. This presents challenges in the process of peer reviewing EER manuscripts, as peer reviewers typically do not receive formal training in critiquing research publications. We investigated how EER scholars develop an understanding of how to conduct peer reviews of manuscripts related to engineering education. We draw from theories of cognition and schema development to address the following question: What aspects of EER manuscripts are reviewers paying attention to, and how do these aspects differ for reviewers from different geographic regions?The context for this study was a mentored peer reviewer program, which pairs less experienced reviewers (mentees) with experienced reviewers (mentors) to review a set of manuscripts. Participants (n=27: nine mentors and 18 mentees) from six countries and varied levels of expertise were organized into triads (one mentor and two mentees). Each triad collaboratively evaluated each manuscript, crafted feedback to authors and formulated recommendations to journal editors. Data collected from participants included Structured Peer Reviews (SPR) that prompted participants to comment on the main strengths and weaknesses of a short EER manuscript and formulate a recommendation to the editor. All participants completed SPRs for one manuscript prior to the program (Pre-SPR) and a second manuscript at the end of the program (Post-SPR). SPRs were coded and categorized into six themes: context, methods, results, discussion, mechanics, and EER relevance. Themes were categorized into codes related to positive or negative assessments.The preliminary analysis presented in this paper focuses on the extent to which reviewers converged in the breadth of what they commented on in the pre- and post-SPR forms. Results were compared based on where participants earned their doctorate. Differences between participants from different geographic regions emerged in the Pre-SPR data but were minimal in the post-SPR. This research provides a foundation for considering peer review as a form of professional development as findings suggest that training affects reviewers' understanding of EER scholarship.
AB - As a field, engineering education research (EER) draws from multiple disciplines and geographic regions. This presents challenges in the process of peer reviewing EER manuscripts, as peer reviewers typically do not receive formal training in critiquing research publications. We investigated how EER scholars develop an understanding of how to conduct peer reviews of manuscripts related to engineering education. We draw from theories of cognition and schema development to address the following question: What aspects of EER manuscripts are reviewers paying attention to, and how do these aspects differ for reviewers from different geographic regions?The context for this study was a mentored peer reviewer program, which pairs less experienced reviewers (mentees) with experienced reviewers (mentors) to review a set of manuscripts. Participants (n=27: nine mentors and 18 mentees) from six countries and varied levels of expertise were organized into triads (one mentor and two mentees). Each triad collaboratively evaluated each manuscript, crafted feedback to authors and formulated recommendations to journal editors. Data collected from participants included Structured Peer Reviews (SPR) that prompted participants to comment on the main strengths and weaknesses of a short EER manuscript and formulate a recommendation to the editor. All participants completed SPRs for one manuscript prior to the program (Pre-SPR) and a second manuscript at the end of the program (Post-SPR). SPRs were coded and categorized into six themes: context, methods, results, discussion, mechanics, and EER relevance. Themes were categorized into codes related to positive or negative assessments.The preliminary analysis presented in this paper focuses on the extent to which reviewers converged in the breadth of what they commented on in the pre- and post-SPR forms. Results were compared based on where participants earned their doctorate. Differences between participants from different geographic regions emerged in the Pre-SPR data but were minimal in the post-SPR. This research provides a foundation for considering peer review as a form of professional development as findings suggest that training affects reviewers' understanding of EER scholarship.
KW - academic publishing
KW - engineering education research
KW - mental models
KW - peer review
UR - http://www.scopus.com/inward/record.url?scp=85143820887&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85143820887&partnerID=8YFLogxK
U2 - 10.1109/FIE56618.2022.9962617
DO - 10.1109/FIE56618.2022.9962617
M3 - Conference contribution
AN - SCOPUS:85143820887
T3 - Proceedings - Frontiers in Education Conference, FIE
BT - 2022 IEEE Frontiers in Education Conference, FIE 2022
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 8 October 2022 through 11 October 2022
ER -