Work in Progress: Reviewing engineering education scholarship in a mentored reviewer program: Trends based on geographic region

As a field, engineering education research (EER) draws from multiple disciplines and geographic regions. This presents challenges in the process of peer reviewing EER manuscripts, as peer reviewers typically do not receive formal training in critiquing research publications. We investigated how EER scholars develop an understanding of how to conduct peer reviews of manuscripts related to engineering education. We draw from theories of cognition and schema development to address the following question: What aspects of EER manuscripts are reviewers paying attention to, and how do these aspects differ for reviewers from different geographic regions?The context for this study was a mentored peer reviewer program, which pairs less experienced reviewers (mentees) with experienced reviewers (mentors) to review a set of manuscripts. Participants (n=27: nine mentors and 18 mentees) from six countries and varied levels of expertise were organized into triads (one mentor and two mentees). Each triad collaboratively evaluated each manuscript, crafted feedback to authors and formulated recommendations to journal editors. Data collected from participants included Structured Peer Reviews (SPR) that prompted participants to comment on the main strengths and weaknesses of a short EER manuscript and formulate a recommendation to the editor. All participants completed SPRs for one manuscript prior to the program (Pre-SPR) and a second manuscript at the end of the program (Post-SPR). SPRs were coded and categorized into six themes: context, methods, results, discussion, mechanics, and EER relevance. Themes were categorized into codes related to positive or negative assessments.The preliminary analysis presented in this paper focuses on the extent to which reviewers converged in the breadth of what they commented on in the pre- and post-SPR forms. Results were compared based on where participants earned their doctorate. Differences between participants from different geographic regions emerged in the Pre-SPR data but were minimal in the post-SPR. This research provides a foundation for considering peer review as a form of professional development as findings suggest that training affects reviewers' understanding of EER scholarship.