Overlooked, Underlying: Understanding tacit criteria of proposal reviewing during a mock panel review

Randi Sims, Kelsey Watts, Evan Ko, Rebecca A. Bates, Gary Lichtenstein, Karin Jensen, Lisa Benson

Research output: Contribution to journalConference articlepeer-review


This research paper study was situated within a peer review mentoring program in which novice reviewers were paired with mentors who are former National Science Foundation (NSF) program directors with experience running discipline-based education research (DBER) panels. Whether it be a manuscript or grant proposal, the outcome of peer review can greatly influence academic careers and the impact of research on a field. Yet the criteria upon which reviewers base their recommendations and the processes they follow as they review are poorly understood. Mentees reviewed three previously submitted proposals to the NSF and drafted pre-panel reviews regarding the proposals' intellectual merit and broader impacts, strengths, and weaknesses relative to solicitation-specific criteria. After participation in one mock review panel, mentees could then revise their pre-review evaluations based on the panel discussion. Using a lens of transformative learning theory, this study sought to answer the following research questions: 1) What are the tacit criteria used to inform recommendations for grant proposal reviews among scholars new to the review process? 2) To what extent are there changes in these tacit criteria and subsequent recommendations for grant proposal reviews after participation in a mock panel review? Using a single case study approach to explore one mock review panel, we conducted document analyses of six mentees' reviews completed before and after their participation in the mock review panel. Findings from this study suggest that reviewers primarily focus on the positive broader impacts proposed by a study and the level of detail within a submitted proposal. Although mentees made few changes to their reviews after the mock panel discussion, changes which were present illustrate that reviewers more deeply considered the broader impacts of the proposed studies. These results can inform review panel practices as well as approaches to training to support new reviewers in DBER fields.

Original languageEnglish (US)
JournalASEE Annual Conference and Exposition, Conference Proceedings
StatePublished - Jun 25 2023
Externally publishedYes
Event2023 ASEE Annual Conference and Exposition - The Harbor of Engineering: Education for 130 Years, ASEE 2023 - Baltimore, United States
Duration: Jun 25 2023Jun 28 2023


  • Case Study
  • Grant Proposal Review
  • Peer Review
  • Transformative Learning Theory

ASJC Scopus subject areas

  • General Engineering


Dive into the research topics of 'Overlooked, Underlying: Understanding tacit criteria of proposal reviewing during a mock panel review'. Together they form a unique fingerprint.

Cite this