Development of a Profile-Based Writing Scale: How Collaboration with Teachers Enhanced Assessment Practice in a Post-Admission ESL Writing Program at a USA University

This study reports on the revision of a rating scale for an ESL writing placement test and demonstrates how collaboration with teachers can both enhance the assessment practice within an ESL writing program and raise teachers’ awareness about assessment literacy. Following a data-driven approach to scale development, teachers participated in a three-stage revision process, where they were asked to (1) reflect on the range of writing performances in ESL courses, (2) evaluate sample essays from the test and revise descriptors for the new scale, and (3) pilot-rate new essays using the new scale. During the first stage, both teachers and testers recognized that test takers display different strengths and weaknesses in argument development and lexico-grammar. However, when evaluating sample essays, teachers seemed to weigh argument development more heavily, whereas testers placed a higher value on lexico-grammatical accuracy. Additionally, when rating argument development, some teachers relied heavily on surface/structural rhetorical features rather than essay content. These contrasts resulted in conflicting ratings on certain essay profiles. Through several rounds of discussion, these differences were eventually mitigated by creating separate criteria for argument development and lexico-grammar. The revised scale strikes a better balance between argument development and lexico-grammar, more accurately covering the range of writing performances among test takers. The revision process standardized the conceptualization and operationalization of writing quality, shifting teachers’ focus from surface rhetorical features to essay content. In return, collaboration with teachers enhanced testers’ understanding of the local instructional contexts. Teachers’ involvement promoted collaborative assessment-related dialogues and practices within the ESL program, strengthening the alignment across curriculum, instruction, and assessment.