What's the Point? How Scores Undermine Written Comments on Open-Ended Work

Patrick Crain, Brian Bailey

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Scaling assessments typically relies on quantifying work quality, yet written comments are the assessment method of choice for open-ended work. Existing scalable solutions compromise by mapping quality scores to pre-authored comments, but how scores influence interpretation of these comments is not well understood. We report results from a study of how 441 participants authored and revised short stories in response to a score, written comments, both types of feedback, or no feedback. We analyzed data from the story-writing task and two surveys to determine task and feedback satisfaction, revision depth and effort, and improvement between drafts for each participant. We found task satisfaction and task performance were positively correlated among participants who were shown a score. Feedback satisfaction, revision effort, and improvement were highest among participants shown written comments. Either type of feedback prompted more deep revisions than no feedback, but together elicited fewer deep revisions than written comments alone. Our work informs the design of scalable open-ended assessment systems by contributing insights regarding how scores influence perceptions of written feedback and subsequent revision outcomes.

Original languageEnglish (US)
Title of host publicationL@S 2021 - Proceedings of the 8th ACM Conference on Learning @ Scale
PublisherAssociation for Computing Machinery, Inc
Pages127-138
Number of pages12
ISBN (Electronic)9781450382151
DOIs
StatePublished - Jun 8 2021
Event8th Annual ACM Conference on Learning at Scale, L@S 2021 - Virtual, Online, Germany
Duration: Jun 22 2021Jun 25 2021

Publication series

NameL@S 2021 - Proceedings of the 8th ACM Conference on Learning @ Scale

Conference

Conference8th Annual ACM Conference on Learning at Scale, L@S 2021
Country/TerritoryGermany
CityVirtual, Online
Period6/22/216/25/21

Keywords

  • automated assessment
  • creative work
  • open-ended work
  • scaling assessment

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'What's the Point? How Scores Undermine Written Comments on Open-Ended Work'. Together they form a unique fingerprint.

Cite this