TY - GEN
T1 - End-to-End Automation of Feedback on Student Assembly Programs
AU - Liu, Zikai
AU - Liu, Tingkai
AU - Li, Qi
AU - Luo, Wenqing
AU - Lumetta, Steven S.
N1 - Publisher Copyright:
© 2021 IEEE.
PY - 2021
Y1 - 2021
N2 - We developed a set of tools designed to provide rapid feedback to students as they learn to write programs in assembly language (LC-3, a RISC-like educational instruction set architecture). At the heart of the system is an extended version of KLEE, KLC3, that enables us to both identify issues and perform equivalence checking between student code and a gold (correct) version of each assignment. Feedback begins when students edit their code using a VSCode extension that leverages static analysis to perform a variety of correctness and style checks, encouraging students to improve their code quality. Each time a student commits code to their Git repository, our system triggers. Using KLC3 (KLEE), the student code is executed along with the gold version, and issues and behavioral differences are delivered back to the student through their Git repository as a human-readable report, test cases, and scripts. A queueing system allows students to monitor progress, but responses are generally available within minutes. We also extended the LC-3 simulation tools to support reverse debugging, making the process of finding complex bugs much more tractable for students, and used Emscripten to develop a browser-based interface for use in testing and debugging. Finally, our system maintains an individual regression test suite for each student and requires a submission to pass all previous tests before re-evaluation in KLC3, thus avoiding encouraging programming-by-guesswork. We deployed the system to provide feedback for the assembly programming assignments in a class of over 100 students in Fall 2020. Students wrote a median of around 700 lines of assembly for these assignments, making heavy use of our tools to understand and eliminate their bugs. Anonymous student feedback on the tools was uniformly positive. Since that semester, we have continued to refine and expand our tools' analysis capabilities and performance, and plan to deploy the system again in the near future (the class is offered every Fall).
AB - We developed a set of tools designed to provide rapid feedback to students as they learn to write programs in assembly language (LC-3, a RISC-like educational instruction set architecture). At the heart of the system is an extended version of KLEE, KLC3, that enables us to both identify issues and perform equivalence checking between student code and a gold (correct) version of each assignment. Feedback begins when students edit their code using a VSCode extension that leverages static analysis to perform a variety of correctness and style checks, encouraging students to improve their code quality. Each time a student commits code to their Git repository, our system triggers. Using KLC3 (KLEE), the student code is executed along with the gold version, and issues and behavioral differences are delivered back to the student through their Git repository as a human-readable report, test cases, and scripts. A queueing system allows students to monitor progress, but responses are generally available within minutes. We also extended the LC-3 simulation tools to support reverse debugging, making the process of finding complex bugs much more tractable for students, and used Emscripten to develop a browser-based interface for use in testing and debugging. Finally, our system maintains an individual regression test suite for each student and requires a submission to pass all previous tests before re-evaluation in KLC3, thus avoiding encouraging programming-by-guesswork. We deployed the system to provide feedback for the assembly programming assignments in a class of over 100 students in Fall 2020. Students wrote a median of around 700 lines of assembly for these assignments, making heavy use of our tools to understand and eliminate their bugs. Anonymous student feedback on the tools was uniformly positive. Since that semester, we have continued to refine and expand our tools' analysis capabilities and performance, and plan to deploy the system again in the near future (the class is offered every Fall).
KW - Assembly
KW - Education
KW - Programming feedback
KW - Symbolic analysis
UR - http://www.scopus.com/inward/record.url?scp=85125498918&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85125498918&partnerID=8YFLogxK
U2 - 10.1109/ASE51524.2021.9678837
DO - 10.1109/ASE51524.2021.9678837
M3 - Conference contribution
AN - SCOPUS:85125498918
T3 - Proceedings - 2021 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021
SP - 18
EP - 29
BT - Proceedings - 2021 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021
Y2 - 15 November 2021 through 19 November 2021
ER -