Silas Hsu
University of Illinois at Urbana-Champaign, Urbana, Illinois, United States
Deployment of AI assessment tools in education is widespread, but work on students' interactions and attitudes towards imperfect autograders is comparatively lacking. This paper presents students' perceptions surrounding a \url{~}90\% accurate automated short-answer grader that determined homework and exam credit in a college-level computer science course. Using surveys and interviews, we investigated students' knowledge about the autograder and their attitudes. We observed that misalignment between folk theories about how the autograder worked and how it actually worked could lead to suboptimal answer construction strategies. Students overestimated the autograder's probability of marking correct answers as wrong, and estimates of this probability were associated with dissatisfaction and perceptions of unfairness. Many participants expressed a need for additional instruction on how to cater to the autograder. From these findings, we propose guidelines for incorporating imperfect short answer autograders into classroom in a manner that is considerate of students' needs.
https://doi.org/10.1145/3411764.3445424
The ACM CHI Conference on Human Factors in Computing Systems (https://chi2021.acm.org/)