Can AI support human grading? Examining machine attention and confidence in short answer scoring

Large language models built upon artificial intelligence (AI) hold great promises to innovate automatic short answer scoring (ASAS) - significantly alleviating educators’ workload in assessing student answers. However, ASAS systems on such basis have seen limited adoption in authentic teaching environments due to the models’ inability to explain the predictions they generate. To address this, we recruited 32 human graders to comparatively analyse the decision-making processes of human graders and AI-driven graders. Specifically, we exploited two types of data to holistically unveil the decision-making processes of human graders throughout grading, namely manual annotation of important words and gaze data of the human graders. The decision-making processes of AI-driven graders were revealed by important words extracted though eXplainable Artificial Intelligence technique and grading confidence reflected by the prediction probability distributions. We measured the alignment in their decision-making regarding their (i) estimated scoring difficulty, (ii) important text segments and (iii) crucial grammatical categories to enhance the transparency and trustworthiness of AI-driven graders. Subsequently, we conducted randomised control studies, presenting machine-extracted insights like important words and estimated scoring difficulty to scrutinise how they affected human grading. Our findings contribute new knowledge regarding the consistency between human and machine scoring and validates machine-extracted insights, such as important words and scoring difficulty, to be valuable in facilitating human grading, encouraging the adoption of ASAS systems and urging the potential collaboration between machine and human grading in pedagogical practices. However, we emphasised the significance of grasping question context and intricacy before leveraging such machine-extracted insights.