Towards the automated evaluation of legal casenote essays

A legal casenote essay is a commonly assigned writing task to first-year law students aiming to promote their understanding of legal reasoning and help them acquire writing skills in a legal domain. To ensure law students master the legal casenote writing, it is critical that instructors monitor and evaluate students’ progress, and provide a timely and specific feedback. This is, however, a challenging task to many instructors as they often need to dedicate excessive time and effort to evaluate writing of and provide formative feedback to each individual student. We posit a computational tool that can afford at-scale evaluation of legal casenote writing may help remedy this challenge. Although quite some automatic writing evaluation (AWE) tools have been applied in the domain of education, the AWE tool that can analyse rhetoric of a legal casenote essay (i.e., specific rhetorical elements required by this task) is yet to be developed. We made the first step towards developing such a tool. We manually annotated each sentence in a corpus of 1,020 authentic casenote essays written over 6 offerings of the first-year legal writing course and developed one traditional machine learning classifier (Random Forest) and two deep learning classifiers (based on vanilla BERT and Legal BERT pre-trained language models). We found that the deep learning classifier based on Legal BERT could correctly identify more than 86% of rhetorical moves in a casenote. Our findings may be of a particular interest for educational researchers and practitioners who seek to use the methods of artificial intelligence to support legal writing education.