Can ChatGPT perform reasoning using the IRAC method in analyzing legal scenarios like a lawyer?

Xiaoxi Kang; Lizhen Qu; Lay-Ki Soon; Adnan Trakic; Terry Yue Zhuo; Patrick Charles Emerton; Genevieve Grant

doi:10.18653/v1/2023.findings-emnlp.929

Can ChatGPT perform reasoning using the IRAC method in analyzing legal scenarios like a lawyer?

Xiaoxi Kang, Lizhen Qu, Lay-Ki Soon, Adnan Trakic, Terry Yue Zhuo, Patrick Charles Emerton, Genevieve Grant

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

6 Citations (Scopus)

Abstract

Large Language Models (LLMs), such as CHATGPT, have drawn a lot of attentions recently in the legal domain due to its emergent ability to tackle a variety of legal tasks. However, it is still unknown if LLMs are able to analyze a legal case and perform reasoning in the same manner as lawyers. Therefore, we constructed a novel corpus consisting of scenarios pertain to Contract Acts Malaysia and Australian Social Act for Dependent Child. CHATGPT is applied to perform analysis on the corpus using the IRAC method, which is a framework widely used by legal professionals for organizing legal analysis. Each scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations. In addition, we conducted the first empirical assessment of CHATGPT for IRAC analysis in order to understand how well it aligns with the analysis of legal professionals. Our experimental results shed lights on possible future research directions to improve alignments between LLMs and legal experts in terms of legal reasoning.

Original language	English
Title of host publication	The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023
Editors	Nadi Tomeh, Atsushi Fujita, Aixin Sun, Bin Wang, Rong Tong, Ryan Cotterell
Place of Publication	Stroudsburg PA USA
Publisher	Association for Computational Linguistics (ACL)
Pages	13900-13923
Number of pages	24
ISBN (Electronic)	9798891760615
DOIs	https://doi.org/10.18653/v1/2023.findings-emnlp.929
Publication status	Published - 2023
Event	Empirical Methods in Natural Language Processing 2023 - , Singapore Duration: 6 Dec 2023 → 10 Dec 2023 https://2023.emnlp.org/ https://aclanthology.org/volumes/2023.findings-emnlp/ (Proceedings) https://aclanthology.org/volumes/2023.emnlp-demo/ (Proceedings)

Conference

Conference	Empirical Methods in Natural Language Processing 2023
Abbreviated title	EMNLP 2023
Country/Territory	Singapore
Period	6/12/23 → 10/12/23
Internet address	https://2023.emnlp.org/ https://aclanthology.org/volumes/2023.findings-emnlp/ (Proceedings) https://aclanthology.org/volumes/2023.emnlp-demo/ (Proceedings)

Access to Document

10.18653/v1/2023.findings-emnlp.929

573342349-oaFinal published version, 795 KB

Cite this

Kang, X., Qu, L., Soon, L.-K., Trakic, A., Zhuo, T. Y., Emerton, P. C., & Grant, G. (2023). Can ChatGPT perform reasoning using the IRAC method in analyzing legal scenarios like a lawyer? In N. Tomeh, A. Fujita, A. Sun, B. Wang, R. Tong, & R. Cotterell (Eds.), The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 13900-13923). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.findings-emnlp.929

Kang, Xiaoxi ; Qu, Lizhen ; Soon, Lay-Ki et al. / Can ChatGPT perform reasoning using the IRAC method in analyzing legal scenarios like a lawyer?. The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023. editor / Nadi Tomeh ; Atsushi Fujita ; Aixin Sun ; Bin Wang ; Rong Tong ; Ryan Cotterell. Stroudsburg PA USA : Association for Computational Linguistics (ACL), 2023. pp. 13900-13923

@inproceedings{cd4fbf60671342dc8c96bbab561d3fb6,

title = "Can ChatGPT perform reasoning using the IRAC method in analyzing legal scenarios like a lawyer?",

abstract = "Large Language Models (LLMs), such as CHATGPT, have drawn a lot of attentions recently in the legal domain due to its emergent ability to tackle a variety of legal tasks. However, it is still unknown if LLMs are able to analyze a legal case and perform reasoning in the same manner as lawyers. Therefore, we constructed a novel corpus consisting of scenarios pertain to Contract Acts Malaysia and Australian Social Act for Dependent Child. CHATGPT is applied to perform analysis on the corpus using the IRAC method, which is a framework widely used by legal professionals for organizing legal analysis. Each scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations. In addition, we conducted the first empirical assessment of CHATGPT for IRAC analysis in order to understand how well it aligns with the analysis of legal professionals. Our experimental results shed lights on possible future research directions to improve alignments between LLMs and legal experts in terms of legal reasoning.",

author = "Xiaoxi Kang and Lizhen Qu and Lay-Ki Soon and Adnan Trakic and Zhuo, {Terry Yue} and Emerton, {Patrick Charles} and Genevieve Grant",

note = "Funding Information: This research is made possible through the generous support of Monash Inter-Faculty Seeding Grant. We wish to extend our sincere gratitude to all the dedicated annotators who have made invaluable contributions to this project. We would like to extend our special thanks to Dr. Sia Chin Chin from Taylor{\textquoteright}s University, whose provision of the examination questions as a reliable source for the legal scenarios, along with assistance in recruiting qualified annotators, has been instrumental in advancing our research endeavors. Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; Empirical Methods in Natural Language Processing 2023, EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

doi = "10.18653/v1/2023.findings-emnlp.929",

language = "English",

pages = "13900--13923",

editor = "Nadi Tomeh and Atsushi Fujita and Aixin Sun and Bin Wang and Rong Tong and Ryan Cotterell",

booktitle = "The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023",

publisher = "Association for Computational Linguistics (ACL)",

url = "https://2023.emnlp.org/, https://aclanthology.org/volumes/2023.findings-emnlp/, https://aclanthology.org/volumes/2023.emnlp-demo/",

}

Kang, X, Qu, L , Soon, L-K , Trakic, A, Zhuo, TY, Emerton, PC & Grant, G 2023, Can ChatGPT perform reasoning using the IRAC method in analyzing legal scenarios like a lawyer? in N Tomeh, A Fujita, A Sun, B Wang, R Tong & R Cotterell (eds), The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL), Stroudsburg PA USA, pp. 13900-13923, Empirical Methods in Natural Language Processing 2023, Singapore, 6/12/23. https://doi.org/10.18653/v1/2023.findings-emnlp.929

Can ChatGPT perform reasoning using the IRAC method in analyzing legal scenarios like a lawyer? / Kang, Xiaoxi; Qu, Lizhen ; Soon, Lay-Ki et al.
The 2023 Conference on Empirical Methods in Natural Language Processing - Findings of the Association for Computational Linguistics: EMNLP 2023. ed. / Nadi Tomeh; Atsushi Fujita; Aixin Sun; Bin Wang; Rong Tong; Ryan Cotterell. Stroudsburg PA USA: Association for Computational Linguistics (ACL), 2023. p. 13900-13923.

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review