Automated question answering for improved understanding of compliance requirements: a multi-document study

Sallam Abualhaija, Chetan Arora, Amin Sleimi, Lionel C. Briand

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

7 Citations (Scopus)

Abstract

Software systems are increasingly subject to regulatory compliance. Extracting compliance requirements from regulations is challenging. Ideally, locating compliance-related information in a regulation requires a joint effort from requirements engineers and legal experts, whose availability is limited. However, regulations are typically long documents spanning hundreds of pages, containing legal jargon, applying complicated natural language structures, and including cross-references, thus making their analysis effort-intensive. In this paper, we propose an automated question-answering (QA) approach that assists requirements engineers in finding the legal text passages relevant to compliance requirements. Our approach utilizes large-scale language models fine-tuned for QA, including BERT and three variants. We evaluate our approach on 107 question-answer pairs, manually curated by subject-matter experts, for four different European regulatory documents. Among these documents is the general data protection regulation (GDPR) - a major source for privacy-related requirements. Our empirical results show that, in $\approx 94$% of the cases, our approach finds the text passage containing the answer to a given question among the top five passages that our approach marks as most relevant. Further, our approach successfully demarcates, in the selected passage, the right answer with an average accuracy of $\approx$91%.

Original languageEnglish
Title of host publicationProceedings - 30th IEEE International Requirements Engineering Conference, RE 2022
EditorsEric Knauss, Gunter Mussbacher, Chetan Arora, Muneera Bano, Jean-Guy Schneider
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages39-50
Number of pages12
ISBN (Electronic)9781665470001
ISBN (Print)9781665470018
DOIs
Publication statusPublished - 2022
Externally publishedYes
EventIEEE International Requirements Engineering Conference 2022 - Online, Australia
Duration: 15 Aug 202219 Aug 2022
Conference number: 30th
https://ieeexplore.ieee.org/xpl/conhome/9919881/proceeding (Proceedings)
https://conf.researchr.org/home/RE-2022 (Website)

Publication series

NameProceedings of the IEEE International Conference on Requirements Engineering
PublisherIEEE, Institute of Electrical and Electronics Engineers
Volume2022-August
ISSN (Print)1090-705X
ISSN (Electronic)2332-6441

Conference

ConferenceIEEE International Requirements Engineering Conference 2022
Abbreviated titleRE 2022
Country/TerritoryAustralia
Period15/08/2219/08/22
Internet address

Keywords

  • BERT
  • Language Models (LMs)
  • Natural Language Processing (NLP)
  • Question Answering
  • Regulatory Compliance
  • Requirements Engineering

Cite this