On robustness of prompt-based semantic parsing with large pre-trained language model: an empirical study on codex

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

5 Citations (Scopus)

Abstract

Semantic parsing is a technique aimed at constructing a structured representation of the meaning of a natural-language question. Recent advances in language models trained on code have shown superior performance in generating these representations compared to language models trained solely on natural language text. The existing fine-tuned neural semantic parsers are vulnerable to adversarial attacks on natural-language inputs. While it has been established that the robustness of smaller semantic parsers can be enhanced through adversarial training, this approach is not feasible for large language models in real-world scenarios, as it requires both substantial computational resources and expensive human annotation on in-domain semantic parsing data. This paper presents the first empirical study on the adversarial robustness of a prompt-based semantic parser based on CODEX, a state-ofthe-art (SOTA) language model trained on code. Our results demonstrate that the large language model of code is vulnerable to carefully crafted adversarial examples. To overcome this challenge, we propose methods for enhancing robustness without requiring substantial amounts of labelled data or intensive computational resources.

Original languageEnglish
Title of host publicationEACL 2023 - The 17th Conference of the European Chapter of the Association for Computational Linguistics - Proceedings of the Conference
EditorsAndreas Vlachos, Isabelle Augenstein
Place of PublicationStroudsburg PA USA
PublisherAssociation for Computational Linguistics (ACL)
Pages1090-1102
Number of pages13
ISBN (Electronic)9781959429449
Publication statusPublished - 2023
EventEuropean Association of Computational Linguistics Conference 2023 - Dubrovnik, Croatia
Duration: 2 May 20236 May 2023
Conference number: 17th
https://2023.eacl.org/ (Website)
https://aclanthology.org/volumes/2023.eacl-main/ (Proceedings)

Conference

ConferenceEuropean Association of Computational Linguistics Conference 2023
Abbreviated titleEACL 2023
Country/TerritoryCroatia
CityDubrovnik
Period2/05/236/05/23
Internet address

Cite this