Abstract
In the world of online interactions, social communities face a significant challenge: the spread of offensive content and hate speech through toxic languages. Such issues led to growing research on text detoxification systems that can automatically rewrite toxic content. A systematic evaluation is required to ensure these systems produce high-quality detoxified text that modifies the original text to be non-toxic while preserving its content. However, this often relies on large amounts of labelled data and human judgement, which may not always be feasible. This limitation is typically known as the oracle problem. Metamorphic testing (MT) has conventionally been used to solve the oracle problem by deriving metamorphic relations (MRs) to test a program's functionality. A new MT approach focused on data validation showed that MRs incorporated with tools can be used to identify defects in machine translation services. This paper draws inspiration from this new MT perspective by presenting four metamorphic relations incorporated with tools to evaluate style transfer accuracy, content preservation, fluency, and a joint of these three. Our proposed approach effectively identifies defective behaviour in state-of-the-art text detoxification systems.
Original language | English |
---|---|
Title of host publication | 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC 2024) |
Editors | Yi Wang, Liming Zhang, Chi Man Pun, Jinyu Tian |
Place of Publication | Piscataway NJ USA |
Publisher | IEEE, Institute of Electrical and Electronics Engineers |
Pages | 1763-1768 |
Number of pages | 6 |
ISBN (Electronic) | 9798350367331 |
ISBN (Print) | 9798350367348 |
DOIs | |
Publication status | Published - 2024 |
Event | Annual Summit and Conference of the Asia-Pacific-Signal-and-Information-Processing-Association (APSIPA) 2024 - Macau, China Duration: 3 Dec 2024 → 6 Dec 2024 https://ieeexplore.ieee.org/xpl/conhome/10848542/proceeding (Proceedings) https://www.apsipa2024.org/ (Website) http://www.apsipa2024.org/ (Website) |
Publication series
Name | Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC |
---|---|
Publisher | IEEE, Institute of Electrical and Electronics Engineers |
ISSN (Print) | 2640-009X |
ISSN (Electronic) | 2640-0103 |
Conference
Conference | Annual Summit and Conference of the Asia-Pacific-Signal-and-Information-Processing-Association (APSIPA) 2024 |
---|---|
Abbreviated title | APSIPA ASC 2024 |
Country/Territory | China |
City | Macau |
Period | 3/12/24 → 6/12/24 |
Internet address |
|