Fairness evaluation in deepfake detection models using metamorphic testing

Muxin Pu, Meng Yi Kuan, Nyee Thoang Lim, Chun Yong Chong, Mei Kuan Lim

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

6 Citations (Scopus)

Abstract

Fairness of deepfake detectors in the presence of anomalies are not well investigated, especially if those anomalies are more prominent in either male or female subjects. The primary motivation for this work is to evaluate how deepfake detection model behaves under such anomalies. However, due to the black-box nature of deep learning (DL) and artificial intelligence (AI) systems, it is hard to predict the performance of a model when the input data is modified. Crucially, if this defect is not addressed properly, it will adversely affect the fairness of the model and result in discrimination of certain sub-population unintentionally. Therefore, the objective of this work is to adopt metamorphic testing to examine the reliability of the selected deepfake detection model, and how the transformation of input variation places influence on the output. We have chosen MesoInception-4, a state-of-the-art deepfake detection model, as the target model and makeup as the anomalies. Makeups are applied through utilizing the Dlib library to obtain the 68 facial landmarks prior to filling in the RGB values. Metamorphic relations are derived based on the notion that realistic perturbations of the input images, such as makeup, involving eyeliners, eye shadows, blushes, and lipsticks (which are common cosmetic appearance) applied to male and female images, should not alter the output of the model by a huge margin. Furthermore, we narrow down the scope to focus on revealing potential gender biases in DL and AI systems. Specifically, we are interested to examine whether MesoInception-4 model produces unfair decisions, which should be considered as a consequence of robustness issues. The findings from our work have the potential to pave the way for new research directions in the quality assurance and fairness in DL and AI systems.

Original languageEnglish
Title of host publicationProceedings - 7th International Workshop on Metamorphic Testing, MET 2022
EditorsXiaoyuan Xie, Upulee Kanewala, Alastair Donaldson
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages7-14
Number of pages8
ISBN (Electronic)9781450393072
ISBN (Print)9781665462303
DOIs
Publication statusPublished - 2022
EventIEEE/ACM International Workshop on Metamorphic Testing 2022 - Pittsburgh, United States of America
Duration: 9 May 20229 May 2022
Conference number: 7th
https://ieeexplore.ieee.org/xpl/conhome/9826118/proceeding (Proceedings)

Conference

ConferenceIEEE/ACM International Workshop on Metamorphic Testing 2022
Abbreviated titleMET 2022
Country/TerritoryUnited States of America
CityPittsburgh
Period9/05/229/05/22
Internet address

Keywords

  • fairness testing
  • Metamorphic testing
  • oracle problem
  • robustness testing

Cite this