EPVT: Environment-aware prompt vision transformer for domain generalization in skin lesion recognition

Siyuan Yan, Chi Liu, Zhen Yu, Lie Ju, Dwarikanath Mahapatra, Victoria Mar, Monika Janda, Peter Soyer, Zongyuan Ge

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review


Skin lesion recognition using deep learning has made remarkable progress, and there is an increasing need for deploying these systems in real-world scenarios. However, recent research has revealed that deep neural networks for skin lesion recognition may overly depend on disease-irrelevant image artifacts (i.e. dark corners, dense hairs), leading to poor generalization in unseen environments. To address this issue, we propose a novel domain generalization method called EPVT, which involves embedding prompts into the vision transformer to collaboratively learn knowledge from diverse domains. Concretely, EPVT leverages a set of domain prompts, each of which plays as a domain expert, to capture domain-specific knowledge; and a shared prompt for general knowledge over the entire dataset. To facilitate knowledge sharing and the interaction of different prompts, we introduce a domain prompt generator that enables low-rank multiplicative updates between domain prompts and the shared prompt. A domain mixup strategy is additionally devised to reduce the co-occurring artifacts in each domain, which allows for more flexible decision margins and mitigates the issue of incorrectly assigned domain labels. Experiments on four out-of-distribution datasets and six different biased ISIC datasets demonstrate the superior generalization ability of EPVT in skin lesion recognition across various environments. Code is available at https://github.com/SiyuanYan1/EPVT.

Original languageEnglish
Title of host publicationMedical Image Computing and Computer Assisted Intervention – MICCAI 2023
Subtitle of host publication26th International Conference Vancouver, BC, Canada, October 8–12, 2023 Proceedings, Part VII
EditorsHayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Syeda-Mahmood, Russell Taylor
Place of PublicationCham Switzerland
Number of pages11
ISBN (Electronic)9783031439902
ISBN (Print)9783031439896
Publication statusPublished - 2023
EventMedical Image Computing and Computer-Assisted Intervention 2023 - Vancouver, Canada
Duration: 8 Oct 202312 Oct 2023
Conference number: 26th
https://link.springer.com/book/10.1007/978-3-031-43901-8 (Proceedings)
https://conferences.miccai.org/2023/en/ (Website)

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceMedical Image Computing and Computer-Assisted Intervention 2023
Abbreviated titleMICCAI 2023
Internet address


  • Debiasing
  • Domain generalization
  • Prompt
  • Skin lesions

Cite this