Skin disease recognition using deep saliency features and multimodal learning of dermoscopy and clinical images

Zongyuan Ge, Sergey Demyanov, Rajib Chakravorty, Adrian Bowling, Rahil Garnavi

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

88 Citations (Scopus)


Skin cancer is the most common cancer world-wide, among which Melanoma the most fatal cancer, accounts for more than 10,000 deaths annually in Australia and United States. The 5-year survival rate for Melanoma can be increased over 90% if detected in its early stage. However, intrinsic visual similarity across various skin conditions makes the diagnosis challenging both for clinicians and automated classification methods. Many automated skin cancer diagnostic systems have been proposed in literature, all of which consider solely dermoscopy images in their analysis. In reality, however, clinicians consider two modalities of imaging; an initial screening using clinical photography images to capture a macro view of the mole, followed by dermoscopy imaging which visualizes morphological structures within the skin lesion. Evidences show that these two modalities provide complementary visual features that can empower the decision making process. In this work, we propose a novel deep convolutional neural network (DCNN) architecture along with a saliency feature descriptor to capture discriminative features of the two modalities for skin lesions classification. The proposed DCNN accepts a pair images of clinical and dermoscopic view of a single lesion and is capable of learning single-modality and cross-modality representations, simultaneously. Using one of the largest collected skin lesion datasets, we demonstrate that the proposed multi-modality method significantly outperforms single-modality methods on three tasks; differentiation between 15 various skin diseases, distinguishing cancerous (3 cancer types including melanoma) from non-cancerous moles, and detecting melanoma from benign cases.

Original languageEnglish
Title of host publicationMedical Image Computing and Computer Assisted Intervention − MICCAI 2017
Subtitle of host publication20th International Conference, Quebec City, QC, Canada, September 11–13, 2017 Proceedings, Part III
EditorsMaxime Descoteaux, Lena Maier-Hein, Alfred Franz, D. Louis Collins, Simon Duchesne
Place of PublicationCham Switzerland
Number of pages9
ISBN (Electronic)9783319661797
ISBN (Print)9783319661780
Publication statusPublished - 1 Jan 2017
Externally publishedYes
EventMedical Image Computing and Computer-Assisted Intervention 2017 - Quebec City Convention Centre, Quebec City, Canada
Duration: 11 Sept 201713 Sept 2017
Conference number: 20th (Proceedings)

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10435 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceMedical Image Computing and Computer-Assisted Intervention 2017
Abbreviated titleMICCAI 2017
CityQuebec City
Internet address

Cite this