Class specific feature disentanglement and text embeddings for multi-label generalized zero shot CXR classification

Dwarikanath Mahapatra; Antonio Jose Jimeno Yepes; Shiba Kuanar; Sudipta Roy; Behzad Bozorgtabar; Mauricio Reyes; Zongyuan Ge

doi:10.1007/978-3-031-43895-0_26

Class specific feature disentanglement and text embeddings for multi-label generalized zero shot CXR classification

Dwarikanath Mahapatra, Antonio Jose Jimeno Yepes, Shiba Kuanar, Sudipta Roy, Behzad Bozorgtabar, Mauricio Reyes, Zongyuan Ge

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

Abstract

Robustness of medical image classification models is limited by its exposure to the candidate disease classes. Generalized zero shot learning (GZSL) aims at correctly predicting seen and unseen classes and most current GZSL approaches have focused on the single label case. It is common for chest x-rays to be labelled with multiple disease classes. We propose a novel multi-label GZSL approach using: 1) class specific feature disentanglement and 2) semantic relationship between disease labels distilled from BERT models pre-trained on biomedical literature. We learn a dictionary from distilled text embeddings, and leverage them to synthesize feature vectors that are representative of multi-label samples. Compared to existing methods, our approach does not require class attribute vectors, which are an essential part of GZSL methods for natural images but are not available for medical images. Our approach outperforms state of the art GZSL methods for chest xray images.

Original language	English
Title of host publication	Medical Image Computing and Computer Assisted Intervention – MICCAI 2023
Subtitle of host publication	26th International Conference Vancouver, BC, Canada, October 8–12, 2023 Proceedings, Part II
Editors	Hayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Syeda-Mahmood, Russell Taylor
Place of Publication	Cham Switzerland
Publisher	Springer
Pages	276-286
Number of pages	11
ISBN (Electronic)	9783031438950
ISBN (Print)	9783031438943
DOIs	https://doi.org/10.1007/978-3-031-43895-0_26
Publication status	Published - 2023
Event	Medical Image Computing and Computer-Assisted Intervention 2023 - Vancouver, Canada Duration: 8 Oct 2023 → 12 Oct 2023 Conference number: 26th https://link.springer.com/book/10.1007/978-3-031-43901-8 (Proceedings) https://conferences.miccai.org/2023/en/ (Website)

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	14221
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Medical Image Computing and Computer-Assisted Intervention 2023
Abbreviated title	MICCAI 2023
Country/Territory	Canada
City	Vancouver
Period	8/10/23 → 12/10/23
Internet address	https://link.springer.com/book/10.1007/978-3-031-43901-8 (Proceedings) https://conferences.miccai.org/2023/en/ (Website)

Keywords

chest x-rays
feature synthesis
GZSL
Multi-label
Text Embeddings

Access to Document

10.1007/978-3-031-43895-0_26

Cite this

Mahapatra, D., Jimeno Yepes, A. J., Kuanar, S., Roy, S., Bozorgtabar, B., Reyes, M., & Ge, Z. (2023). Class specific feature disentanglement and text embeddings for multi-label generalized zero shot CXR classification. In H. Greenspan, A. Madabhushi, P. Mousavi, S. Salcudean, J. Duncan, T. Syeda-Mahmood, & R. Taylor (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference Vancouver, BC, Canada, October 8–12, 2023 Proceedings, Part II (pp. 276-286). (Lecture Notes in Computer Science; Vol. 14221). Springer. https://doi.org/10.1007/978-3-031-43895-0_26

Mahapatra, Dwarikanath ; Jimeno Yepes, Antonio Jose ; Kuanar, Shiba et al. / Class specific feature disentanglement and text embeddings for multi-label generalized zero shot CXR classification. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference Vancouver, BC, Canada, October 8–12, 2023 Proceedings, Part II. editor / Hayit Greenspan ; Anant Madabhushi ; Parvin Mousavi ; Septimiu Salcudean ; James Duncan ; Tanveer Syeda-Mahmood ; Russell Taylor. Cham Switzerland : Springer, 2023. pp. 276-286 (Lecture Notes in Computer Science).

@inproceedings{dbb498646990454589f0f18e576950ac,

title = "Class specific feature disentanglement and text embeddings for multi-label generalized zero shot CXR classification",

abstract = "Robustness of medical image classification models is limited by its exposure to the candidate disease classes. Generalized zero shot learning (GZSL) aims at correctly predicting seen and unseen classes and most current GZSL approaches have focused on the single label case. It is common for chest x-rays to be labelled with multiple disease classes. We propose a novel multi-label GZSL approach using: 1) class specific feature disentanglement and 2) semantic relationship between disease labels distilled from BERT models pre-trained on biomedical literature. We learn a dictionary from distilled text embeddings, and leverage them to synthesize feature vectors that are representative of multi-label samples. Compared to existing methods, our approach does not require class attribute vectors, which are an essential part of GZSL methods for natural images but are not available for medical images. Our approach outperforms state of the art GZSL methods for chest xray images.",

keywords = "chest x-rays, feature synthesis, GZSL, Multi-label, Text Embeddings",

author = "Dwarikanath Mahapatra and {Jimeno Yepes}, {Antonio Jose} and Shiba Kuanar and Sudipta Roy and Behzad Bozorgtabar and Mauricio Reyes and Zongyuan Ge",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.; Medical Image Computing and Computer-Assisted Intervention 2023, MICCAI 2023 ; Conference date: 08-10-2023 Through 12-10-2023",

year = "2023",

doi = "10.1007/978-3-031-43895-0_26",

language = "English",

isbn = "9783031438943",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "276--286",

editor = "Hayit Greenspan and Anant Madabhushi and Parvin Mousavi and Septimiu Salcudean and James Duncan and Tanveer Syeda-Mahmood and Russell Taylor",

booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2023",

url = "https://link.springer.com/book/10.1007/978-3-031-43901-8, https://conferences.miccai.org/2023/en/",

}

Mahapatra, D, Jimeno Yepes, AJ, Kuanar, S, Roy, S, Bozorgtabar, B, Reyes, M & Ge, Z 2023, Class specific feature disentanglement and text embeddings for multi-label generalized zero shot CXR classification. in H Greenspan, A Madabhushi, P Mousavi, S Salcudean, J Duncan, T Syeda-Mahmood & R Taylor (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference Vancouver, BC, Canada, October 8–12, 2023 Proceedings, Part II. Lecture Notes in Computer Science, vol. 14221, Springer, Cham Switzerland, pp. 276-286, Medical Image Computing and Computer-Assisted Intervention 2023, Vancouver, British Columbia, Canada, 8/10/23. https://doi.org/10.1007/978-3-031-43895-0_26

Class specific feature disentanglement and text embeddings for multi-label generalized zero shot CXR classification. / Mahapatra, Dwarikanath; Jimeno Yepes, Antonio Jose; Kuanar, Shiba et al.
Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference Vancouver, BC, Canada, October 8–12, 2023 Proceedings, Part II. ed. / Hayit Greenspan; Anant Madabhushi; Parvin Mousavi; Septimiu Salcudean; James Duncan; Tanveer Syeda-Mahmood; Russell Taylor. Cham Switzerland: Springer, 2023. p. 276-286 (Lecture Notes in Computer Science; Vol. 14221).

Research output: Chapter in Book/Report/Conference proceeding › Conference Paper › Research › peer-review

TY - GEN

T1 - Class specific feature disentanglement and text embeddings for multi-label generalized zero shot CXR classification

AU - Mahapatra, Dwarikanath

AU - Jimeno Yepes, Antonio Jose

AU - Kuanar, Shiba

AU - Roy, Sudipta

AU - Bozorgtabar, Behzad

AU - Reyes, Mauricio

AU - Ge, Zongyuan

N1 - Conference code: 26th

PY - 2023

Y1 - 2023

N2 - Robustness of medical image classification models is limited by its exposure to the candidate disease classes. Generalized zero shot learning (GZSL) aims at correctly predicting seen and unseen classes and most current GZSL approaches have focused on the single label case. It is common for chest x-rays to be labelled with multiple disease classes. We propose a novel multi-label GZSL approach using: 1) class specific feature disentanglement and 2) semantic relationship between disease labels distilled from BERT models pre-trained on biomedical literature. We learn a dictionary from distilled text embeddings, and leverage them to synthesize feature vectors that are representative of multi-label samples. Compared to existing methods, our approach does not require class attribute vectors, which are an essential part of GZSL methods for natural images but are not available for medical images. Our approach outperforms state of the art GZSL methods for chest xray images.

AB - Robustness of medical image classification models is limited by its exposure to the candidate disease classes. Generalized zero shot learning (GZSL) aims at correctly predicting seen and unseen classes and most current GZSL approaches have focused on the single label case. It is common for chest x-rays to be labelled with multiple disease classes. We propose a novel multi-label GZSL approach using: 1) class specific feature disentanglement and 2) semantic relationship between disease labels distilled from BERT models pre-trained on biomedical literature. We learn a dictionary from distilled text embeddings, and leverage them to synthesize feature vectors that are representative of multi-label samples. Compared to existing methods, our approach does not require class attribute vectors, which are an essential part of GZSL methods for natural images but are not available for medical images. Our approach outperforms state of the art GZSL methods for chest xray images.

KW - chest x-rays

KW - feature synthesis

KW - GZSL

KW - Multi-label

KW - Text Embeddings

UR - http://www.scopus.com/inward/record.url?scp=85174697784&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-43895-0_26

DO - 10.1007/978-3-031-43895-0_26

M3 - Conference Paper

AN - SCOPUS:85174697784

SN - 9783031438943

T3 - Lecture Notes in Computer Science

SP - 276

EP - 286

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2023

A2 - Greenspan, Hayit

A2 - Madabhushi, Anant

A2 - Mousavi, Parvin

A2 - Salcudean, Septimiu

A2 - Duncan, James

A2 - Syeda-Mahmood, Tanveer

A2 - Taylor, Russell

PB - Springer

CY - Cham Switzerland

T2 - Medical Image Computing and Computer-Assisted Intervention 2023

Y2 - 8 October 2023 through 12 October 2023

ER -

Mahapatra D, Jimeno Yepes AJ, Kuanar S, Roy S, Bozorgtabar B, Reyes M et al. Class specific feature disentanglement and text embeddings for multi-label generalized zero shot CXR classification. In Greenspan H, Madabhushi A, Mousavi P, Salcudean S, Duncan J, Syeda-Mahmood T, Taylor R, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference Vancouver, BC, Canada, October 8–12, 2023 Proceedings, Part II. Cham Switzerland: Springer. 2023. p. 276-286. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-43895-0_26