Opening the black box of neural networks: methods for interpreting neural network models in clinical applications

Zhongheng Zhang, Marcus W Beck, David A Winkler, Bin Huang, Wilbert Sibanda, Hemant Goyal, AME Big-Data Clinical Trial Collaborative Group

Research output: Contribution to journalReview ArticleResearchpeer-review

Abstract

Artificial neural networks (ANNs) are powerful tools for data analysis and are particularly suitable for modeling relationships between variables for best prediction of an outcome. While these models can be used to answer many important research questions, their utility has been critically limited because the interpretation of the “black box” model is difficult. Clinical investigators usually employ ANN models to predict the clinical outcomes or to make a diagnosis; the model however is difficult to interpret for clinicians. To address this important shortcoming of neural network modeling methods, we describe several methods to help subject-matter audiences (e.g., clinicians, medical policy makers) understand neural network models. Garson’s algorithm describes the relative magnitude of the importance of a descriptor (predictor) in its connection with outcome variables by dissecting the model weights. The Lek’s profile method explores the relationship of the outcome variable and a predictor of interest, while holding other predictors at constant values (e.g., minimum, 20th quartile, maximum). While Lek’s profile was developed specifically for neural networks, partial dependence plot is a more generic version that visualize the relationship between an outcome and one or two predictors. Finally, the local interpretable model-agnostic explanations (LIME) method can show the predictions of any classification or regression, by approximating it locally with an interpretable model. R code for the implementations of these methods is shown by using example data fitted with a standard, feed-forward neural network model. We offer codes and step-by-step description on how to use these tools to facilitate better understanding of ANN.
Original languageEnglish
Article number216
Pages (from-to)1-11
Number of pages11
JournalAnnals of Translational Medicine
Volume6
Issue number11
DOIs
Publication statusPublished - 2018

Cite this

Zhang, Z., Beck, M. W., Winkler, D. A., Huang, B., Sibanda, W., Goyal, H., & AME Big-Data Clinical Trial Collaborative Group (2018). Opening the black box of neural networks: methods for interpreting neural network models in clinical applications. Annals of Translational Medicine, 6(11), 1-11. [216]. https://doi.org/10.21037/atm.2018.05.32
Zhang, Zhongheng ; Beck, Marcus W ; Winkler, David A ; Huang, Bin ; Sibanda, Wilbert ; Goyal, Hemant ; AME Big-Data Clinical Trial Collaborative Group. / Opening the black box of neural networks : methods for interpreting neural network models in clinical applications. In: Annals of Translational Medicine. 2018 ; Vol. 6, No. 11. pp. 1-11.
@article{d9d4d3f1f86d4d4793aace9498281114,
title = "Opening the black box of neural networks: methods for interpreting neural network models in clinical applications",
abstract = "Artificial neural networks (ANNs) are powerful tools for data analysis and are particularly suitable for modeling relationships between variables for best prediction of an outcome. While these models can be used to answer many important research questions, their utility has been critically limited because the interpretation of the “black box” model is difficult. Clinical investigators usually employ ANN models to predict the clinical outcomes or to make a diagnosis; the model however is difficult to interpret for clinicians. To address this important shortcoming of neural network modeling methods, we describe several methods to help subject-matter audiences (e.g., clinicians, medical policy makers) understand neural network models. Garson’s algorithm describes the relative magnitude of the importance of a descriptor (predictor) in its connection with outcome variables by dissecting the model weights. The Lek’s profile method explores the relationship of the outcome variable and a predictor of interest, while holding other predictors at constant values (e.g., minimum, 20th quartile, maximum). While Lek’s profile was developed specifically for neural networks, partial dependence plot is a more generic version that visualize the relationship between an outcome and one or two predictors. Finally, the local interpretable model-agnostic explanations (LIME) method can show the predictions of any classification or regression, by approximating it locally with an interpretable model. R code for the implementations of these methods is shown by using example data fitted with a standard, feed-forward neural network model. We offer codes and step-by-step description on how to use these tools to facilitate better understanding of ANN.",
author = "Zhongheng Zhang and Beck, {Marcus W} and Winkler, {David A} and Bin Huang and Wilbert Sibanda and Hemant Goyal and {AME Big-Data Clinical Trial Collaborative Group}",
year = "2018",
doi = "10.21037/atm.2018.05.32",
language = "English",
volume = "6",
pages = "1--11",
journal = "Annals of Translational Medicine",
issn = "2305-5847",
number = "11",

}

Zhang, Z, Beck, MW, Winkler, DA, Huang, B, Sibanda, W, Goyal, H & AME Big-Data Clinical Trial Collaborative Group 2018, 'Opening the black box of neural networks: methods for interpreting neural network models in clinical applications' Annals of Translational Medicine, vol. 6, no. 11, 216, pp. 1-11. https://doi.org/10.21037/atm.2018.05.32

Opening the black box of neural networks : methods for interpreting neural network models in clinical applications. / Zhang, Zhongheng; Beck, Marcus W; Winkler, David A; Huang, Bin ; Sibanda, Wilbert ; Goyal, Hemant ; AME Big-Data Clinical Trial Collaborative Group.

In: Annals of Translational Medicine, Vol. 6, No. 11, 216, 2018, p. 1-11.

Research output: Contribution to journalReview ArticleResearchpeer-review

TY - JOUR

T1 - Opening the black box of neural networks

T2 - methods for interpreting neural network models in clinical applications

AU - Zhang, Zhongheng

AU - Beck, Marcus W

AU - Winkler, David A

AU - Huang, Bin

AU - Sibanda, Wilbert

AU - Goyal, Hemant

AU - AME Big-Data Clinical Trial Collaborative Group

PY - 2018

Y1 - 2018

N2 - Artificial neural networks (ANNs) are powerful tools for data analysis and are particularly suitable for modeling relationships between variables for best prediction of an outcome. While these models can be used to answer many important research questions, their utility has been critically limited because the interpretation of the “black box” model is difficult. Clinical investigators usually employ ANN models to predict the clinical outcomes or to make a diagnosis; the model however is difficult to interpret for clinicians. To address this important shortcoming of neural network modeling methods, we describe several methods to help subject-matter audiences (e.g., clinicians, medical policy makers) understand neural network models. Garson’s algorithm describes the relative magnitude of the importance of a descriptor (predictor) in its connection with outcome variables by dissecting the model weights. The Lek’s profile method explores the relationship of the outcome variable and a predictor of interest, while holding other predictors at constant values (e.g., minimum, 20th quartile, maximum). While Lek’s profile was developed specifically for neural networks, partial dependence plot is a more generic version that visualize the relationship between an outcome and one or two predictors. Finally, the local interpretable model-agnostic explanations (LIME) method can show the predictions of any classification or regression, by approximating it locally with an interpretable model. R code for the implementations of these methods is shown by using example data fitted with a standard, feed-forward neural network model. We offer codes and step-by-step description on how to use these tools to facilitate better understanding of ANN.

AB - Artificial neural networks (ANNs) are powerful tools for data analysis and are particularly suitable for modeling relationships between variables for best prediction of an outcome. While these models can be used to answer many important research questions, their utility has been critically limited because the interpretation of the “black box” model is difficult. Clinical investigators usually employ ANN models to predict the clinical outcomes or to make a diagnosis; the model however is difficult to interpret for clinicians. To address this important shortcoming of neural network modeling methods, we describe several methods to help subject-matter audiences (e.g., clinicians, medical policy makers) understand neural network models. Garson’s algorithm describes the relative magnitude of the importance of a descriptor (predictor) in its connection with outcome variables by dissecting the model weights. The Lek’s profile method explores the relationship of the outcome variable and a predictor of interest, while holding other predictors at constant values (e.g., minimum, 20th quartile, maximum). While Lek’s profile was developed specifically for neural networks, partial dependence plot is a more generic version that visualize the relationship between an outcome and one or two predictors. Finally, the local interpretable model-agnostic explanations (LIME) method can show the predictions of any classification or regression, by approximating it locally with an interpretable model. R code for the implementations of these methods is shown by using example data fitted with a standard, feed-forward neural network model. We offer codes and step-by-step description on how to use these tools to facilitate better understanding of ANN.

U2 - 10.21037/atm.2018.05.32

DO - 10.21037/atm.2018.05.32

M3 - Review Article

VL - 6

SP - 1

EP - 11

JO - Annals of Translational Medicine

JF - Annals of Translational Medicine

SN - 2305-5847

IS - 11

M1 - 216

ER -