TY - JOUR

T1 - Computing Second Derivatives in Feedforward Networks

T2 - A Review

AU - Buntine, Wray L.

AU - Weigend, Andreas S.

PY - 1994/1/1

Y1 - 1994/1/1

N2 - The calculation of second derivatives is required by recent training and analysis techniques of connectionist networks, such as the elimination of superfluous weights, and the estimation of confidence intervals both for weights and network outputs. We here review and develop exact and approximate algorithms for calculating second derivatives. For networks with |u| weights, simply writing the full matrix of second derivatives requires 0(|w|2) operations. For networks of radial basis units or sigmoid units, exact calculation of the necessary intermediate terms requires of the order of 2h + 2 backward/forward-propagation passes where h is the number of hidden units in the network. We also review and compare three approximations (ignoring some components of the second derivative, numerical differentiation, and scoring). Our algorithms apply to arbitrary activation functions, networks, and error functions (for instance, with connections that skip layers, or radial basis functions, or cross-entropy error and Softmax units, etc.).

AB - The calculation of second derivatives is required by recent training and analysis techniques of connectionist networks, such as the elimination of superfluous weights, and the estimation of confidence intervals both for weights and network outputs. We here review and develop exact and approximate algorithms for calculating second derivatives. For networks with |u| weights, simply writing the full matrix of second derivatives requires 0(|w|2) operations. For networks of radial basis units or sigmoid units, exact calculation of the necessary intermediate terms requires of the order of 2h + 2 backward/forward-propagation passes where h is the number of hidden units in the network. We also review and compare three approximations (ignoring some components of the second derivative, numerical differentiation, and scoring). Our algorithms apply to arbitrary activation functions, networks, and error functions (for instance, with connections that skip layers, or radial basis functions, or cross-entropy error and Softmax units, etc.).

UR - http://www.scopus.com/inward/record.url?scp=0028424954&partnerID=8YFLogxK

U2 - 10.1109/72.286919

DO - 10.1109/72.286919

M3 - Article

AN - SCOPUS:0028424954

SN - 1045-9227

VL - 5

SP - 480

EP - 488

JO - IEEE Transactions on Neural Networks

JF - IEEE Transactions on Neural Networks

IS - 3

ER -