Computing Second Derivatives in Feedforward Networks: A Review

Wray L. Buntine, Andreas S. Weigend

Research output: Contribution to journalArticleResearchpeer-review

60 Citations (Scopus)

Abstract

The calculation of second derivatives is required by recent training and analysis techniques of connectionist networks, such as the elimination of superfluous weights, and the estimation of confidence intervals both for weights and network outputs. We here review and develop exact and approximate algorithms for calculating second derivatives. For networks with |u| weights, simply writing the full matrix of second derivatives requires 0(|w|2) operations. For networks of radial basis units or sigmoid units, exact calculation of the necessary intermediate terms requires of the order of 2h + 2 backward/forward-propagation passes where h is the number of hidden units in the network. We also review and compare three approximations (ignoring some components of the second derivative, numerical differentiation, and scoring). Our algorithms apply to arbitrary activation functions, networks, and error functions (for instance, with connections that skip layers, or radial basis functions, or cross-entropy error and Softmax units, etc.).

Original languageEnglish
Pages (from-to)480-488
Number of pages9
JournalIEEE Transactions on Neural Networks
Volume5
Issue number3
DOIs
Publication statusPublished - 1 Jan 1994

Cite this