Regularizer design for fault tolerant neural networks

Abstract

Fault tolerance is an important issue in neural networks. While there are some fault tolerant training algorithms, they often have their own tradeoffs. Recently, an open node fault regularizer (ONFR) was proposed to obtain fault tolerant radial basis function (RBF) networks. This regularization technique is computationally simple and does not alter the network structure. Following similar concept, this thesis introduces an ONFR for multilayer feedforward networks (MFN). With the linearization technique, the training objective function can be decomposed into two simple terms, the training error and the ONFR. Gradient based learning methods can then be employed to obtain fault tolerant networks. Notice that the goal of training a fault tolerant network is to minimize the generalization error over faulty networks, however, the current ONFRs minimizes the training error over faulty networks only. This thesis thus presents a design strategy for the ONFR to optimize the generalization ability for faulty RBF networks. A mean prediction error (MPE) formula, which consists of the training error and the trained weight, is developed to predict the generalization ability of a faulty RBF network. Thus, we can optimize the ONFR parameter in terms of the generalization ability over faulty networks efficiently. The formula can also be used to select an appropriate RBF width.