A Novel Fast Constructive Algorithm for Neural Classifier

Size: px

Start display at page:

Download "A Novel Fast Constructive Algorithm for Neural Classifier"

Lindsey Burke
5 years ago
Views:

1 A Novel Fast Constructive Algorith for Neural Classifier Xudong Jiang Centre for Signal Processing, School of Electrical and Electronic Engineering Nanyang Technological University Nanyang Avenue, Singapore Republic of Singapore Abstract This paper presents a novel fast algorith to construct feedforward neural networks for pattern classification tasks. The algorith constructs the neural network by adding eleentary neural eleents to the first layer, in response to the distribution of the patterns in the training set. Each eleentary neural eleent of the network is trained with different pattern subsets and fors a hyper-plane in the input space. The decision regions of these eleentary neural eleents are restricted and cobined through the second layer of siple fixed weights. This network perfors the nonlinear ap of input to output and realizes a piecewise linear classifier. The learning rule for the weights is very siple and the network can be very fast constructed and trained. In addition, the algorith autoatically deterines the nuber of hidden nodes.. Introduction Neural network is a powerful tool in the area of pattern classification because of its nonlinear processing capability. The ain drawbacks of neural networks are slowness of the learning process, local inia and deterination of the optial nuber of neurons. There is currently no satisfactory general answer to the question of how to choose a network-architecture for a given application. A onolithic network learns the global nature of approxiations by eans of increental adaptation of connection strengths in the direction of a decrease in error function based on soe learning rule. The extensive flat areas and local inia of a global error surface often require very long learning ties or result in unsuccessful training. To siplify the coplicated nonlinear learning task, the nonlinear processing can be approxiated by a piecewise linear processing. Various piecewise linear classifiers and piecewise linear neural networks were developed. Duda and Fossu [], Sklansky and Michelotti [] developed a supervised training algorith for the piecewise linear classifier. Kohonen s learning vector quantization [3] realized also a piecewise linear classifier. All of the require the nuber of prototypes (hidden nodes) to be defined before training. But the nuber of prototypes required can not be assued beforehand since it entirely depends on the shape of the classification regions. These are static neural networks. A constructive piecewise linear network has been introduced by Mezard and Nadal, called tiling algorith [4], [5]. Such an algorith will converge after a finite nuber of layers. It is easy to show that this is only et with binary patterns and is not the case with realvalued patterns as the whole of the input space is used. A siilar constructive algorith for piecewise linear network is upstart [6]. The upstart algorith utilizes soe single unit training algorith to train a parent unit, and then in a siilar anner builds and trains daughter units to correct any errors ade by the parent unit. It is easy to show that in order to correct any errors ade by the parent unit, the daughter unit ust separate soe subset of one class fro the entire sets of other classes with its hyper-plane. Unfortunately, alike tiling algorith, this is only et with binary patterns. Another constructive piecewise linear classifier has been proposed by Mangasarian, called ultisurface ethod [7]. In this ethod, linear prograing probles are recursively solved. However, this ethod is restricted on two pattern classes and obtains an error-free solution, where any pattern recognition probles can not be error-free solved. Additionally, linear prograing takes long coputation ties and requires uch eory space of coputer in use, in order to iniize the distance between the two parallel hyper-planes. The proposed HSC algorith constructs the network architecture, coencing with an epty network and, during training, increasing the size of the network by adding eleentary neural eleents to the hidden layer when necessary in order to accoodate the learning task. At the sae tie, the training data set is decoposed into different siple subsets, which can be fast griped by the constructed network. Through the decoposing of the training data, the network is constructed very quickly. In addition, the algorith autoatically deterines the nuber of hidden nodes. Each eleentary neural eleent in the first layer of the network fors a hyper-plane in the input space. The decision regions of these eleentary neural eleents are restricted and at sae tie cobined through the second layer of siple fixed weights. The constructed neural network, called HSC (high-speedconstructed) neural network, perfors the nonlinear ap of input to output and realizes a piecewise linear classifier in the input space.

2 . Constructing hidden layer of network Consider a classification proble with M classes C = C C... C M. Each learning saple in training set C is represented by an N-diensional pattern feature vector X = (x, x,... x N ) T, whose corresponding classes are known. Beginning with the whole training set C that contains coplete training saples, the algorith searches a good weight vector in soe sense, which projects the high diensional input vector X onto a one-diensional space y = X. In this one-diensional space, a bias can be easily deterined to separate a set of saples of one class only fro the reaining overlapped training saples. The weight vector and the bias ipleent a hyper-plane X b + = 0 in the input space, which classifies the training set C into two sets D () () { X X + 0} (3) = b = + b and R { X X 0 }. (4) The set D contains only training saples of one class and the overlapped set R contains training saples of ore than one class. A threshold neural unit is constructed with the weight vector, the bias b and a threshold transfer function. This threshold unit activates with unit output if and only if X + b 0. Otherwise it does not activate with zero output. If the set D contains saples of class C we say that the corresponding neural unit represents the class C. The weight vector and bias b construct the first neural threshold unit in the hidden layer of the network. The next unit to be added to the network ais to separate another set of saples of one class only fro the reaining overlapped set R. The procedure of finding the second weight vector and bias b to classify the R into D and R is as sae as that of finding and b, but now only with the reduced training saples R (R = C D ). Once the weight vector and bias b are found, the second threshold unit is added to the hidden layer and the set D is reoved fro the training set to get a new training set R (R =R D ). The construction of the hidden layer continues with each subsequent unit to classify a set of saples of one class only fro the reaining overlap set until the reaining training set R K contains only saples of one class. The construction of the hidden layer is then copleted and the nuber of the hidden nodes is K. Fig. illustrates a siple nonlinear separable classification task. The first hidden unit separates a set of points of one class only fro the reaining training (5) saples. After reoving this set fro the training set, the second hidden unit can separates another set of points of one class only fro the reaining training saples. After reoving it, the last hidden unit can successfully classify the reaining training saple into two classes without error. To iniize the nuber of hidden nodes, the weight vector i and bias b i should be so deterined that the set D i contains as any training saples as possible. A linear prograing [7] can be used for searching i and b i. Fisher s linear discriinant vector provides an optial project weight vector in soe sense. However, for a nonlinear separable task, both of the is not necessary optial although they consue a large coputation tie. The optial search ethod is coputationally too expensive. Even if an optial weight vector i can be obtained, it is only local optial. A series of local optial weight vectors still do not always result a inial nuber of hidden nodes. Therefore, we use a siple and relative good or reasonable (but not necessarily optial) ethod to deterine the weight vectors. For each class C, we define an opposite class C o = C C. e choose the weight vector, so that after the projection y = X, the difference between the y ean values of saples in class C and C o reaches axiu. That is o ( M M ) axiu. (6) Because the length of the vector is irrelevant for the proble, we obtain the largest difference between the y ean values when we chose the weight vector: Fig.. Classifying nonlinear separable saples by a series of hyper-planes. = 3 o ( M M ). (7) Therefore, we can obtain M weights. Obtained the weight vector, feature vectors of the training data are projected on a one-diensional feature space. In this onediensional feature space the bias b can be siply deterined. The weight vector i and bias b i of hidden unit

3 i is chosen aong these M weight vectors and biases which separates the axial nuber of saples of one class only fro the reaining training saples. This ethod to find the weight vector is very siple and consequently, very fast. However, it is not necessary optial and in soe cases, it ay far deviates fro the optial solution. To copensate that, we can try with a certain nuber of rando vectors and find one, which axiizes the nuber of saples that are correctly separated. If this rando vector can correctly separate ore saples than the weight vector deterined by equation (7) we adopt it as the weight vector of the hidden neural unit. 3. Constructing output layer of network The output units ust be deterined to coplete the network construction once the construction of the hidden layer is copleted. Having hidden unit representation the task of the output layer is to correctly classify the hidden unit representation. There will be M output units for an M classes task. Fig. illustrates a two-diensional exaple of the classification by hidden units. The filled sall circles are training saples of one class and the non-filled sall circles are those of another class. In Fig., each dotted line is ipleented by the corresponding hidden threshold unit. The class boundary should be the solid lines. That eans the acting region of dotted lines should be restricted (see exaples of the line section arked with in Fig. ) and soe sub-decision regions should be cobined (see exaples of the line section arked with // in Fig. ). These are the tasks of the output layer of the network. // // Fig.. An exaple of hyper-planes and decision boundary fored by proposed HSC network. Each hidden unit only correctly separates a portion of the training saples of one class fro a sub-training set. This eans that on the one hand the acting region of a hidden unit should be restricted to the sub-region corresponding to the sub-training set and on the other hand the various sub decision regions of the sae class should be cobined. It is easy to see fro the constructing procedure of the hidden layer that the activate of the hidden unit j should be restricted in the region in which all hidden units i, i<j, do not activate. Furtherore, the activate of the hidden unit j should not be affected by any hidden unit k, kj. This restriction can be ipleented through the output layer with threshold units, in which the weight w i between the hidden node i and output node is w i i, if hidden unit i represents class C = i, otherwise. The transfer functions in the output layer are also threshold functions. It is easy to see that the output units defined above restrict the acting regions of the hidden units and cobine various sub-decision region of the sae class. The cobination of sub-decision regions is obvious. The restriction of the acting regions is based on the fact: i i+ -j. By using such an output layer, the dotted lines in Fig. will be constricted and cobined into the decision boundary as shown by the solid lines. For an M classes task, the HSC neural network has M outputs x 3, x 3,... x M 3. x 3 = corresponds to the entire decision region of class C, =,..., M. It is easy to see that, when a pattern X is given as the input to the network, there exists one output and only one output is one and all other outputs are zeros. The proposed HSC neural network learns the set of exeplary input-output pairs in serial. hen the first neural eleent in the hidden layer copletes learning and there still exist training patterns that can not be correctly classified the second eleent is added to this layer of the network, then the third... In the recall phase the HSC network works parallel. Because the learning procedure is very fast, we call this neural network HSC (high-speedconstructed) neural network. It realizes a piecewise linear classifier and copletes nonlinear apping with the first neural layer and the second siple fixed-weight layer. In addition, the nuber of hidden nodes is autoatically deterined by the algorith. For any classification task over a finite set of training saples HSC will generate a single layer threshold network which correctly classifies all saples in the training set. That eans that the HSC network perfors an error-free classification. For a statistical classification task, especially if the training saples are heavy overlapped and the theoretical Bayes classifier has a high classification error, HSC network will produce a large nuber of hidden nodes to achieve a error-free classification and therefore ay lead to a poor generalization perforance. In such cases we can reove soe hidden units fro the HSC network, which only correctly separate a sall nuber of training saples. In this way we can reduce the network size and iprove its generalization perforance. It is worth noting that the HSC network can only perfor a piecewise linear decision boundary. Consequently, its generalization capacity is liited. However, if the feature vector of training saples is (8) (9) 3

4 transfored into a higher diensional space fored with its high order polynoial ters, the HSC network then can for a arbitrary nonlinear boundary in the original feature space. In this way the generalization perforance of the HSC network can be iproved. The HSC algorith sequentially generates neural units, trains each unit only using the reaining overlapped training set. A coplicated learning task represented by a coplicated data structure of the whole training set will then be decoposed into a nuber of siple sub-probles represented by different training subsets. The HSC algorith ais to fast grip these subsets one by one. This can be successfully achieved because the coplicated data structures out of the training sub-set will not ebarrass the training. Thus, this training procedure overcoes the probles such as Moving-Target or Herd-Effect, Crosstalk and Catastrophic Interference, which often occur in a onolithic network learning. linear boundary is overcoe. For a coparison we applied the Rprop algorith [] to training a single hidden layer network with 40 hidden nodes. After 6 seconds of training (3850 epochs), the network achieved an error free classification. The decision region is depicted in Fig. 6. Fig. 3. Solution by HSC network of 36 hidden nodes. Fig. 4. Solution by LP ethod (96 hidden nodes). 4. Experiental studies To illustrate the nonlinear learning capability and the learning speed of the HSC networks we eployed two learning tasks in our experiental studies. One is the difficult double spiral proble and another is the high diensional realistic sonar data proble. In experients we used MATLAB in a Pentiu II 450 MHz PC to siulate our proposed algorith and learning processes. The double spiral proble has becoe a coon benchark for connectionist learning algoriths. According to Bau and Lang, a -50- BP network sees unable to find a correct solution to this proble [9]. A network with shortcut connections solved this proble after 0,000 iterations of the BP algorith [0]. The HSC network was applied to this double spiral proble of 94 saples. In the 50 trials, HSC network achieved the perfect classification with an average network size of 34 hidden nodes and an average training tie of only 0. seconds. An exaple decision region of the HSC network is shown in Fig. 3. Mangasarian s ulti-surface ethod based on linear prograing (LP) described in [7] was also applied to this double spiral proble. After the training tie of.86 seconds, 48 hyper-planes were obtained, which could correctly classify all patterns of the double spiral proble. The decision region is shown in Fig. 4. In our experients, LVQ network with 00 hidden nodes and hours training tie can not obtain an error-free solution. The above algoriths realize piece wise linear classifiers. The piecewise linear decision boundaries are not surprise. Because of the highly nonlinear nature of the double spiral proble, better solutions ay be obtained by using the high order HSC networks. Fig. 5 shows the decision region fored by the proposed HSC network with hidden nodes and the input vector containing the second order polynoial ters of the feature vector. For constructing and training this network, only 0.7 seconds were needed. One sees that the restriction of the piece wise Fig. 5. Solution by -order HSC network of hidden nodes. Fig. 6. Solution by Rprop algorith with 40 hidden nodes. These experients illustrated the capability of the HSL to solve a high nonlinear proble that is extreely hard to solve using soe existed approaches. The extreely short training tie was shown in these experients. Sonar data [] set was contributed to the benchark collection by Terry Sejnowski, now at the Salk Institute and the University of California at San Deigo. The data set was developed in collaboration with.r. Paul Goran of Allied- Signal Aerospace Technology Center. The diensional saples were divided into the 04-eber training set and the 04-eber test set. Using neural networks to solve this classification proble, the network had 60 inputs and output units, one indicating a cylinder and the other a rock. Because this is a statistical classification proble, in the training processes of HSC networks we reoved the hidden nodes out of the network that only classified a single training saple fro the training set. In the 50 trails of training, the proposed HSC algorith generated on average 8 hidden nodes in 0.6 seconds and achieved an average classification error of.7% on the training set and.4% on the test set. Mangasarian s ulti-surface ethod based on linear prograing generated 4 hidden nodes in 47 4

5 seconds of training and achieved an error-free classification on the training set but a classification error of 5.96% on the test set. LVQ networks with 48 hidden nodes could achieve an average classification error of 5.77% on the training set and.98% on the test set. Rprop algorith was used to train networks with hidden nodes. After 600 epochs of training (9.4 seconds) the networks achieved the error-free classification on the training set and had average classification error of 9.86% on the test data. Above experiental results again clearly showed the extreely short training tie of the proposed HSC network. Its generalization perforance (classification error on the test data) is also better than other two piecewise classifiers (LVQ and ulti-surface ethod based on linear prograing). This experient also illustrated the capability of the HSC network to solve the high diensional classification probles. However, for this sonar data proble, the Rprop algorith showed its advantage of generalization capability. 5. Conclusions A novel fast algorith to construct feed-forward neural networks for pattern classification tasks has been presented. The proposed HSC algorith constructs the neural network by adding eleentary neural eleents to the first layer rather than by odifying the weights of connections in a pre-wired network, in response to the observed distribution of the patterns in the training set. Each eleentary neural eleent of the HSC network is trained with different pattern subsets and fors a hyper-plane in the input space. A coplicated learning task represented by a coplicated data structure of the whole training set is then decoposed into a nuber of siple sub-probles represented by different training subsets. These subsets are fast griped one by one by the eleentary neural eleents in the hidden layer. The decision regions of these eleentary neural eleents are restricted and at sae tie cobined through the second layer of siple fixed weights. This neural network perfors the nonlinear ap of input to output and realizes a piecewise linear classifier. By transforing the feature vector of training saples into a higher diensional space fored with its high order polynoial ters, the HSC network can for a arbitrary nonlinear boundary in the original feature space. In this way the restriction of the piece wise linear boundary can be overcoe and the generalization perforance of the HSC network can be iproved. The ain advantage of the HSC network is that its learning rate is very rapid and the required nuber of the neural eleents is autoatically deterined. Two experiental studies show the rapid learning rate and the acceptable generalization perforance of this new developed HSC network. References [] R.O. Duda and H. Fossu, Pattern classification by iteratively deterined linear and piecewise linear discriinate function, IEEE Trans. Electron. Coput., Vol. 5, pp. 0-3, 966. [] J. Sklansky and L. Michelotti, Locally trained piecewise linear classifiers, IEEE Trans. PAMI, Vol., pp.0-, 980. [3] T. Kohonen, Learning vector quantization, Self- Organization and Associative Meory, pp. 99-0, Springer-Verlag, 984. [4] M. Mezard and J.P. Nadal, Learning in feedforward layered networks: The tiling algorith, J. Phys.: Math. and Gen. 989, pp [5] J.P. Nadal, Study of a growth algorith for a feedforward network, International Journal of neural systes, Vol., 989. [6] M. Frean, The upstart algorith: A ethod for constructing and training feedforword neural networks, Neural Coputation Vol., pp.98-09, 990. [7] O.L. Mangasarian, Multisurface ethod of pattern separation, IEEE Trans. Infor. Theory, Vol. 4, No. 6, pp , Noveber 968. [8] Y.Q. Cheng, Y. M. Zhuang and J.Y. Yang, "Optial Fisher discriinant analysis using the rank decoposition," Pattern Recognition, Vol. 5, pp. 0-, 99. [9] E.B. Bau and K. J. Lang, Constructing hidden units using exaples and queries, Advances in neural inforation processing systes Vol. 3 pp San Mateo, CA: Morgan Kaufann. 99. [0] K.J. Lang and M. J. itbrock. Learning to tell two spirals apart, Proc. of the Connectionist Models Suer School, Morgan Kaufann, pp. 5-59, 988. [] M. Riediller and H. A. Braun, A direct adaptive ethod for faster backpropagation learning: The RPROP algorith, IEEE ICNN-93, San Francisco, CA, 993, pp [] R.P. Goran, T.J. Sejnowski, Analysis of hidden units in a layered network trained to classify Sonar targets, Neural Networks, Vol., pp ,

TensorFlow and Keras-based Convolutional Neural Network in CAT Image Recognition Ang LI 1,*, Yi-xiang LI 2 and Xue-hui LI 3

2017 2nd International Conference on Coputational Modeling, Siulation and Applied Matheatics (CMSAM 2017) ISBN: 978-1-60595-499-8 TensorFlow and Keras-based Convolutional Neural Network in CAT Iage Recognition