A Piecewise Linear Network Classifier

Transcription

1 Proeedings of International Joint Conferene on eural etworks, Orlando, Florida, USA, August -7, 007 A Pieewise Linear etwork Classifier Abdul A. Abdurrab, Mihael T. Manry, Jiang Li, Sanjee S. Malalur and Robert G. Gore Abstrat A pieewise linear network is disussed whih lassifies -dimensional input etors. The network uses a distane measure to assign inoming input etors to an appropriate luster. Eah luster has a linear lassifier for generating lass disriminants. A training algorithm is desribed for generating the lusters and disriminants. Theorems are gien whih relate the network s performane to that of nearest neighbor and k-nearest neighbor lassifiers. It is shown that the error approahes Bayes error as the number of lusters and patterns per luster approah infinity. I. ITRODUCTIO eedforward neural nets with the uniersal approximation F property [,] mimi Bayes disriminants [3,4] and hae been suessfully used for many lassifiation tasks. Howeer, training time is slow, and onergene of the lassifiation error to Bayes error has not been shown. Suh onergene theorems do exist for nearest neighbor lassifiers (C) and k-nearest neighbor lassifiers (k-c) [5, 6], whih also hae the adantage of being easy to design. Howeer, the C and k-c are rarely used beause it is ery time-onsuming to apply them. Pieewise linear networks hae long been used for funtion approximation and lassifiation [7-] where speed of operation and simpliity are ery important. One approah is to train a multilayer pereptron (MLP) haing pieewise linear atiations [, 3]. This approah is useful in hardware implementations, and results in a ontinuous approximation. Howeer, if we are willing to gie up ontinuous approximation, a simpler pieewise linear network an be deised. In this paper we deelop a disontinuous pieewise linear network lassifier (PLC), based upon the work in [4]. The struture, operation and training of the PLC are desribed in setion II. In setion III the estimation of Bayesian a- posteriori probabilities by the PLC is disussed, along with its relation to the C and k-c. Simulations and performane omparisons are presented in setion IV. Manusript reeied January 3, 007 A. A. Abdurrab, M. T. Manry and S. S. Malalur are with the Department of Eletrial Engineering, Image Proessing and eural etworks Lab, Uniersity of Texas at Arlington, Arlington, TX USA ( abdulaziz.abdurrab@uta.edu, manry@uta.edu, smalalur@uta.edu). R. G. Gore is with Lokheed Martin, Mission Systems, Fort Worth, TX 760, USA. J. Li is with ECE Department and VMASC at Old Dominion Uniersity, orfolk VA 359, USA ( JLi@odu.edu). II. PIECEWISE LIEAR ETWORK CLASSIFIER A. Struture and Operation As shown in figure, the PLC has input elements in the first layer, the hidden units in the seond and outputs in the third. The -dimensional input etor x f has elements x f (n) with means and standard deiations µ(n) and σ(n) respetiely where n. First, x f is normalized as xf ( n) xf ( n) µ ( n) / σ( n) () The normalized (+)-dimensional etor x is formed by augmenting x f as x= ( x T :) T () f The hidden layer onsists of K lusters, eah haing an - dimensional mean etor m, where K. A Eulidean distane measure d( ) is used in lustering. Eah luster also has a weight matrix A of dimension by (+), where is the number of lasses in the lassifiation problem. d() Fig.. PLC Struture Gien an input etor x, we find luster index suh that d(x,m ) is minimized. Then we form the output etor y as y = A x (3) The estimate of the orret lass i is gien by i = arg max[ y ] (4) = = = K where y i is the i th element of output etor y and i. B. Training A lassifiation problem typially inoles a feature spae i i A A A K X/07/$ IEEE

2 with numerous feature etors or samples that hae to be assigned arious lass labels. In superised learning, the training dataset inludes the lass label, i, of eah of the feature etors. The label is transformed into an - dimensional target etor t suh that ( ) otherwise + b i = i p t pi = (5) b where p and b is a positie integer. Before the network is used for lassifiation, it has to be trained, whih inoles designing the weight matries A and the luster mean etors m. Clustering of the input etors x p is performed using the self-organizing map (SOM) [5]. Elaborating on (3), the elements of the output etor y, are alulated as + (6) y = a x pi in pn n= where a in denotes an element from the weight matrix A matrix belonging to row i and olumn n. The mean-squared error (MSE) for the th luster is then gien by ( ) + E = qi in qn ( ) t a x (7) i= n= where () is the number of feature etors in luster and t qi is the desired output whih ould be +b or -b. The error gradient with respet to a jm is gien by + = ajm On further simplifiation we get + = tqj xqm ajn xqnxqm ajm n= ( ) tqj ajnxqn x (8) qm n= (9) Equation (9) an be written in terms of auto-orrelation and ross-orrelation elements, r and respetiely, as + = ( j, m) ajnr( n, m) a (0) jm n= Equating the gradient in (0) to zero, in order to minimize the error, yields + (, ) (, ) j m = a r n m () n= The set of equations in () is soled using the Output Reset (OR) method [6-8] and a Shmidt based linear equation soler [9-0]. After the training is omplete, the PLC s m etors and A matries are fixed and an be used to lassify alidation data. jn III. EURAL ETWORKS AD THE BAYES OPTIMAL DISCRIMIAT FUCTIO A. eural networks and Bayes Disriminant Oer the past seeral years, neural networks hae been used for many tasks, inluding lassifiation and funtion approximation. There hae been many useful theoretial results onerning their apabilities inluding the following [4] theorem. Theorem: When neural net lassifiers are trained to minimize the mean-squared error (MSE), the MSE approahes a onstant alue plus the expeted squared error between the neural net output and Bayes disriminant, as the number of training patterns approahes infinity. Speifially, lim tpi yi ( p ) x = p= i= () i= ( i( x) i( x) ) E b y + a where a is a onstant, independent of p, t pi is defined in (5), and y i (x p ) is the i th output of the network. The Bayes disriminant b i (x), is the probability that the i th lass is orret, gien x, whih is written as P(i x). The aboe theorem, howeer, leaes room for some doubts: ) The neural network s probability of error is not bounded or shown to onerge to Bayes probability of error. ) The mean-square error in the theorem treats positie and negatie errors the same. Howeer, it is good if the orret lass disriminant is larger than desired, but bad if it is smaller [6-8]. This problem leads to suboptimal networks. 3) The theorem makes no use of the neural network s struture. It applies equally well to any disriminant designed by minimizing the MSE. B. earest eighbor and the PL Classifiers In the C, an input feature etor is assigned to the lass of its nearest training sample, usually determined ia the Eulidean distane. The feature spae is diided into onex regions or lusters eah haing a speifi lass label. Speifially, the distane measure performs a Voronoi tessellation of the -dimensional input feature spae. ow we onsider the relationship between the PLC and C. As K approahes infinity, the onex Voronoi ells in the feature spae get smaller in olume and the optimal deision boundaries in eah luster beome linear. Hene, eah luster an hae its own linear disriminant and oerall, a more omplex deision boundary is ahieed. Therefore, for a gien alue of K, the PLC should perform better than the C. We begin to address this idea in the following lemmas. Lemma : If a PLC and C hae the same distane measure and idential luster mean etors, the PLC has at

3 least as good a performane as the C. Proof: Sine the PLC s augmented input etor x inludes the onstant ; the PLC s output etor y an hae a for the same lass as piked by the C. Other lass outputs an be 0. Suppose that the th C luster belongs to lass i. If we would like the PLC s th luster to always map patterns to lass i, the elements of the by (+) matrix A are defined as m= i and n= + a ( m, n) = (3) 0 otherwise When a PLC s A matrix initially performs worse than the C for a gien th luster, replae A with A as defined in (3). Lemma : If a PLC and a C hae the same distane measure and idential luster mean etors, then as the number of lusters, K and patterns, approah infinity, P P P P (4) e( PLC) e( C) where P e(plc), P e(c) and P respetiely denote the PLC, C and Bayes probabilities of error. Proof: As K approahes infinity, we know that [5, 6] P P P (5) ( ) e C Using lemma and (5) yields the result of (4). C. Conergene of the PLC Error Probability We now wish to be more speifi and ealuate the aerage probability of error for the PLC, as the amount of training data inreases. First, reall the following result desribing the onergene of the k-c probability of error, P e(k-c). Here k is the number of nearest training etors or samples to the test sample x. Lemma 3 [5, 6]: As k and ( /k) approah infinity, the k- C an be iewed as an attempt to estimate the a- posteriori probabilities from the training samples. Under this ondition, k-c beomes optimal and lim P = P (6) k, k e(k-c) We now inestigate the relationship between Bayes disriminant and the PLC in more detail. We start by obsering that the lassifiation error for the th luster of a PLC is gien by (5). Let us take into onsideration only the feature etors that hae been assigned lass label i. In this ase, the lassifiation error would be gien by + Ei, = tqi ainxqn (7) n= The partial deriatie of the error with respet to the weight matrix elements, a im, is gien by: + = tqi ainxqn xqm aim () (8) n= Before going any further, we onsider how letting K and () approah infinity effets the range and ariability of the input feature etors x. Assuming the feature spae is a bounded ompat subset of R, eah luster falls within a ell of the Voronoi tessellation. As K and () simultaneously inrease towards infinity, the maximum radius of eah luster dereases towards zero. The range of alues the elements of a feature etor in a luster an take dereases so muh that they beome onstant, i.e, ariability within a luster approahes zero. Within a gien luster, as x beomes independent of the pattern number p, we an replae x pn with mean etor element m n in (8). We also equate the gradient to zero, so as to minimize the MSE. Equation (8) redues to + tqi ainmn mm 0 () = (9) n= whih beomes + tqi ainmn = 0 (0) n= Sine the terms a in and m n are onstants, we an replae the sum oer n by the single onstant a i(+) whih yields This an be written as whih yields tqi a i( + ) = 0 () a tqi = () a () i( + ) qi ( ) = i + t (3) Without loss of generality, we an represent t qi as t = δ i q i (4) qi ( ( ) ) where δ is the Kroneker delta funtion. In turn (3) beomes a ( + ) i = δ ( i ( q) i) (5) δ(i (q) i) equals if i (q) equals i, so, the numerator summation equals the number of training etors in luster that belong to lass i. This number ould be represented by (,i), hene reduing (5) to (,) i ai( ) = + (6) whih onerges to the a-posteriori probability P(i x). For a gien luster, a i(+) is largest for the lass i whih has the most input etors in the luster. This is preisely how the k-c makes deisions. So, eah PLC luster with () members emulates a k-c deision with k = (). Summarizing the results aboe, we hae the following

4 lemma. Lemma 4: As K and () approah infinity, the output of a PLC approximates the a-posteriori probability funtions of the lass labels, gien the input etor. Under this ondition, the PLC hene beomes optimal and lim P e(plc) = P (7) K, ( ) window. For this omparison too, the PLC and C are allowed to design and use their own set of luster enter etors. The results of this omparison are shown in fig. 4. As in the preious examples, the PLC outperforms the C. IV. SIMULATIO RESULTS AD COMPARISOS In this setion, we ompare the PLC with the C, MLP and a pieewise linear MLP (PLMLP) [] using different data sets. Eah data set onsists of a training file and a alidation file. A. PLC s C Here, we ompare the PLC to a C haing the same number of luster mean etors. Three data sets are used for the omparison. ) grng [] This geometri shape reognition data set has four shapes haing different degrees of rotation, saling, translation, and oblique distortions. The shapes are ellipse, triangle, quadrilateral, and pentagon. Eah image is 64 x 64 pixels. The input etors ontain 6 ring-wedge features. For this omparison, the C and PLC, both use the same set of luster enter etors for lassifiation. In fig., the C performs poorly beause it is fored to use the same lusters as the PLC, for this example only. ) gong [] The raw data onsists of images from hand printed numerals olleted from 3,000 people by the Internal Reenue Serie. Images are 3 by 4 binary matries, whih are saled to remoe harater size ariation. The feature set ontains 6 elements. The 0 lasses orrespond to the Arabi numerals. For this omparison, the PLC and C are allowed to design and use their own sets of luster enter etors. The results of this omparison are shown in fig. 3. The linear lassifier in eah PLC luster allows the PLC to outperform the C. 3) omf8 [3] This data set onsists of texture features orresponding to an image segmentation problem. Eah segmented region is separately histogram equalized to 0 leels. Then the joint probability density of pairs of pixels separated by a gien distane and a gien diretion is estimated. We use 0, 90, 80, 70 degrees for the diretions and, 3, and 5 pixels for the separations. The density estimates are omputed for eah lassifiation window. For eah separation, the oourrenes for the four diretions are folded together to form a triangular matrix. From eah of the resulting three matries, six features are omputed: angular seond moment, ontrast, entropy, orrelation, and the sums of the main diagonal and the first off diagonal. This results in 8 features for eah lassifiation Fig.. Comparison of PLC and C for shape reognition data Fig. 3. Comparison of PLC and C for numeral reognition problem B. PLC s MLP s PLMLP Here, we ompare the PLC to a sigmoidal MLP and to a PLMLP []. Unlike the PLC, the PLMLP produes ontinuous mappings. Two data sets are used for the omparison. ) F7C This prognostis data set onsists of parameters that are aailable in the basi health usage monitoring system (HUMS) by Bell Heliopter Textron. The data was obtained from the M430 flight load leel surey onduted in Mirabel Canada. Eah input etor ontains 7 elements. The 39 lasses represent different

5 flight maneuers like taking off, landing, turning right or left et. The results of this omparison are shown in fig. 5. In this example, the PLC outperforms both the MLP and PLMLP. Fig. 6. Comparison of PLC, PLMLP and MLP for phoneme reognition problem Fig. 4. Comparison of PLC and C for segmentation problem C. Comparisons Based On the umbers of Multiplies Here, we re-do the plots of subsetion B, so that the three full trained networks are ompared based upon the numbers of multiplies required to proess one input etor. For the sigmoidal MLP it is assumed that two multiplies are required to ealuate the atiation. The PLMLP, in ontrast, requires only one multiply for alulating the atiation. Figures 7 and 8 show that a PLC requires fewer multipliations to obtain the same lassifiation error as ompared to the MLP and PLMLP. Fig. 5. Comparison of PLC, PLMLP and MLP for prognostis ) speeh - Speeh samples are first pre-emphasized and are onerted into frequeny domain by taking their DFT. They are then passed through Mel filter banks and the inerse DFT is applied on the output to get Mel- Frequeny Cepstrum Coeffiients (MFCC). Eah of MFCC(n), MFCC(n)-MFCC(n-) and MFCC(n)- MFCC(n-) would hae 3 features, whih results in a total of 39 features. Eah lass orresponds to a phoneme. The results of this omparison are shown in fig. 6. The PLC outperforms the PLMLP and the MLP for larger networks. V. COCLUSIOS In this paper we hae desribed a pieewise linear network lassifier (PLC) and gien a training algorithm for it. This algorithm has fast onergene speed beause sets of linear equations are soled in eah luster. Using existing theorems for the C and k-c, we hae proed that the output of the PLC approximates a Bayes optimal disriminant when trained to minimize the mean square error (MSE). Using seeral data sets, we hae shown that the PLC often performs better than equialent Cs, MLPs, and PLMLPs. It an also be stated that the PLC always trains faster than the MLP and often faster than the C, whih usually requires a lot more lusters. Future work will ompare arious alternatie methods for training the linear lassifiers. PL pruning tehniques [4] will also be extended to the PLC.

6 Fig. 7. Comparison of PLC, PLMLP and MLP for prognostis data, based on number of multiplies Fig. 8. Comparison of PLC, PLMLP and MLP for phoneme reognition problem, based on number of multiplies REFERECES [] K. Hornik, M. Stinhombe, and H. White, Multilayer Feedforward etworks Are Uniersal Approximators, eural etworks, Vol., o. 5, 989, pp [] K. Hornik, M. Stinhombe, and H. White, Uniersal Approximation of an Unknown Mapping and its Deriaties Using Multilayer Feedforward etworks, eural etworks, ol. 3, 990, pp [3] Mihael D. Rihard and Rihard P. Lippman, eural etwork Classifiers estimate Bayesian a-posteriori probabilities, eural Computation, ol. 3, no. 4, pp , 99. [4] Dennis W. Ruk, Steen K. Rogers, Matthew Kabrisky, Mark E. Oxley and Brue W. Suter, The Multilayer Pereptron as an approximation to a Bayes optimal disriminant funtion, IEEE Trans eural etworks, T-(4):96-98, 990. [5] K. Fukunaga, Introdution to Statistial Pattern Reognition, nd ed., Aademi Press, 990. [6] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classifiation, John Wiley & Sons, nd ed., 00. [7] L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classifiation and Regression Trees, Wadsworth, Belmont, CA, 984. [8] D. B. Fogel, An information riterion for optimal neural network seletion, IEEE Trans. eural etworks, ol., no. 5, pp , Sept.99. [9] J. H. Friedman, Multiariate adaptie regression splines, Annals of Statistis, ol. 9, no., pp. -4, 99. [0] S. Subbarayan, K. Kim, M. T. Manry, V. Dearajan and H. Chen, Modular neural network arhiteture using pieewise linear mapping, 30th Asilomar Conferene on Signals, Systems & Computers, ol., pp. 7-75, o [] W. Li, J.-. Lin, and R. Unbehauen, Canonial representation of pieewise polynomial funtions with nondegenerate linear domain partitions, IEEE Trans. Ciruits and Systems I: Fundamental Theory and Appliations, ol. 45, no. 8, pp , Aug [] D.R. Hush and B. Horne, Effiient algorithms for funtion approximation with pieewise linear sigmoidal networks, IEEE Trans. eural etworks, Vol. 9, o. 6, pp. 9-4, 998. [3] E.F. Gad, A.F. Atiya, S. Shaheen, A. El-Dessouki, A new algorithm for learning in pieeswise-linear neural networks, eural etworks 3 (000), pp [4] H. Chandrasekaran, J. Li, W.H. Delashmit, P.L. arasimha, C. Yu and M.T. Manry, Conergent Design of Pieewise Linear eural etworks, eurocomputing, ol. 70, 007, pp [5] T. Kohonen, Self-Organization and Assoiatie Memory, nd ed., Springer-Verlag, 987. [6] L-M Liu, M.T. Manry, F. Amar, M.S. Dawson, and A.K. Fung, "Iteratie Improement of Image Classifiers Using Relaxation," Conferene Reord of the Twenty-Eighth Annual Asilomar Conferene on Signals, Systems, and Computers, ol., 0/3/94 to //94, pp [7] Jiang Li, Mihael T. Manry, Li-Min Liu, Changhua Yu, and John Wei, Iteratie Improement of eural Classifiers, Proeedings of the Seenteenth International Conferene of the Florida AI Researh Soiety, May 004, pp [8] R.G. Gore, Jiang Li, Mihael T. Manry, Li-Min Liu, Changhua Yu, and John Wei, "Iteratie Design of eural etwork Classifiers through Regression". International Journal on Artifiial Intelligene Tools, Vol. 4, os. & (005) pp [9] F. J. Maldonado, M. T. Manry, Tae-Hoon Kim, "Finding optimal neural network basis funtion subsets using the Shmidt proedure", Proeedings of the International Joint Conferene on eural etworks, ol., pp , July 003. [0] F. J. Maldonado, M. T. Manry, "Optimal Pruning of Feed-forward eural etworks Based upon the Shmidt Proedure," The 36th Asilomar Conferene on Signals, Systems, & Computers '0, pp [] H. C. Yau, M. T. Manry, "Iteratie Improement of a earest eighbor Classifier," eural etworks, Vol. 4, 99, pp [] W. Gong, H. C. Yau, and M. T. Manry, "on-gaussian Feature Analyses Using a eural etwork," Progress in eural etworks, ol., 994, pp [3] R.R. Bailey, E. J. Pettit, R. T. Borohoff, M. T. Manry, and X. Jiang, "Automati Reognition of USGS Land Use/Coer Categories Using Statistial and eural etwork Classifiers," Proeedings of SPIE OE/Aerospae and Remote Sensing, April -6, 993, Orlando Florida.