AUTOMATIC gender classification based on facial images

Transcription

1 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 1 Gender Cassification Using a Min-Max Moduar Support Vector Machine with Incorporating Prior Knowedge Hui-Cheng Lian and Bao-Liang Lu, Senior Member, IEEE Abstract Gender cassification based on facia images is a arge-scae, compicated two-cass cassification probem by nature. The reason is that few knowedge is known about the mechanism of human beings discriminating mae and femae from facia images, and a arge number of various facia images is required to train the gender cassifier. This paper presents a method for deaing with the gender cassification probem using min-max moduar support vector machines. The main contributions of this work are twofod. One is that some expicit domain or prior knowedge concerning the probem to be soved can be easiy incorporated into the procedure of dividing the gender cassification probem into a series of reativey smaer and simper two-cass subprobems. The other is that the gender cassification probem can be efficienty soved by using highperformance computers such as custer computing systems. To demonstrate the effectiveness of the method, we perform some experiments on the CAS-PEAL face database, which contains 14,384 facia images of 1,040 individuas with different poses. The experimenta resuts indicate that the proposed method has severa attractive features. 1) Incorporation of domain or prior knowedge into decomposition of the gender cassification probem can both speed up training and improve generaization performance. 2) The method is appreciaby faster than the existing approaches that are based on conventiona support vector machines and muti-ayer perceptrons. 3) The method scae up to arge-scae, compex gender cassification probems. 4) The gender cassifier impemented by the method has more powerfu capabiity of incrementa earning in comparison with the existing approaches. Index Terms Gender cassification, incorporation of knowedge, part-versus-part strategy, min-max moduar support vector machine, muti-ayer perceptron, parae earning, facia image, oca binary pattern. I. INTRODUCTION AUTOMATIC gender cassification based on facia images has attracted much attention due to its potentia appications such as human identification, smart human computer interface, visua surveiance, and robot with vision. In addition, gender cassification is a arge-scae, compicated twocass pattern cassification probem, because few knowedge is known about the mechanism of human beings discriminating mae and femae by facia images, and a arge number of various facia images is required to train the gender cassifier. Manuscript received January 20, 2006; revised November 18, Asterisk indicates corresponding author. This work was supported in part by the Nationa Natura Science Foundation of China under the grants NSFC and NSFC , and the Microsoft Laboratory for Inteigent Computing and Inteigent Systems of Shanghai Jiao Tong University. H. C. Liang and B. L. Lu are with the Department of Computer Science and Engineering, Shanghai Jiao Tong University, 800 Dong Chuan Rd., Shanghai , P. R. China. E-mai: {ianhc; bu}@sjtu.edu.cn The work of Goomb and coeagues [1] may be one of the most eary papers for gender cassification. They trained a fuy connected two-ayer neura network to identify gender from face images. Their experiments on a set of 90 photos gave an average error rate of 8.1 percent. Cottre [2] aso appied neura networks to emotion and gender cassification. Yen et a. [3] compared different neura network architectures incuding back-propagation neura networks and radia basis function (RBF) networks for deaing with the gender cassification probem. Vaentin et a. [4] investigated the gender cassification probem based on eigenfaces and neura networks. They found that both the Caucasian and the Japanese autoassociative memories perform better with own-race faces than with other-race faces. Gutta et a. [5] proposed a hybrid cassifier based on radia basis function neura networks and inductive decision trees with Quinan s C4.5 agorithm. The best average error rate was found to be 4 percent on 3,000 facia images from the FERET face database. Moghaddam and Yang [6] investigated support vector machines (SVMs) [13] for gender cassification with 1755 facia images from the FERET face database. Hayashi [9] researched about gender cassification based on wrinke texture and coor of facia images by using a specia Hough transform. Iga et a. [10] deveoped a system which discriminates mae and femae using SVMs. The reported correct rate was 93.1% with 101 persons [10]. More recenty, the AAM mode was aso introduced into the area of gender cassification [7], [8]. However this kind of method is quite compex and has not been widey expoited yet. Despite the fact that many researchers have been invoved in deveoping gender cassification systems in the past severa years, their discussions were usuay based on either sma face databases or fronta faces. In rea-word appications, however, a gender cassification system shoud hande a arge amount of facia variations due to head pose, ighting condition, persona age, coor of skin, etc. For exampe, at the entrance of an Oympic game, a robust gender cassification system is certainy required to work we on various conditions. At such situations, a arge-scae training data set that covers most possibe conditions is needed in order to make the trained gender cassifier have satisfactory generaization performance. On the other hand, with increasing the number of training data, to train very ong time shoud be However, the increment of training data set wi ead the earning of cassifiers to be very sow [14] or hard to convergence [19]. One of the chaenges is how to train gender cassifiers more efficienty

2 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 2 and effectivey. Another is how to incorporate some expicit domain or prior knowedge such as pose, iumination, age and ethnicity information to improve the correct cassification rate. Actuay, many methods of incorporating prior knowedge for pattern cassification and machine earning have been proposed [31], [32], [33], [34]. Basig [31] showed that the use of prior knowedge in the probem domain can consideraby support the network in finding the reevant structures inherent in the training data and can thus improve the generaization performance. Thompson and Kramer [32] presented a method to combine prior knowedge and artificia neura networks into a hybrid mode. This hybrid mode provides more accurate predictions, which are consistent with the process constraints, and more reiabe extrapoation. Wang and coeagues [33] proposed to incorporate some prior knowedge into SVMs to improve their performance for image retrieva. Their experimenta resuts demonstrated that the proposed method can effectivey improve the earning and retrieva performance of SVMs. Schapire and coeagues [34] presented an extended AdaBoost agorithm that permits incorporation of prior knowedge, and appied this boosting methods for ca cassification in spoken anguage understanding. Their experimenta resuts showed that prior knowedge can substantiay improve cassification performance. In our previous work [17], [18], [20], we have proposed a part-versus-part decomposition method for massivey parae training of muti-cass network and deveoped a min-max moduar (M 3 ) neura network for soving arge-scae pattern cassification probems. Hereto we have successfuy appied min-max moduar muti-ayer perception (M 3 -MLP) and minmax moduar support vector machine (M 3 -SVM) to various pattern recognition probems such as arge-scae text categorization [23], muti-view face recognition [24], prediction of subceuar muti-ocations of protein sequence [25], mutiabe and imbaance probems [26], gender cassification [21], [27] and age estimation [22]. In this paper, we propose how to incorporate prior knowedge into the earning of M 3 -cassifier to improve the performance. We appy this technique to gender cassification which cassify facia images incuding not ony fronta images but aso facia images with different poses. The CAS-PEAL face database [28], which is a arge-scae database that contains 14,384 pose facia images, is investigated to expain the priorities of incorporating prior knowedge into M 3 -cassifier. Three different feature extraction methods, et the gray pixe method, the we-known Gabor waveet method and the stateof-the-art oca binary patterns (LBP) method, are empoyed to argey guarantee the generaity of comparison. II. FEATURE EXTRACTION Feature extraction is one of the most important steps for gender cassification. As for the feature extraction, most existing work used the gray pixe method or Gabor waveet method [1], [2], [3], [5], [6], [9], [10]. Many research [11], [12], [22], [41], [42] showed that the Gabor waveet method yieded better performance for face recognition than other commony used representation methods such as principa components anaysis (PCA) and gray pixe method. Recenty a powerfu way of texture description caed oca binary pattern (LBP) has been proposed for texture cassification [37], face detection [38], and face recognition [39], [40]. Various work [38], [39], [40], [16] showed that LBP image representation yieded better performance for face recognition than other representation methods such as gray pixe, PCA, eastic bunch graph matching (EBGM), and Gabor waveet method. In order to systematicay compare our proposed method with existing approaches, three typica feature extraction methods, namey gray pixe method, Gabor waveet method and oca binary pattern method, are used in this paper. A. Gray Pixe The gray pixe faces are preprocessed with an automatic face processing system which compensates for transation, scae as we as rotation. This processing system uses geometric normaization for shape aignment and histogram normaization for ambient ighting variations. The resuting output faces in the top ine of Fig. 1 are standardized to resoution. These faces are further sub-samped to pixes for being fed into the gender cassifier. B. Gabor Waveet Fiter The Gabor waveets, which capture the properties of spatia ocaization, orientation seectivity, spatia frequency seectivity, and quadrature phase reationship, seem to be a good approximation to the fiter response profies encountered experimentay in cortica neurons [11], [12], [22], [41], [42]. The Gabor waveets have been found to be particuary suitabe for image decomposition and representation when the goa is the derivation of oca and discriminating features. Most recenty, Liu et a. [11], [12] have experimentay shown that the Gabor fiter representation gave better performance for face recognition. Here, we briefy introduce Gabor waveets, describe the Gabor feature representation of images, and derive a Gabor feature vector for gender cassification. Gabor fiter can be expressed as foows: ψ µ,ν (z) = k µ,ν 2 σ 2 e kµ,ν [ 2 z 2 2σ 2 e ikµ,νz e σ2 2 where µ and ν define the orientation and the scae of the Gabor kernes, respectivey, z = (x, y), denotes the norm operator, and the wave vector k µ,ν is defined as foows: ] (1) k µ,ν = k ν e iφµ, (2) where k ν = k max /f µ, φ µ = πµ/8, k max is the maximum frequency, and f is the spacing factor between kernes in the frequency domain. The Gabor waveet representation of an image is the convoution of the image with a famiy of Gabor kernes as defined by (1). In this paper, the Gabor waveets of five different scaes, ν {0,..., 4} and eight orientations, µ {0,..., 7} are adopted. The Gabor coefficients of the magnitudes are fitted together to form a vector for the gender cassification [11], [12], [22], [41], [42].

3 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 3 Fig. 1. Iustration of feature extraction by using LBP method. C. Loca Binary Patterns The LBP operator, which is introduced by Ojaa et a. [37], is a powerfu means of texture description. The operator abes the pixes of an image by threshoding the 3 3-neighbourhood of each pixe with the center vaue and considering the resut as a binary number. An extension to the LBP operator is to use so caed uniform pattern [37]. A oca binary pattern is caed uniform if it contains at most two bitwise transitions from 0 to 1 or vice versa when the binary string is considered circuar. For exampe, , , and are uniform patterns. In this paper, we use uniform LBP operator in a neighborhood of eight samping points on a circe of radius one. The number of abes of this LBP is 59 [16] The procedure of LBP method is iustrated in Fig. 1. The origina image is preprocessed by ocating eye positions, geometric normaization, cropping and histogram normaization and a so caed LBP face is obtained by performing uniform LBP operator on this preprocessed facia image. P P equa size bocks are divided from the LBP face with a grid on it and their histograms fitted together form an input for the gender cassifier [16]. A histogram of the abeed image f(x, y) can be defined as H i = I[f(x, y) = i], i = 0, 1,..., n 1, x,y where n is the number of different abes produced by the uniform LBP operator and { 1, A is true I[A] = 0, A is fase. This histogram contains information about the distribution of the oca micro-patterns such as edges, spots and fat areas over the whoe facia image. For efficient face representation, one shoud retain spatia information. For this purpose, the image is divided into a series of regions R 0, R 1,..., R m 1 (see Fig. 1) and the spatiay enhanced histogram is defined as H i,j = x,y I [f(x, y) = i] I [(x, y) R j ], (3) where i = 0, 1,..., n 1 and j = 0, 1,..., m 1. III. MIN-MAX MODULAR CLASSIFIER The min-max moduar neura network was firsty introduced in paper [17]. In this framework, a K-cass cassification probem is decomposed into K(K 1)/2 two-cass probems. These two-cass probems are to discriminate cass C i from cass C j for i = 1,..., K and j = i + 1, whie the existence of the training data beonging to the other K 2 casses is ignored. If the two-cass probem of discriminating cass C i from cass C j is sti hard to be earned, we can further break down it into a set of two-cass subprobems as sma as we expect. Since a of the two-cass subprobems are independent each other, they can be earned in a parae way. Consequenty, a arge-scae and compex K-cass cassification probem can be soved effortessy and efficienty by earning a series of smaer and simper two-cass subprobems in parae. A. Task Decomposition Let X 1 and X 2 be the training input sets of a two-cass cassification probem and the abes of two casses are represented by C 1 and C 2, respectivey. The training input sets are expressed as { X i = X (i) } Li =1 for i = 1, 2 (4) where L i is the number of training inputs in cass C i, X (i) is the th training input beonging to cass C i. The training data set for the two-cass cassification probem can be expressed as { } T = (X (1) L1 { }, +1) (X (2) L2, 1) (5) =1 =1 This two-cass probem may be too arge and too compex to be earned by a singe cassifier. We have suggested that T defined by (5) can be further decomposed into a number of two-cass subprobems as sma as needed according to the cass reations among training data [17], [18]. Assume the training input set X i defined by (4) is partitioned into N i (1 N i L i ) subsets in the form of { X i,j = X (ij) } L (j) i =1 for j = 1,..., N i, i = 1, 2 (6)

4 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 4 a seria or a parae way, because the modue combination procedure is competey independent of both the structure of individua trained sub-cassifiers and their performance. The other is that amost a the existing pattern cassification techniques such as k-nearest neighbor agorithm, decision trees, neura networks, and SVMs can be directy used as subcassifiers for a M 3 -cassifier. In this paper, we choose mutiayer perceptions (MLPs) and SVMs as sub-cassifiers for M 3 -cassifier. We ca them as min-max moduar muti-ayer perceptrons (M 3 -MLPs) [17] and min-max moduar support vector machines (M 3 -SVMs), respectivey [17], [20], [21]. Fig. 2. Structure of the min-max moduar cassifier for a two-cass probem. where L (j) i is the number of training inputs incuded in X ij, and Ni j=1 X ij = X i. According to (6), the training set for each of the smaer and simper two-cass subprobems can be given by { } (u) L { } (v) T (u,v) = (X (1u) 1 L, +1) (X (2v) 2, 1) (7) =1 =1 for u = 1,..., N 1, v = 1,..., N 2 where X (1u) X 1,u and X (2v) X 2,v are the input vectors beonging to cass C 1 and cass C 2, respectivey. Here N1 u=1 L(u) 1 = L 1, and N 2 v=1 L(v) 2 = L 2. Foowing (6) and (7), a compex two-cass probem can be decomposed into N = N 1 N 2 smaer and simper twocass subprobems. In the earning phase, these subprobems are independent each other and therefore they can be earned in a parae way. B. Modue Combination After training a the individua two-cass subprobems defined by (7), the trained N 1 N 2 sub-cassifiers are integrated into a moduar cassifier caed M 3 -cassifier with N 1 MIN integrating units and one MAX integrating unit according to the minimization principe and the maximization principe [17], [18], [20] as foows, h(x) = max N1 u=1 [ ] N2 min v=1 (hu,v 1,2 (x)) where h u,v 1,2 (x) denotes the transfer function of the trained cassifier M u,v 1,2 corresponding to the two-cass subprobem T (u,v) i,j defined by (7). Fig. 2 iustrates the structure of a M 3 -cassifier for a twocass probem. Here the two-cass probem is decomposed into N 1 N 2 sub-probems. Therefore the M 3 -cassifier consists of N 1 N 2 individua sub-cassifiers, N 1 MIN units, and one MAX unit [18]. The M 3 -cassifier has the foowing two attractive features for deaing with rea-word compex pattern cassification probems. One is that a the sub-cassifiers can be trained in (8) C. Expansibiity When appying neura network soutions to rea-word probems, we usuay face an important chaenge: How does one efficienty add new sampes of training data to a previousy trained network? To date, most existing neura network modes, such as MLPs [?], cascade-correation networks [?], and hierarchica mixtures of expert networks [?], have poor expansibiity in the sense that amost a the parameters of these trained networks must be readjusted, even in cases in which ony a few new sampes of training data are added. In this context then, expansibiity refers to the abiity of a neura network to expand efficienty after earning [?]. When a trained network requires expansion, a conventiona approach is to retrain the network on the revised training data set. Because a the parameters of trained networks must be destroyed in the retraining process, this retraining method is inefficient, especiay in cases in which the trained networks were achieved at a great cost in earning time. In this subsection we focus on the expansibiity of our proposed method based on two-cass probem. Suppose that a given training data set T defined by (5) were successfuy earned by a M 3 -network, namey M 3 T. When some new requirements are to be added to this network, a set of new sampes of training data S shoud be added to M 3 T. Let X 1 and X 2 be the new training input sets of the two-cass cassification probem defined by (5) and the abes of two casses are represented by C 1 and C 2, respectivey. The new training input sets are expressed as { } Ji X i = X(i) for i = 1, 2 (9) =1 where J i is the number of new training inputs in cass C i, X (i) is the th new training input beonging to cass C i. The new training data set for the two-cass cassification probem can be expressed as { } J1 { } J2 (1) (2) S = ( X, +1) ( X, 1) (10) =1 =1 Assume the training input set X i defined by (9) is partitioned into N i (1 N i J i ) subsets in the form of { } (j) J X i,j = X(ij) i for j = 1,..., N i, i = 1, 2 (11) =1 where J (j) i is the number of new training inputs incuded in X ij, and Ni j=1 X ij = X i.

5 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 5 Fig. 3. Structure of the min-max moduar network M 3 T probem with 2 2 modues. for a two-cass We now discuss the probem of how to add S to the previousy trained network M 3 T. According to the task decomposition and modue combination procedures mentioned earier, the probem of adding S to M 3 T can be impemented by training new network modues on the foowing two-cass subprobems and adding these trained network modues to M 3 T. The new two-cass subprobems that need to be further earned are S (u,v), T S (u,v) and ST (u,v) given by S (u,v) = T S (u,v) = ST (u,v) = { } (u) J { } (v) (1u) 1 J ( X, +1) (2v) 2 ( X, 1) =1 =1 for u = 1,..., N 1, v = 1,..., N 2 { } (u) L { } (v) (X (1u) 1 J, +1) (2v) 2 ( X, 1) =1 =1 for u = 1,..., N 1, v = 1,..., N 2 { } (u) J { (1u) 1 ( X, +1) =1 (X (2v), 1) } L (v) 2 =1 for u = 1,..., N 1, v = 1,..., N 2 (12) (13) (14) According to (12), (13), and (14), the number of new twocass subprobems (different from those in (7), is given by N = N 1 N 2 + N 1 N 2 + N 1 N 2 (15) To iustrate the expansibiity of the proposed method, we present a simpe exampe. Suppose that a M 3 network, namey M 3 T as shown in Fig. 3, has been successfuy trained on a two-cass probem with N = 2 2 modues. To add new training subsets to M 3 T, suppose one new training subset for both casses, five network modues, M 1,3 1,2, M2,3 1,2, M3,1 1,2, M3,2 1,2, and M 3,3 1,2 shoud be trained on five new two-cass subprobems, T S 1,3, T S 2,3, ST 3,1, ST 3,2, and S 3,3 respectivey. After training these network modues, the previousy trained network modues in M 2 T were integrated into a new M3 network, namey M 3 T S, as shown in Fig. 4. Comparing M3 T with M3 T S, we see that a the previousy trained network modues in M 3 T, i.e., M 1,1 1,2, M1,2 1,2, M1,2 2,1, and M2,2 1,2, were reused in M3 T S in their origina condition. Since the MIN and MAX units do not require training, and the number of fan-in of the MIN and MAX units are increased easiy in both software and hardware, adding new sampes of training data to the trained M 3 networks can be achieved efficienty. Fig. 4. Structure of the min-max moduar network M 3 T S for a two-cass probem with 3 3 modues. The modues M 1,1 1,2, M1,2 1,2, M1,2 2,1, and M2,2 1,2 are previousy trained modues from M 3 T shown in Fig. 3, whie the M1,3 1,2, M2,3 1,2, M 3,1 1,2, M3,2 1,2, and M3,3 1,2 are new expanded modues. IV. INCORPORATING PRIOR KNOWLEDGE INTO LEARNING When everything fais, ask for additiona domain knowedge is the current motto of machine earning. Therefore, making maxima use of the rea-word prior or domain knowedge has its both deep and practica meanings. In this section, we firsty interpret the importance of prior knowedge to machine earning with no free unch theorem and a meaningfu dimensionaity reduction. Then we present how to incorporate expicit prior knowedge into M 3 -cassifiers. As an iustrative exampe, we demonstrate how to incorporate the pose information of facia images into M 3 -cassifiers for gender cassification. A. No Free Lunch Theorem The No Free Lunch Theorem (NFL) [29], [30] indicates that if there are no context information or intrinsic information invoved in cassification probems, we have no reasons to favor one cassification method over another. This means that if one agorithm seems to outperform another in a particuar situation, it is usuay a consequence of its fit to the particuar cassification probem, but not the genera superiority of the agorithm. Since the incorporating of intrinsic prior knowedge heps to fit a cassifier to the particuar pattern cassification probem, the cassifier with prior knowedge is going to obtain better generaization performance than the origina one. Therefore, when a new cassification probem is confronted, what we consider are what kinds of prior knowedge can be usefu, and how to incorporating intrinsic prior knowedge that we have obtained to improve performances. As mentioned before, gender cassification systems usuay encounter difficuties in handing arge amounts of facia variations due to head pose, ighting conditions, persona age

6 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 6 and ethnicity difference. Actuay, these kinds of information are very usefu from the viewpoint of the NFL theorem. The theorem inspires us to propery utiize such kinds of prior knowedge to improve the system s performance. Therefore, how to incorporate usefu prior knowedge into the earning of M 3 -cassifier becomes an important question we want to answer. The No Free Lunch (NFL) Theorem [29], [30] indicates that if the goa is to obtain good generaization performance, there are no context-independent or usage-independent reasons to favor one earning or cassification method over another. This means that if one agorithm seems to outperform another in a particuar situation, it is a consequence of its fit to the particuar earning or pattern cassification probem, not the genera superiority of the agorithm. Therefore, when a new earning or pattern cassification probem is confronted, appreciation of this theorem reminds us to focus more on the most-prior information that we can obtain. From the viewpoint of knowedge acquisition, prior knowedge can be cassified into two forms: expicit prior knowedge and impicit prior knowedge. For most earning or pattern cassification probems, we can easiy obtain various expicit prior knowedge from training data, especiay for image cassification probems such as gender cassification based on facia images. For exampe, the detai pose information of each facia image is given in both the FERET and the CAS-PEAL face databases. The NFL theorem tes us that the more prior knowedge is utiized in the earning, the better generaization performance of a cassifier can be achieved. Therefore, how to fuy incorporate various prior knowedge into earning is a vita important issue in both machine earning and pattern cassification societies. B. Incorporating Prior Knowedge into Task Decomposition Various kinds of prior knowedge can be found in reaword appications. For exampe, Schapire et a. [34] manuay drew out twenty kinds of prior knowedge for text categorization. Soberg and Espen [35] proposed to incorporate prior knowedge about the externa conditions ike wind eve and sick surroundings for automatic detection of oi spis in SAR images. Papathanassiou and Petrou [36] proposed to incorporate prior knowedge about the movement of the eyes, the breathing process and the heart for extracting MEG signas. To deveop a practica gender cassification system by using machine earning techniques and guarantee satisfactory generaization performance, various facia images shoud be coected as training inputs. These facia images shoud come from different head poses, different ighting conditions, different persona ages, and different races, et a. Consequenty, a gender cassification probem usuay is a arge-scae, compex two-cass pattern cassification probem. On the other hand, various expicit prior knowedge can be easiy obtained from the facia images. For exampe, the exact pose information for each facia image is given in both the FERET and CAS-PEAL face databases. The probem that we consider in this study is to answer the foowing two questions: a) how to incorporate the given prior knowedge into the M 3 -cassifier for gender cassification? and b) can the generaization performance of the M 3 -cassifier be improved by incorporating the prior knowedge into earning? Suppose that a set of expicit prior knowedge associated to the training input set X i has been obtained before earning, and can be expressed as foows: B i = {P i,1, P i,2,..., P i,ni } for i = 1, 2 (16) where P i,j means the jth prior knowedge incuded in the training input set X i. According to the part-versus-part task decomposition strategy mentioned before, a compex two-cass pattern cassification probem can be divided into a series of two-cass subprobems as smaer as needed. In our previous work, we have proposed severa task decomposition strategies such as random strategy, hyper-pane strategy, and equa custering strategy. It is these task decomposition strategies provide us with hepfu hints for introducing a new task decomposition approach based on prior knowedge. We ca this new task decomposition method prior knowedge strategy. By using the given prior knowedge set B i, the prior knowedge strategy can decompose the training sampe set X i into the foowing N i subsets: X i,j P i,j (X i ) for j = 1,..., N i (17) where means that the prior knowedge P i,j is utiized to partition training input set X i into subset X i,j. C. An Iustrative Exampe Suppose that the foowing two training input sets for a gender cassification probem are seected from the CAS- PEAL face database: and X 1 = {1800 mae facia images with 0, 15, and 30 poses, respectivey} (18) X 2 = {1800 femae facia images with 0, 15, and 30 poses, respectivey} (19) Aso suppose that the foowing expicit prior knowedge is obtained from the CAS-PEAL face database: P 1,1 = P 2,1 = {facia images beonging to 0 pose}, (20) P 1,2 = P 2,2 = {facia images beonging to 15 pose}, (21) and P 1,3 = P 2,3 = {facia images beonging to 30 pose} (22) From Eqs. (20) through (22), two sets of prior knowedge associated with the training input sets, X 1 and X 2, can be respectivey expressed as foows: and B 1 = {P 1,1, P 1,2, P 1,3 } (23) B 2 = {P 2,1, P 2,2, P 2,3 }, (24)

7 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 7 Fig. 5. Iustration of partitioning training input data sets for mae and femae into three subsets, respectivey, according to the given pose information. By using the proposed prior knowedge strategy, and B 1 and B 2, we can partition X 1 and X 2 into the foowing three subsets, respectivey and X 1,j P 1,j (X 1 ) for j = 1, 2, 3 (25) X 2,j P 2,j (X 2 ) for j = 1, 2, 3 (26) Fig. 5 iustrates the mae and femae facia images are respectivey divided into three subsets according to the head pose information. D. Reated Evidence Recenty, Tenenbaum et a. proposed a goba geometric framework for noninear dimensionaity reduction [46]. They found that the data s intrinsic geometric structure can be visuay reveaed when the reduced vectors were projected to 2D axis system. By using a compete isometric feature mapping technique, a canonica probem in dimensionaity reduction from the domain of visua perception is deepy investigated. The inputs consist of many images of a person s face observed under different poses and ighting conditions without any particuar order. Through a two-dimensiona projection of isometric feature mapping, images with information about pose and ighting direction are custered together. This phenomenon better discovers how the brain comes to represent the dynamic appearance of objects. However, the most important thing is that this research reminds us to make use of prior knowedge if the goa is to obtain good generaization performance. We have presented how to incorporate prior knowedge into the earning of M 3 -cassifier reated to pose information. Other kinds of prior knowedge such as iumination, age and ethnicity information can simiary be incorporated into the M 3 -cassifier, as ong as these sampes can be obtained. The consequent difficuty is that the size of train set become very arge. At this situation, the training of conventiona cassifiers, such as MLPs and SVMs, turn to be very sow [14] or hard to convergence [19]. In contrast to it, the part-vs-part structure of M 3 -cassifier guarantees its massivey parae earning on arge-scae data set [20]. And the incorporating of prior knowedge guarantees M 3 -cassifier to obtain higher generaization abiity than conventiona cassifiers as we as M 3 -cassifier with random partition strategy. MLPs and SVMs are both empoyed to investigate the performance of M 3 -cassifier when prior knowedge is incorporated into the earning. In this paper, PK-M 3 MLP and PK- M 3 SVM denote M 3 -MLP and M 3 -SVM that trained on the sampe sets partitioned by PK strategy. To ook into the difference between methods with incorporated prior knowedge and other conventiona methods, a random partition (RN) strategy [18] is aso simuated as baseine. Thereby, RN-M 3 MLP and RN-M 3 SVM denote M 3 -MLP and M 3 -SVM that trained on the sampe sets partitioned by RN strategy, respectivey. The comparisons are discussed in the next two sections. V. EXPERIMENT SETUP In order to evauate the effectiveness of the proposed method for deaing with gender cassification probems, the CAS- PEAL face database [28] and the FERET face database [48] are used to construct training data sets. Three typica feature extraction methods and four pattern cassification approaches are seected. A. Face Database TABLE I DESCRIPTION OF TRAINING AND TEST DATA SETS SELECTED FROM THE CAS-PEAL FACE DATABASE FOR GENDER CLASSIFICATION. Descrip. No. Femae No. Mae No. Training No. Test P00 1,335 1, ,920 P15 2,536 3, ,432 P30 2,536 3, ,432 Tota 6,407 7,977 3,600 10,784

8 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 8 The pubished CAS-PEAL database [28] is a arge-scae facia image database that currenty contains 30,864 facia images of 1,040 subjects. A the subjects in this database are orienta. We choose the images of three poses, namey 0 degree, 15 degree, and 30 degree, as the data sets for gender cassification in this paper. Totay, there are 14,384 facia images of 1,040 individuas. Tabe I shows the detais about the database, where Pnn denotes a of the pose information about the data sets. The character P represents pose variation. The nn indicates the azimuth of the camera from which the image is obtained. Taking the first row for exampe, it incudes 3,120 facia images from 1,040 individuas. Among them, 1,200 images of mae and femae are used as training inputs and the rest 1,920 images are used as test inputs. Thus, the tota number of training sampes is 3,600 and the tota number of test sampes is 10,784. The reasons why use this database are twofod. One is that it is the most arge-scae face database, which contains gender information and has been pubished. The other is that it contains poses information which can be incorporated into the training of M 3 -network convenienty. TABLE II DESCRIPTION OF DATA SETS SELECTED FROM THE FERET FACE DATABASE FOR GENDER CLASSIFICATION. Description Tota No. Femae No. Mae fa fb dupi dupii training Tota 3,427 1,226 2,201 The FERET face database [48] is a standard database for testing and evauating state-of-the-art face recognition agorithms. There are gaery and probe image sets that were used in the origina FERET test. These sets are: fa, fb, fc, dup I, dup II, and training sets. Each set contains at most one image per person. We choose the fu-fronta face images from these sets and seect the non-asian faces as gender data sets. Tabe II shows the detais about the chosen gender sampes. The tota gender sampes are 3,427, and tota femae sampes are 1,226 and tota mae sampes are 2,201. We randomy choose 600 femae and 600 mae faces among them for training set and the rest sampes are used as testing set. The reason why use these non-asian facia images is that they have different ethnic information when comparing to CAS-PEAL database. As a resut, the pose information among CAS-PEAL database and the ethnic information among PERET database provide much good materias to evauate the performances of M 3 -network convenienty. B. Parameters for Feature Extraction The parameters for three feature extraction methods are seected as foows: (1) A the grey facia images are preprocessed and normaized to pixes as described in section II. (2) The Gabor waveets of five different scaes and eight orientations are adopted. The Gabor vectors are down-samped to 6,912 dimensions. (3) The LBP method uses LBP u 8,1 for the feature extraction and the LBP faces are divided into P P equay sized bocks. The histograms of the bocks are extracted and concatenated into a vector for cassification [16]. As a resut, the dimensions of LBP features with different bocks are P P 59 [16]. The LBP methods with different P are denoted as LBP P. We scae a the training data to be in [-1, 1]. Then test data are adjusted to [-1,1] accordingy. C. Pattern Cassification Methods In the simuations presented beow, a the network modues in the M 3 -MLP networks are chosen to be MLPs with one hidden ayer. To compare the performance of the proposed PK- M 3 MLP method with non-prior knowedge based methods, the origina gender probem is aso earned by RN-M 3 MLP and singe MLPs. A the MLPs used in both the proposed method and the existing approaches are trained using the same back-propagation agorithm [44]. The momentums of the backpropagation agorithm are a set to 0.9 and the earning rate is set to 0.01, 0.02 or The number of hidden units is 1,500 for singe MLPs and 500 for a modues of M 3 -MLP. In the process of training the network modues for the M 3 -cassifiers, earning is stopped when the sum of squared error between the desired and actua outputs is ess than 0.1 or when the number of epochs reached 500. A of the simuations are performed on a 2.8 GHz P4 PC with MATLAB circumstance. On the other hand, a the network modues in the M 3 -SVM networks are chosen to be SVMs with the same kerne. To compare the performance of the proposed PK-M 3 SVM method with non-prior knowedge based methods, the origina gender probems are aso earned by RN-M 3 SVM and conventiona SVMs. A the SVMs are carried out using C-coded SVM packages of LIBSVM [43] running in the same PC. The parameter C is tuned and set as 1,000 and the Gamma vaue is set as the reciproca of sampe dimension when simuating on the inear, poynomia and Radia Basis Function (RBF) kernes respectivey. In addition, different C vaues and Gamma vaues are aso simuated on RBF kerne by a way of fixing one parameter whie varying another parameter, to widey investigate the performance on various conditions. D. Prior Knowedge versus Non-prior Knowedge To incorporate the pose information into the earning of M 3 - cassifier, the training sets shown in Tabe I are partitioned into six subsets according to formuas (25) and (26) given in section IV. These subsets are fed into the PK-M 3 MLP or PK- M 3 SVM for their moduar earning. Through this way, prior knowedge is incorporated into the earning of M 3 -cassifier. The RN strategy aso partitions training sets into six subsets for the earning of RN-M 3 MLP and RN-M 3 SVM. However, it does not invove any prior knowedge because it partitions training sets into subsets randomy. In addition, the whoe training sets are used into training the conventiona MLPs and SVMs. E. Expansibiity versus Non-expansibiity To show the expansibiity of M 3 -cassifier, four phases of earning are simuated as foows. 1) Phase 1: Submit the 0

9 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 9 TABLE III DESCRIPTION OF THREE LEANING PHASES FOR EVALUATING THE Phase 1 Phase 2 Phase 3 Phase 4 EXPANSIBILITY. Description Submit the 0 degree faces of mae and femae Phase 1 has been earned and now the 15 degree faces are added to earning system Phase 1 and Phase 2 have been earned and now the 30 degree faces are added to earning system Phase 1, Phase 2, and Phase 3 have been earned and now FERVT faces are added to earning system degree faces of mae and femae in Tabe I to the earning system, 2) Phase 2: Suppose Phase 1 has been earned and now the 15 degree faces in Tabe I are added to the earning system, 3) Phase 3: Suppose Phase 1 and Phase 2 have been earned and now the 30 degree faces in Tabe I are added to the earning system, 4) Phase 4: Suppose Phase 1, Phase 2, and Phase 3 have been earned and now the FERET faces in Tabe II are added to the earning system. The training data sets added at Phase 1, Phase 2, and Phase 3 are 600 femae and 600 mae faces from Tabe I, respectivey. For consistency, at Phase 4 we randomy choose 600 femae and 600 mae faces from FERET database shown in Tabe II. These 1,200 sampes are added into earning system at Phase 4, and the remainder are used as testing sampes. Notice that, at the former three phases, the different kinds of pose information are incorporated into the earning. Whie at the Phase 4, the ethnic information is incorporated into the earning. Tabe III shows the detais. TABLE IV DESCRIPTION OF TYPE I AND TYPE II METHODS FOR EVALUATING THE EXPANSIBILITY. Type I Type II No. Training No. Testing No. Training No. Testing Phase , ,011 Phase 2 1, ,352 1, ,011 Phase 3 1, ,784 1, ,011 Phase 4 2, ,011 2, ,011 We evauate the expansibiity of M 3 -network through two types as shown in Tabe IV. Type I and Type II have the training procedures. These two training procedures have the same training phases as shown in Tabe III. However, the testing procedures of Type I and Type II are different from each other. The testing procedure of Type I use the remained sampes of each phase, whie the testing procedure of Type II use the tota remained sampes of the four phases. The purposes of the two types of evauation are different from each other. Type I aims to evauate the performances of cassifiers at such a case: how can the expansion of sampes or say network modues improve the cassification accuracy when cassifiers are expected to facing a new pre-known mission. For exampe, at Phase 4 the new mission is to cassify non-asian faces. Differenty, Type II aims to evauate the performances of cassifiers at such a case: how can the expansion of sampes or say network modues improve the cassification accuracy to approaching the requirement of rea-word appications. For exampe, we suppose that the requirement of a rea-word gender system is to dea with 0, 15, and 30 degrees faces of Asian and fronta faces non-asian. Actuay, there are aso other kinds of requirement can be met in practice, such as ighting condition, persona age, etc. Simiar tasks can be soved by the same way of moduar expansion of M 3 -network. The proposed PK-M 3 SVM is compared with traditiona cassifier SVMs during a phases of Type I and Type II. At Phase 1, PK-M 3 SVM has not difference with SVMs since PK- M 3 SVM ony has one modue, which is the same structure of conventiona SVMs. However, at Phase 2, PK-M 3 SVM ony need to integrate the modues that have been trained at Phase 1 with the new modues trained on the added training sets, i.e. 15 degree faces. Foowing the same way, at Phase 3, PK-M 3 SVM ony need to integrate the modues that have been trained at Phase 2 with the new modues trained on the added training sets, i.e. 30 degree faces. Lasty, at Phase 4, PK-M 3 SVM ony need to integrate the modues that have been trained at Phase 3 with the new modues trained on the added training sets, i.e. FERET faces. Differenty, conventiona SVMs have to retrain a of training sampes at these four phases. The proposed PK-M 3 SVM are compared with conventiona SVMs on LBP 8 feature extraction method with RBF kerne. Here we ony take LBP 8 feature extraction method as an iustration. Actuay, simiar comparison resuts can be obtained from other feature extraction methods. The parameters C and Gamma are set as the same to those of section V- C. The correct rate, precision (P), reca (R) and F-measures [47] are recorded during the different earning phases of PK- M 3 SVM and SVMs to demonstrate the expansibiity of M 3 - cassifier more detaied. Simutaneousy, the modue number, SV number, training time, and testing time are a compared between conventiona SVMs and PK-M 3 SVM. VI. EXPERIMENTAL COMPARISONS The comparisons of PK-M 3 MLP with RN-M 3 MLP and singe MLPs, and the comparisons of PK-M 3 SVM with RN- M 3 SVM and singe SVMs are both performed in this section. Their experimenta resuts are aso anayzed and discussed in this section. A. Comparisons on MLP Tabe V presents the resuts of singe MLPs, RN-M 3 MLP and PK-M 3 MLP simuated on the database. Three kinds of feature extraction methods gray pixe method, Gabor waveet method and LBP methods are a proposed for comparison. The number of bocks, P, in the LBP methods ranges from 5 to 10. It can be seen that the correct rates of PK-M 3 MLP are higher than singe MLPs as we as RN-M 3 MLP s on a the feature extraction methods. For exampe, on LBP 8 method, singe MLPs can ony obtain a correct rate 86.90% and RN-M 3 MLP can occupy a correct rate 88.68%, whie at the same situation, PK-M 3 MLP can obtain a highest rate 91.11%. This indicates that a randomy partition strategy can ony sighty improve the cassifying abiity of M 3 -cassifier. In contrast to it, the PK partition strategy, which tends to incorporate prior knowedge into the earning of PK-M 3 MLP, can argey improve the accuracy of gender cassification.

10 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 10 TABLE V A COMPARISON OF CORRECT RATE (%) USING DIFFERENT METHODS ON CAS-PEAL DATABASE FOR GENDER CLASSIFICATION. HERE THE MODULAR CLASSIFIER IS MLPS AND LR MEANS LEARNING RATE. LR Method Gray-pixe Gabor LBP 5 LBP 6 LBP 7 LBP 8 LBP 9 LBP 10 Singe MLPs RN-M 3 MLP PK-M 3 MLP Singe MLPs RN-M 3 MLP PK-M 3 MLP Singe MLPs RN-M 3 MLP PK-M 3 MLP Simiar concusions can aso be drawn from the experimenta resuts of gray method, Gabor waveet method and other LBP methods, which show that the correct rates of PK-M 3 MLP are higher than singe MLPs as we as RN-M 3 MLP s. The reason why PK-M 3 MLP can be better than singe MLPs as we as RN-M 3 MLP is the incorporation of pose information to partition the mae and femae sets into three subsets with 0, 15 and 30, respectivey. These subsets are decomposed into nine subprobems according to the partversus-part strategy of M 3 -cassifier to be earned. Since a the subprobems become simper than the origina one and more proper than subprobems obtained from random partition strategy, they can be earned more efficienty and deiberatey by moduar MLPs of PK-M 3 MLP. So the combination of a subprobems through PK-M 3 MLP is guaranteed to be better than singe MLPs as we as RN-M 3 MLP s. B. Comparisons on SVM Tabe VI presents the resuts of singe SVMs, RN-M 3 SVM and PK-M 3 SVM simuated on the same database and feature extraction methods. The inear, poynomia and RBF kernes are investigated respectivey. It can be seen that the correct rates of PK-M 3 SVM are higher than both singe SVMs and RN-M 3 SVM s on most of the simuations. For exampe on LBP 10 method, when the kerne of SVM is RBF, SVMs can ony obtain a correct rate 92.82% and RN-M 3 SVM can occupy a correct rate 92.78%, whie at the same situation, PK-M 3 SVM can obtain a highest rate 93.43%. Simiar resuts can aso be seen from the other experiments based on gray pixe method and Gabor waveet method, which show that the correct rates of PK-M 3 SVM are higher than correct rates of singe SVMs as we as RN-M 3 SVM s. These comparisons indicate that the incorporation of prior knowedge can substantiay hep to improve the generaization abiity of M 3 -cassifier. We aso estimate the generaization abiity of singe SVMs, RN-M 3 SVM and PK-M 3 SVM using various C vaues and Gamma vaues. Tabe VII presents the resuts of three methods on LBP 8 with RBF kerne and different C vaues. Tabe VIII presents the resuts of three methods on the same conditions with different Gamma vaues. From these two tabes, we can see that, despite under various C vaues and Gamma vaues, PK-M 3 SVM sti occupies the best generaization performance when comparing with singe SVMs and RN-M 3 SVM. TABLE VII A COMPARISON OF CORRECT RATE (%) USING DIFFERENT C VALUES ON 1 LBP 8 WHERE SVM S KERNEL IS RBF AND GAMMA= = C Singe SVMs RN-M 3 SVM PK-M 3 SVM TABLE VIII A COMPARISON OF CORRECT RATE (%) USING DIFFERENT GAMMA VALUES ON LBP 8 WHERE SVM S KERNEL IS RBF AND C=1000. Gamma Singe SVMs RN-M 3 SVM PK-M 3 SVM The reason why PK-M 3 SVM can be better than singe SVMs as we as RN-M 3 SVM is same to the case of MLPs. That is, the PK partition strategy divide training set into three subsets according the same pose information. These subsets are decomposed into nine subprobems according to the partversus-part strategy of M 3 -cassifier. Since a the subprobems become simper than the origina one and more proper than subprobems obtained from random partition strategy, they can be earned more efficienty and deiberatey by each moduar SVMs of PK-M 3 SVM. So the combination of a subprobems through PK-M 3 SVM is guaranteed to be better than singe SVMs as we as RN-M 3 SVM s. Furthermore, it can be seen that Tabe V and Tabe VI are consistent that the methods incorporated prior knowedge perform better than those without any prior knowedge. This consistency aso indicates that the incorporation of prior knowedge can substantiay improve the generaization abiity of M 3 -cassifier. C. Comparisons on Running Time Tabe IX presents a comparison of training time, test time and number of hidden units or support vectors on LBP 8 method. Notice that the resuts of M 3 -cassifier are counted by two ways. One is a sequentia way that tota vaues of a modues are counted, and the other is a parae way that

11 SUBMITTED TO IEEE TRANSACTIONS ON NEURAL NETWORKS 11 TABLE VI A COMPARISON OF CORRECT RATE (%) USING DIFFERENT METHODS ON CAS-PEAL DATABASE FOR GENDER CLASSIFICATION. HERE THE MODULAR CLASSIFIER IS SVMS. Kerne Method Gray-pixe Gabor LBP 5 LBP 6 LBP 7 LBP 8 LBP 9 LBP 10 Singe SVMs RBF RN-M 3 SVM PK-M 3 SVM Singe SVMs Linear RN-M 3 SVM PK-M 3 SVM Singe SVMs Poy RN-M 3 SVM PK-M 3 SVM TABLE IX A COMPARISON OF TRAINING TIME, TESTING TIME AND NUMBER OF HIDDEN UNITS OR SUPPORT VECTORS ON LBP 8 METHOD WITH MLPS AND SVMS. NOTICE THAT MLPS RUN UNDER MATLAB WITH 0.01 LEARNING RATE, WHILE SVMS RUN UNDER C++ CIRCUMSTANCE WITH RBF KERNEL. Method Hidden unit Training time Testing time or SV number (Sec.) (ms/per) Max Tota Max Tota Max Tota MLPs 1,500 1,500 3, , RN-M 3 MLP 500 4, , PK-M 3 MLP 500 4, , SVMs 1,289 1, RN-M 3 SVM 707 3, PK-M 3 SVM 647 2, maxima vaues of a the independent modues are counted. It can be seen that, by parae way M 3 -cassifier can occupy much ess training or test time than by sequentia way. For exampe, PK-M 3 MLP needs ony seconds for training gender probem by a parae way, whie singe MLPs needs 3, seconds for training, which is much time consumed comparing with PK-M 3 MLP or RN-M 3 MLP. The simiar resuts can aso be seen from comparison among PK-M 3 SVM, RN-M 3 SVM and singe SVMs. The reason why the training time of M 3 -MLP or M 3 -SVM is much ess than that of singe MLPs or SVMs, is that the arge-scae probem has been decomposed into a series of smaer and independent subprobems. These independent subprobems can easiy be earned and trained by the independent modues of M 3 -MLP or M 3 -SVM. These independent modues aso can easiy be reaized in parae machines or a grid computing system. Actuay, parae M 3 -MLP and M 3 - SVM have been reaized on a Fujitsu VPP700E vector parae computer [19] and IBM p690 [23]. However they are beyond the discussion of this paper. D. Comparisons on Expansibiity Tabe X shows a comparison of Type I correct rate, precision, reca and F-measure of PK-M 3 SVM with SVMs on LBP 8 feature extraction method. It can be seen that at Phase 1, PK-M 3 SVM has the same performance with SVMs. This is because that at this phase, both casses of mae and femae are not decomposed into modues for earning of PK-M 3 SVM. At this time, PK-M 3 SVM does not have any difference with SVMs. However, when gender sampes of different poses or ethnicity are added to the earning systems, namey at Phase 2, Phase 3 and Phase 4, it can be seen that PK-M 3 SVM has better correct rate than SVMs, and aso has better precision, reca and F-measure than SVMs on both genders. The reason is that prior knowedge about pose and ethnicity are incorporated into the task decomposition of PK-M 3 SVM. Through the task decomposition, the compex gender cassification probem is decomposed into a series of independent modues. These independent modues can be earned efficienty and can be combined together by using minimization principe and maximization principe of M 3 -network. Therefore, generaization performance of M 3 -network can be higher than conventiona SVMs. From these four Phases, it can be seen that the correct rate of SVMs has drawn from 94.63% to 91.59%, whie PK-M 3 SVM has ony drawn from 94.63% to 93.67%. This resut demonstrates that, during network expansion or say adding new missions, PK-M 3 SVM can keep much better generaization performance than SVMs. Tabe XI shows a comparison of Type II correct rate, precision, reca and F-measure of PK-M 3 SVM with SVMs on the same feature extraction method with Type I. It can be seen that during the network expansion of PK-M 3 SVM, the correct rate of PK-M 3 SVM rises from Phase 1 s 87.83% to Phase 4 s 93.67%. Whie at the same situation, the correct rate of SVMs ony rises from 87.83% to 91.59%. The reason that PK-M 3 SVM can perform better than SVMs is that different kinds of prior knowedge are incorporated into the task decomposition of PK-M 3 SVM during the sampes increase or say modue expansion phases. The independent sma modues decomposed by using prior knowedge can be