A New Supervised Clustering Algorithm Based on Min-Max Modular Network with Gaussian-Zero-Crossing Functions

Transcription

1 2006 Internationa Joint Conference on Neura Networks Sheraton Vancouver Wa Centre Hote, Vancouver, BC, Canada Juy 16-21, 2006 A New Supervised Custering Agorithm Based on Min-Max Moduar Network with Gaussian-Zero-Crossing Functions Jing Li and Bao-Liang Lu Department of Computer Science and Engineering, Shanghai Jiao Tong University 800 Dong Chuan Rd., Shanghai , China Emai: {jingee, bu}@sjtu.edu.cn Abstract In this paper, we show that the shape, size and ocation of the receptive fied around each instance are different and decided by the distribution of training data in a min-max moduar network with Gaussian-zero-crossing functions. Based on this property, we propose a new supervised custering agorithm which has the foowing features: First, the incrementa custering abiity, which means the number of custers need not to be predefined, it can grow up automaticay, aso, the training data need not to be processed iterativey; Second, attaching more importance to border instances than non-border instances, which guarantees the good generaization performance and training data reduction ratio; Third, outier remova abiity, which removes noise instances from training data; Last, custer combination abiity, which reduces the number of custers further. Experiments on an artificia probem and severa reaword appications demonstrate these attractive features of our new custering agorithm. I. INTRODUCTION Custer anaysis aims to organize a coection of data items into custers, such that items within a custer are more simiar to each other than they are to items in the other custers [3]. If there is no information avaiabe concerning the membership of data items to predefined casses, the corresponding custer anaysis, such as the we-known k- means [16] or sef-organizing map (SOM) [5], is caed unsupervised custering. But in many cases a sma amount of knowedge is avaiabe concerning either pairwise constraints between data items or cass abes for some items. This information can be used to guide or adjust the custering process. The corresponding approach is caed semi-supervised custering [3], such as semi-supervised custering by seeding [1]. If the cass abes of a the data item are avaiabe, the custer anaysis beongs to supervised custering, such as eaning vector quantization (LVQ) [6] and correation custering [2]. Supervised custering may be very usefu as a preprocessing to reduce training data set for further cassification, since in rea-word appications the training data set are aways too arge for traditiona cassifiers. In many rea-word appications, training data often become avaiabe in sma and separate batches at different times. A custering agorithm must have the incrementa earning abiity to dea with this situation. But soving custering probems through optimization is NP-compete in most cases [14], many custering methods, such as k-means, SOM and LVQ, have to use heuristic agorithms. They need predefine the number of custers and process a the training data set iterativey, which make the incrementa earning abiity unattainabe. A straightforward approach of incrementay custering data points is to group a new instance into an existing custer that is cosest to it or make the new instance as a new custer. However, there exists the probem of how to decide in which circumstance the new instance shoud be grouped into an existing custer and in which circumstance it shoud be treat as a new custer. Li et a. have proposed a supervised custering agorithm caed Custering and Cassification Agorithm- Supervised (CCAS) [7], [8] to sove this probem. CCAS divide input space into some grid ces, ony instances in the same grid ce can be custered. But the probem of choosing the number of grid ces is sti unsoved. On the other hand, Lu et a. have proposed min-max moduar network with Gaussian-zero-crossing functions (M 3 - GZC) [9], [10] which has ocay tuned response characteristic and emergent incrementa earning abiity. In this paper, we anayze the properties of receptive fied around each instance in M 3 -GZC network. We discover that the shape, size and ocation of each receptive fied are determined by the distribution of training data. The receptive fied can be viewed as the grid ces in input space and can be used as a criterion to decide whether the new instance shoud be grouped into an existing custer or shoud be treat as a new custer. Aso, we find the receptive fieds around border instances are smaer than that around non-border instances in M 3 -GZC network, which means the corresponding custering agorithm wi generate more custers near the border and ess custers far from the border. More custers near the border guarantees that the distortion of border instances is smaer whie ess custers far from the border guarantees the higher reduction ratio of training data. We do not discard the non-border instances for further reduction because in the incrementa earning process the non-border instances may become border instances after more training data avaiabe, discarding them may ead to great and unwanted changes to the decision boundaries. Since not a the non-border instance wi become border instance in future, we pay ess attention to them than the instances that are aready border instances. The remainder of the paper is organized as foows. In Section II, M 3 -GZC network is introduced briefy. In Section III, we anayze the properties of receptive fied in M 3 -GZC network. Based on these properties, our supervised custering method is proposed in Section IV, post-processing incuding /06/$20.00/ 2006 IEEE 786

2 outier remova and custer combination are aso presented in this section. Experimenta resuts on an artificia probem and severa rea-word appications are isted in Section V. Finay, concusions are given in Section VI. II. MIN-MAX MODULAR NETWORK WITH GZC FUNCTION A. Min-Max Moduar Network Let T be the training set for a K-cass probem, T = {(X, D )} L =1 (1) where X R n is the input vector, D R K is the desired output, and L is the tota number of training data. According to the min-max moduar network [11], [12], a K-cass probem defined in equation (1) can be divided into K (K 1) two-cass probems that are trained independenty. The decomposition process is described as foowing. First we divide the input vectors into K subsets according to cass reations. { X i = X (i) } Li =1, for i = 1, 2,..., K (2) where L i is the number of data of X i, a of X (i) ɛx i have the same desired outputs, and K i=1 L i = L. Then we combine X i and X j as the training set of a two-cass probem T i,j, {( T ij = X (i), 1 e )} Li =1 {( X (j), e )} Lj =1 for i, j = 1,..., K and j i (3) After these two-cass probems are trained in parae, they are integrated according to a modue combination rue, namey the minimization principe. T i (x) = K min j=1 T ij(x) (4) where T ij (x) denotes the transfer function of the trained network corresponding to the two-cass subprobem T ij, and T i (x) denotes the transfer function of distinguish cass i from other casses. If these two-cass probems are sti in arge-scae or imbaanced, they can be further decomposed into reativey smaer two-cass probems. Suppose the training set X i defined in equation (2) is partitioned into N i (1 N i L i ) subsets in the form X ij = {X (ij) } L(j) i =1, for j = 1,..., N i (5) where L (j) i is the number of data of X ij, and Ni j=1 X ij = X i. The training set of each smaer two-cass probem can be given by T (u,v) ij = {( X (iu) )} (u) L {( )} (v) i L, 1 e X (jv) j, e =1 =1 for u = 1,..., N i, u = 1,..., N j, i, j = 1,..., K and j i (6) where X (iu) X iu and X (jv) X jv are the input vectors beonging to cass C i and C j, respectivey. After these smaer two-cass probems T (u,v) ij have been trained, they wi be integrated according to the minimization principe and maximization principe, respectivey, as foows: T (u) ij B. M 3 -GZC Network (x) = Nj min v=1 T (u,v) ij (x) (7) T ij (x) = max Ni T (u) u=1 ij (x) (8) Suppose the training set of each subprobem has ony two different instances, then they can be separated by a Gaussian zero-crossing discriminate function [9] defined by [ ( ) ] 2 x ci f ij (x) = exp σ [ ( ) ] 2 x cj exp (9) σ where x is the input vector, c i and c j are the given training inputs beonging to cass C i and cass C j (i j), respectivey, σ = λ c i c j, and λ is a user-defined constant. The output of M 3 -GZC network is defined as foows: 1 if y i (x) > θ i g i (x) = Unknown if θ j y i (x) θ i (10) 1 if y i (x) < θ j where θ i and θ j are the threshod imits of cass C i and C j, respectivey, and y i denotes the transfer function of the M 3 network for cass C i, which discriminates the pattern of the M 3 network for cass C i from those of the rest of the casses. From equations (9) and (10), we can see that the interpoation and extrapoation capabiities can be easiy controed by seecting different vaues of threshod imits, as shown in Fig.1. And in the next section, we wi further anayze the reationship between threshod imits and decision boundaries. III. PROPERTIES OF RECEPTIVE FIELD IN M 3 -GZC NETWORK Definition 1) Receptive Fied: the input space that can be cassified to one cass in a M 3 -GZC network. RF = {x xɛr n, i, g i (x) = 1} (11) Lemma 1: Suppose there are ony two instances c i and c j, and we ony concentrate on the receptive fied around c i. Then the reationship between the ongest receptive fied radius r max and the distance between c i and c j can be expressed as r max = k 1 c i c j (12) where k 1 is ony correated with λ and θ i. Proof: According to the axiom of norm, the foowing equation is satisfied. c i c j x c i x c j c i c j + x c i (13) 787

3 Proof: Suppose RF i,j is the receptive fied around instance c i based on c i and c j, according to Lemma 1, the radius r j of RF i,j satisfies the foowing inequaity k 2 c i c j r j k 1 c i c j (18) (c) Fig. 1. Decision boundaries at different threshod imits. θ i = θ j = 0.8; θ i = θ j = 0.4; θ i = θ j = 0.1; (d) θ i = θ j = 0; The red area denotes the Unknown decision regions. So the ongest receptive fied radius r max can be achieved when x c j = c i c j + x c i. From equations (9) and (12), we get [ ( ) ] 2 k1 c i c j θ i = exp λ c i c j [ ( ) ] 2 k1 c i c j + c i c j exp = exp [ ( ) ] 2 k1 exp λ (d) λ c i c j [ ( ) ] 2 k1 + 1 λ (14) which means k 1 is a function of λ and θ i. This competes the proof. Aso, we can prove that the reationship between the shortest receptive fied radius r min and c i c j can be expressed as: r min = k 2 c i c j (15) where k 2 satisfies the foowing equation [ ( ) ] [ 2 ( ) ] 2 k2 1 k2 θ i = exp exp λ λ (16) Theorem 1: Suppose instance c j is the nearest instance to instance c i, c j and c i beong to different casses, RF i is the receptive fied around c i, then the radius r i of RF i is between rmin i and ri max, where r i min = k 2 c i c j r i max = k 1 c i c j (17) Suppose c k1, c k2,... are other instances that beong to different casses from the cass of instance c i, RF i,k1, RF i,k2,... are the receptive fieds around instance c i based on c i and c k1, c k2,..., r k1, r k2,... are the radius of RF i,k1, RF i,k2,..., respectivey. According to Lemma 1, r k1, r k2,... satisfy the foowing inequaities k 2 c i c k1 r k1 k 1 c i c k1 k 2 c i c k2 r k2 k 1 c i c k2... (19) Since c j is the nearest neighbor in different cass to instance c i, c i c k1 and c i c k2 satisfy the foowing inequaities c i c k1 c i c j c i c k2 c i c j... (20) Since the roe of minimization principe is simiar to the ogica AND [11], the fina receptive fied RF i around c i satisfies the foowing equation So and RF i = RF i,j RF i,k1 RF i,k2... (21) r i min (k 2 c i c j, k 2 c i c k1, k 2 c i c k2,...) =k 2 c i c j (22) r i min (k 1 c i c j, k 1 c i c k1, k 1 c i c k2,...) =k 1 c i c j (23) This competes the proof. From the anaysis above, we can concude that the receptive fied around every instance has the foowing attribute: 1) The size, shape and ocation of receptive fied around each instance are ony correated with λ, θ and the neighbors in different casses around this instance. The receptive fied is oca and its size is mainy determined by the nearest neighbor in different casses around it (as depicted in Fig.1). 2) Receptive fieds around border instances are smaer than that around non-border instances since the distance between border instances are smaer than that between non-border instances (as depicted in Fig.1). 3) The radius of receptive fied is decreased with the increase of threshod imits (as depicted in Fig.1). A. C-M 3 -GZC IV. CLUSTERING ALGORITHM BASED ON M 3 -GZC NETWORK Since the receptive fied of every instance is the oca area around it, we can view a new instances that ocates in the area beongs to the same custer as this instance. Since the 788

4 receptive fied of every instance may be overapped, we ony consider the nearest instance to the new instances. When a new instance (x, d) is avaiabe, we can find its nearest neighbor (x, d) in the same cass as (x, d). If the output of the MIN unit around (x, d) is 1 (we say that (x, d) is accepted by the MIN unit around (x, d)), the fina output of the M 3 -GZC network wi be d, which means that (x, d) ocates in the receptive fied around (x, d). So we treat (x, d) and (x, d) beong to a same custer. If the output of the MIN unit around (x, d) is Unknown or 1, the fina output of the M 3 -GZC network wi be Unknown or incorrect, we treat (x, d) as a new custer center. Aso, since the radius of receptive fied is decreased with the increase of threshod imits, if a test sampe is accepted by a M 3 -GZC network with high vaues of threshod imits, it wi be accepted by the same network with ower vaues of threshod imits. The threshod imits can be viewed as a degree of confidence of correct cassification. Since the confidence of correct cassification wi become higher and higher with more and more instances avaiabe, we can adjust the threshod imits during custering process. At the beginning of custering, there are ony few instances avaiabe, they distribute sparsey and it is difficut to cassify new instance correcty according to them. The distance between each instance may be very arge, and the receptive fied of each instance may be arge, too. We can set a arge vaue of θ to decrease the size of receptive fied. With more and more instances avaiabe, the confidence of correct cassification becomes higher and we can set a smaer vaue of θ. The time varying threshod imits can guarantee a more robust incrementa custering agorithm. According to the above anaysis, our custering agorithm based on M 3 -GZC network (C-M 3 -GZC) is presented in Agorithm 1, where S is the custering set, and it can start from scratch or have some members trained before. The items of S can be expressed as (x, d, n), where x is the centroid coordinates of the custer, d is the cass ID of this custer, and n is the tota number of instances in this custer. Agorithm 1 C-M 3 -GZC Input: Training set: S new Previousy trained custer set: S Parameter of M 3 -GZC network: λ Time varying function of threshod imits: f(t) Output: New custer set: S for each data (x, d) in S new do Find its nearest neighbor (x, d, n) in S; if d = d and (x, d) can be accepted by the MIN unit based on (x, d ) then Update the centroid coordinates of (x, d, n): x = (nx + x)/(n + 1); Update the number of points in this custer: n=n+1; ese Treat (x, d) as a new custer center: S = S (x, d, 1); end if Update the threshod imits: θ = f(t); end for A simpe iustration of the proposed custering process is depicted in Fig.2. Here the time varying function of threshod imits is defined by ] θ = (θ max θ min )exp [ t2 2σ 2 + θ min (24) (c) (d) We chose θ max = 1.0, θ min = 0.2, σ = 1/2 2, and t = i/n, where i indicates the ith instances that presented to the M 3 -GZC network, and n indicates the tota number of training data. Fig.2 is the distribution of training data that beong to two casses abeed by pus and circe, respectivey. Fig.2 shows the decision boundaries of M 3 -GZC network after two instances have been presented. The third instance abeed by triange ocates in the Unknown area, so it wi be treated as a new custer center. But the forth instance ocates in the receptive fied of the third instance, and they beong to the (e) Fig. 2. A simpe iustration of the proposed custering process. The red area denotes the Unknown decision regions. Origina training data; -(e) Decision boundaries of M 3 -GZC network after more and more instances are earned, the trianges denote the instances waiting to be processed; (f) The fina custers. (f) 789

5 same cass, so they wi be custered as shown in Fig.2 (d). Fig.2 (f) shows the fina resuts of three custers as we have expected. B. Post-Processing In this subsection, we propose an outier remova agorithm and a custer combination agorithm for post-processing of C-M 3 -GZC. A custer that has few instances may ocate on border or be noise instances. If it is a border instance, it wi have some neighbor custers that beong to the same cass as it. But if it a noise instance, its neighbor custers wi beong to different casses. So our noise remova or outier remova agorithm works as Agorithm 2. Agorithm 2 Outier Remova Input: Custer set: S Limit of instance number: n 1 Number of neighbor custers to be considered: n 2 Output: New custer set: S for each custer (x, d, n) in S do if n n 1 then find its n 2 nearest neighbor custers in S; if a the n 2 neighbors beong to different casses of (x, d, n) then Remove (x, d, n) form S: S = S\(x, d, n); end if end if end for After the outier remova, some custers that are divided by noise can be combined. Aso, since the threshod imits are decreasing with time, some custers that are divided under high threshod imits can be combined under ow threshod imits. And with more training data avaiabe, some custers that are divided under few training data can be combined under more training data. So we need a custer combination agorithm to reduce custer number for further compression. If a custer can be accepted by its nearest neighbor custer, we treat the two custers are simiar and combine them into one custer. Based on this idea, our custer combination agorithm works as Agorithm 3. The custer combination process can be impemented ony once or repeated for many times, unti a certain custer number is reached. C. Compexity Anaysis Let M be the fina number of custers, and N be the number of training data. In the custering process, the time compexity of finding the nearest neighbor for one training instance is O(M), deciding whether it is accepted by the corresponding MIN unit is O(M). So the tota time compexity is O(MN). In the outier remova process, the time compexity of finding the nearest neighbor for one custer is O(M). The tota time compexity is O(M 2 ). In the Agorithm 3 Custer Combination Input: Custer set: S Parameter of M 3 -GZC network: λ Threshod imits of M 3 -GZC network: θ com Output: New custer set: S for each custer (x, d, n) in S do Find its nearest neighbor (x, d, n ) in S\(x, d, n); if d = d and (x, d) can be accepted by the MIN unit based on (x, d ) then Update the centroid coordinates of custer (x, d, n ): x = (n x + nx)/(n + n); Update the number of points in this custer: n = n + n; Remove (x, d, n) from S: S = S\(x, d, n). end if end for custer combination process, the time compexity of finding the nearest neighbor for one training instance is O(M), deciding whether it is accepted by the corresponding MIN unit is O(M). The tota time compexity is O(M 2 ). If we use some searching techniques (such as k-dimensiona trees [15]) the time compexity of finding the nearest neighbor for one instance can be reduced to O(ogM). The tota time compexity of C-M 3 -GZC, outier remova and custer combination agorithms can be reduced to O(N ogm), O(MogM) and O(MogM), respectivey. V. EXPERIMENTAL RESULTS In order to verify our method, we present three experiments. The first is an artificia probem and the other two are rea-word probems. A the experiments were performed on a 2.8GHz Pentium 4 PC with 1GB RAM. The time varying function of threshod imits we used in theses experiments is the same as the simpe iustration we presented in Fig.2. A. Checkerboard Probem A checkerboard probem is depicted in Fig.3. The checkerboard divides a square into four quadrants. The points abeed by dot and pus are positive and negative instances, respectivey. In this experiment, we randomy generate 1000 instances as shown in Fig.3. We use the C-M 3 -GZC agorithm to find custers at different threshod imits on this data set. Since there is no noise in the training data, we just use the custer combination process as the post processing. The custer centers at different threshod imits are shown in Figs.3, and (c). Resuts of LVQ are aso shown in Fig.3 (d) for comparison. We aso randomy generate instances as training data set, and another instances as test data set. We use the C-M 3 -GZC agorithm and the custer combination process on the training data set, and then cassify the test instances to the same cass of their nearest custers. Two ways of using custer combination are isted in Tabe I. One 790

6 TABLE I RESULTS AT DIFFERENT THRESHOLD LIMITS ON CHECKERBOARD PROBLEM. θ min C1-M 3 -GZC LVQ C2-M 3 -GZC LVQ Number Accuracy Time Accuracy Time Number Accuracy Time Accuracy Time % % % % % % % % % % % % % % % % % % % % % % % % % % % % (c) Fig. 3. A checkerboard probem and its custering resut at different threshod imits. The points abeed by dot and pus are positive and negative instances, respectivey. The points abeed by dot with circe and pus with circe are custer centers. A checkerboard probem; θ min = θ com = 0.01, θ max = 1.0; (c) θ min = θ com = 0.5, θ max = 1.0; (d) Resuts of LVQ with same custer number as. is using custer combination ony once, the other is repeating it unti the custer number is stabe. They are indicated as C1-M 3 -GZC and C2-M 3 -GZC in Tabe I, respectivey. The resuts of using LVQ are aso isted for comparison. Since the custer number of LVQ is defined by user, we do the experiments of each data set twice, which have the same custer number as that of C1-M 3 -GZC and the that of C2- M 3 -GZC, respectivey. From Fig.3 and Tabe I, we can draw the foowing concusions. 1) The number of training data can be greaty reduced by C-M 3 -GZC, and it is determined by the threshod imits. The higher the threshod imits are, the more (d) custers are generated. We aso find that if we set θ min = 0 and use the custer combination process unti a stabe custer number is reached, the fina custer number is 4, which is just the same as the minimum custer number as we have expected; 2) Using higher threshod imits can obtain higher cassification accuracy; 3) As shown in Fig.3 and (c), custers near the border are distributed more densey than that far away from border. That is because the receptive fieds around border instances are smaer than that around non-border instances. More custers around border can obtain a better generaization performance, and fewer custers around non-border can obtain a higher data reduction rate. We do not discard the non-border instances for further reduction because in the incrementa earning process the non-border instances may become border instances after more training data avaiabe, discarding them may ead to great and unwanted changes to the decision boundaries. Since not a the non-border instance wi become border instance in future, we pay ess attention to them than the instances that are aready border instances. Attaching more importance to border instances is a baance between data reduction rate and generaization performance. Compared with Fig.3 (d), we can see that custers distribute eveny in LVQ. And if the custer numbers of LVQ and C-M 3 -GZC are same, the generaization performance of the atter is better than the former, as shown in Tabe I. 4) Compared with LVQ, our agorithm spend ess time, ony 0.016% of that of LVQ on average. That is because LVQ wi process a the training data set iterativey to achieve a good resut, whie our method wi ony treat each input once. To test our noise remova program, we randomy generate instances with different ratio of noise instances among them. The parameters of each experiments is θ min = θ com = 0.2, θ max = 1.0, n 1 = 2, and n 2 = 2. The resut are shown in Fig.4. Compared with Tabe I, the number of custers are arger at the same parameters, that is because the noise instances separate some custers into smaer custers. 791

7 Accuracy No Custer C M3 GZC CP M3 GZC Number of Custers Noise Ratio No Custer C M3 GZC CP M3 GZC Noise Ratio Fig. 4. Custering resut of checkerboard probem with noise instances. Accuracy; Number of custers. The resuts show that after noise remova programm, the cassification accuracy can be improved greaty, and is better than the origina training data set. B. UCI Database In this experiment, our agorithm is tested on three benchmark data sets from the Machine Learning Database Repository [13]. The parameters of each experiments are θ max = 1.0, θ min = 0.2, and θ com = 0.2. The resuts are isted in Tabe II, where C-M 3 -GZC denotes our custering method without post-processing and CP-M 3 -GZC denotes our custering method with post-processing. Experiments resuts of using LVQ with the same custer number as that of C-M 3 -GZC and CP-M 3 -GZC are aso isted for comparison. From Tabe II, we can see that the accuracy of our methods is comparative or better than origina data set whie maintaining sma size of instances. The size ratio of each data set are different because each data set has different redundant training data. The accuracy of our method is aso better than LVQ and the training time is much faster than that of LVQ. Fig. 5. Gass-board images in the industry image cassification project. A good gass-board; A faut gass-board. C. Industry Image Cassification We aso use our custering method on an industry image cassification project [4]. The purpose of this project is to distinguish faut gass-boards from good gass-boards in an industria product ine, as shown in Fig.5. The abiity of incrementa custering is urgenty needed in this project since the gass-board images can be obtained everyday and vary with the changes of circumstances. In our experiment, each gass-board image is converted into a 4096 dimension vector, and we divided the gassboard images into eight groups according to the time that they were coected. The number of images in each group is 1133, 1227, 1149, 1160, 1138, 1147, 1088, and 1197, respectivey. We use the first to the seventh groups as the training set and the eighth group as the test set. At first, we use C-M 3 -GZC to obtain the custer C 1 on the first training set. Then we presented the second training set to C 1 and use C-M 3 -GZC to obtain the custer C 2. After a the training set have been presented, we use outier remova and custer combination agorithm on C 7 to get the fina custers. The cassification accuracy and the number of custers are shown in Fig.6. We aso show the resut of not using custering method for comparison. Since LVQ has not the abiity of incrementa earning, experiments of using LVQ are not done for comparison. From Fig.6, we can see that the size of training data can be greaty reduced whie maintaining the generaization performance by using C-M 3 -GZC agorithm. Aso, after the post-processing, the noise can be removed, the generaization performance becomes better, and the number of custers can be decreased further. VI. CONCLUSIONS In this paper we have anayzed the properties of receptive fied in M 3 -GZC network. Based on these properties, we propose a new supervised custering agorithm. It has the foowing attractive features. The incrementa custer abiity. Unike traditiona custering method, it need not retreat training instances. The number of custer need not be predefined, it grows automaticay and is determined by the distribution of training data. 792

8 TABLE II RESULTS ON UCI DATABASE. PARAMETERS OF EACH NET: λ = 0.5; θ max = 1.0; θ min = theta com = 0.2; n 1 = 2; n 2 = 2. THE LEFT COLUMNS IN ACCURACY AND TIME INDICATE THE RESULTS OF C-M 3 -GZC OR CP-M 3 -GZC WHILE THE RIGHT COLUMNS INDICATE THE RESULTS OF LVQ. Data set C-M 3 -GZC / LVQ CP-M 3 -GZC / LVQ No Custer Size Accuracy Time Size Accuracy Time Accuracy baance 92.0% 84.00% 87.20% % 88.00% 86.40% % iris 16.0% 97.33% 90.67% % 97.33% 90.67% % optdigits 77.8% 97.94% 75.63% % 98.11% 75.85% % Accuracy Number of Custers Taining Data Set Taining Data Set Fig. 6. Custering resut of industry image cassification probem. The soid ines and dashdot ines represent the resuts of using our custering method and not using custering method, respectivey. Accuracy; Number of custers. It attach different attention to border and non-border instances. The post processing of outier remova make the agorithm robust and custer combination reduce the size of custer further. Experimenta resuts on the artificia checkerboard probem and severa rea-word appications verify the vaidity of our agorithm. But in our experiments, we simpy set n 1 = 2 and n 2 = 2 for a the data base. How to choose the optimum vaues of n 1 and n 2 for rea-word appications needs to be anayzed in the future work. ACKNOWLEDGMENT This work was supported in part by the Nationa Natura Science Foundation of China via the grants NSFC and NSFC REFERENCES [1] S. Basu, A. Banerjee, and R. J. Mooney, Semi-supervised Custering by Seeding, Proc. 19th Internationa Conference on Machine Learning, pp.19-26, [2] N. Bansa, A. Bum, and S. Chawa, Correation Custering, Machine Learning, vo.56, pp , [3] N. Grira, M Crucianu, and N Boujemaa, Unsupervised and Semisupervised Custering: a Brief Survey, A Review of Machine Learning Techniques for Processing Mutimedia Content, Report of the MUSCLE European Network of Exceence (FP6). [4] B. Huang and B. L. Lu, Faut diagnosis for industria images using a min-max moduar neura network, ICONIP 2004, Lecture Notes in Computer Science, vo.3316, pp , [5] T. Kohonen, The Sef-Organizing Map, Proc. Institute of Eectrica and Eectronics Engineers, vo.78, pp , [6] T. Kohonen, improved versions of earning vector quantization, IEEE Internationa Joint Conference on Neura Networks, vo.1, pp , [7] X. Li and N. Ye, Grid-And Dummy-Custer-Based Learning Of Norma And Intrusive Custers For Computer Intrusion Detection, Quaity and Reiabiity Engineering Internationa, vo.18, pp , [8] X. Li and N. Ye, A Supervised Custering Agorithm for Computer Intrusion Detection, Knowedge and Information Systems, vo.8, pp , [9] B. L. Lu and M. Ichikawa, A Gaussian zero-crossing discriminant function for min-max moduar neura networks, proc. 5th Inte Conf. Knowedge-Based Inteigent Information Engineering Systems & Aied Technoogies, pp , N.Baba et a. Eds., IOS Press, [10] B. L. Lu and M. Ichikawa, Emergent On-Line Learning with a Gaussian Zero-Crossing Discriminant Function, IJCNN 02, vo.2 pp , [11] B. L. Lu and M. Ito, Task decomposition based on cass reations: a moduar neura network architecture for pattern cassification, Lecture Notes in Computer Science, vo.1240 pp , 1997 [12] B. L. Lu and M. Ito, Task Decomposition and Modue Combination Based on Cass Reations: a Moduar Neura Network for Pattern Cassification, IEEE Trans. Neura Networks, Vo.10, pp , [13] P. M. Murphy and D. W. Aha, UCI Repository of Machine Learning Database, Dept. of Information and Computer Science, Univ. of Caif., Irvine, [14] C. M. Procopiuc, Custering probems and their appications (a survey), Department of Computer Science,Duke University.1997 [15] R. Sprou, Refinements to Nearest-Neighbor Searching in k- Dimensiona Trees, Agorithmica, pp , [16] A. R. Webb, Statistica Pattern Recognition, 2nd, Ed. London, U.K.: Wiey,