Unsupervised Learning for Hierarchical Clustering Using Statistical Information

Transcription

1 Unsupervised Learning for Hierarcical Clustering Using Statistical Information Masaru Okamoto, Nan Bu, and Tosio Tsuji Department of Artificial Complex System Engineering Hirosima University Kagamiyama 1-4-1, Higasi-Hirosima, Hirosima, JAPAN {okamoto, bu, ttp:// Abstract. Tis paper proposes a novel ierarcical clustering metod tat can classify given data witout specified knowledge of te number of classes. In tis metod, at eac node of a ierarcical classification tree, log-linearized Gaussian mixture networks [2] are utilized as classifiers to divide data into two subclasses based on statistical information, wic are ten classified into secondary subclasses and so on. Also, unnecessary structure of te tree can be avoided by training in a cross-validation manner. Validity of te proposed metod is demonstrated wit classification experiments on artificial data. 1 Introduction Recently, tere ave been growing interests in using bioelectric signals suc as electromyogram (EMG) to conduct man-macine interface. In order to discriminate an operater s intentions from bioelectric signals efficiently, several attempts ave been made so far [1], [2]. Generally, suc pattern discrimination is performed by estimating te relationsip between te bioelectric signals as feature vectors and te corresponding intentions as class labels. However, difference between classes in te bioelectric signals of elderly or andicapped people is ambiguous, and tis relates to poor reliability of te class labels available. To overcome tis problem, clustering analysis as been widely adopted, in wic a collection of patterns is organized into clusters based on similarity. Te previously proposed clustering analysis tecniques can be dicotomized as eiter k-means algoritm or ierarcical clustering. Te k-means algoritm identifies a partition of te input space. On te oter and, te ierarcical clustering performs a nested series of partitions and finally performs a grouping wit a suitable number of classes. Also, in order to determine te number of class automatically, a clustering algoritm using self organizing maps (SOM) [5] as been proposed [6]. In tis metod, estimation of te number of classes is carried out based on te number of te data belonging to eac node of SOM. However, wen parameters in tis metod were not set up appropriately, suc metod may fail to perform satisfying clustering for complicated data. F. Yin, J. Wang, and C. Guo (Eds.): ISNN 2004, LNCS 3173, pp , c Springer-Verlag Berlin Heidelberg 2004

2 Unsupervised Learning for Hierarcical Clustering 835 X1 (1,1) (1) w1 O1 1,1 O1,1 O1 x1 x2 xd Nonlinear transformation X2 X X H (c,m) w (1) OH 1,M1 c,m C,MC Oc,m O1,M 1 Oc OC OC,M C Fig. 1. Structure of LLGMN In tis paper, a novel ierarcical clustering metod is proposed. In tis metod, a probabilistic NN tat is derived from te Gaussian mixture model (GMM), called a log-linearized Gaussian mixture network (LLGMN) [2], is utilized for partition at eac non-terminal node. Te proposed metod can estimate te number of terminal nodes corresponding to te number of classes according to te statistical information obtained solely from te training data. 2 LLGMN 2.1 Structure of LLGMN Te structure of LLGMN is sown in Fig.1. First, an input vector x R D is converted into a modified vector X as follows: X =[1, x T,x 2 1,x 1 x 2,,x 1 x D,x 2 2,x 2 x 3,,x 2 x D,,x 2 D] T. (1) Te first layer consists of H units corresponding to te dimension of X, and an identity function is used for an activation function of eac unit. Te variable (1) O ( = 1,,H) denotes te output of te t unit in te first layer. Eac unit in te second layer receives te output (1) O weigted by coefficients (c =1,,C; m =1,,M c ). C denotes te number of classes and M c is te number of components belonging to class c. Te relationsip between te input I c,m and te output O c,m of unit {c, m} in te second layer can be defined as w (c,m) I c,m = O c,m = H =1 (1) O w (c,m), exp[ I c,m ] C Mc, c =1 m =1 exp[ I c,m ]

3 836 M. Okamoto, N. Bu, and T. Tsuji were w (C,M C) = 0. Te relationsip between te input I c and te output is described as, O c = I c = M c O c,m. (4) Te output of te tird layer O c P (c x) of class c. m=1 corresponds to te posterior probability 2.2 Supervised Learning Algoritm [2] Consider a training set {x (n), T (n) } (n =1,,N), were T (n) = {T (n) 1,, T (n) C }. If te input vector x(n) belongs to class c, T c (n) = 1, and T (n) c = 0 for all of te oter class c. An energy function according to te minimum log-likeliood training criterion can be derived as: J SV = N C n=1 c=1 T c (n) log O c (n). (5) In te training process, modification of te LLGMN s weigt w (c,m) is defined as: N w (c,m) JSV n = η, (6) n=1 w (c,m) JSV n =( O w (c,m) c,m (n) were η>0 is te learning rate. 3 Hierarcical Clustering O c,m (n) O c (n) T c (n) )X (n), (7) Te divisive clustering starts from a single cluster, and terminates wen a termination criterion as been satisfied, so tat te training data are divided into te appropriate number of clusters. At eac non-terminal node, LLGMN is used to acieve binary splits. Even for data of complicated distributions, interpretable clustering can be made after a nested series of binary splits. In tis section, after te description of te proposed unsupervised learning algoritm of LLGMN, division validation according to te statistical properties of te training data and pruning law are explained. 3.1 Unsupervised Learning Algoritm Given te number of classes, C, te entropy used as cost function is defined as: J SO = N C n=1 c=1 O c (n) log O c (n), (8)

4 Unsupervised Learning for Hierarcical Clustering 837 were N is te number of total data. Te proposed unsupervised learning algoritm modifies weigts by minimizing Eq. (8), However, for some initial weigts, te LLGMN may be trained to cluster all training data into one class, and cost function, J SO, may converge to suc a local minimum. Terefore, in te proposed metod, te initialization of te weigts is carried out to prevent te LLGMN to converge to one of te local minima, and te number of classes is restricted to two. Let us consider tat LLGMN clusters data into two classes: C 1 and C 2. First, x 1 and x 2 are cosen for te initialization of weigts from te total training data set A according to te following equation, (x 1, x 2 ) = argmax ( x (i) x (j) ). (9) x (i),x (j) A Ten te set B, wic means te set of te utilized data, is set as {x 1, x 2 }. Assuming tat x 1 and x 2 are labeled wit C 1 and C 2, respectively. Training of LLGMN is performed using te supervised learning rule [2] in order to classify x 1 and x 2 into C 1 and C 2 respectively. Ten, wit te initialized weigts, unsupervised learning of te LLGMN is performed using te set B. Te mean values of x C1 and x C2 are calculated using te training data clustered into C 1 and C 2, respectively. One datum x A B, from wic te distance to eiter x C1 or x C2 w (c,m) is te smallest, is added into te set B. Ten, modification of te weigt is defined as: w (c,m) J (n) SO w (c,m) = η J(n) SO w (c,m), (10) = (J SO log O c (n) ) O c,mx (n) (n). (11) After training wit a pre-defined number of times, anoter training datum is selected from te set A B and added into te set B. Tis step of training repeats, until all te training data is added into B, tat is to say, B = A. 3.2 Division Validation Wit te proposed metod, unnecessary splits may occur wen te ierarcy of te tree becomes too deep. In tis metod, cross-validation is adopted and te posterior probabilities of te validation data is utilized to determine weter to split a node or not. First, te validation data is prepared and te entropy H(x) is defined as: C H(x) = O c (n) log O c (n). (12) c=1 Ten, te average value H E of H(x) is utilized as te termination criterion. H E = 1 H(x (n) ), (13) N c x (n) N c

5 838 M. Okamoto, N. Bu, and T. Tsuji C2 C4 C5 y C3 0.2 C x Fig. 2. Examples of artificial data were N c stands for te set of validation data belonging to te node under consideration, and N c is te number of validation data in N c.ifh E is iger tan a tresold H T, splitting of te corresponding node is terminated. On te oter and, if all validation data of te node in consideration are clustered into one class, outliers may exist in te training data and te division of tis node must be terminated. Also, for occasions wen tere is only one training data in a node, furter split of tis node must be terminated, since division is impossible. Wit tis validation, a classification tree can be constructed based on te statistical properties of te training data, and can cluster complicated data into a proper number of classes. 3.3 Pruning Law In te proposed metod, outliers are always classified into some terminal nodes (clusters) separated from oter major clusters. Especially, wen te ierarcy of te tree grows too large, te influence of outliers becomes prominent because of a decrease of te number of training data in eac node. After te classification tree is constructed, pruning is conducted to improve te clustering efficiency. Te number of training data left in eac terminal node is utilized as a decision index of pruning. If te ratio of te number of training data in a terminal node to te total training data number is lower tan tresold α T, tis node and its counter are merged into teir fater node. Wit tis pruning law, excessive splits may be prevented, and te number of clustering may not increase corresponding to te number of outlier data. 3.4 Experiments Numerical simulations were carried out in order to verify te proposed metod. Te feature data is illustrated in Fig. 2: Tere are 2-dimensional data x R 2, and generated from five classes, C i (i =1, 2,, 5). Eac class consists of one normal distribution. Te number of training data for eac class is 100, and te number of validation data for eac class is 200. Te LLGMN includes seven units

6 Unsupervised Learning for Hierarcical Clustering 839 in te first layer, two units in te second layer corresponding to te total number of components, and two units in te tird layer. To construct te classification tree, tresold of entropy H T is set as 0.2, tresold of pruning α t as 0.01, learning rate η as 0.01, and training times in eac addition of training data as 100. Te classification tree starts from te root node, were training data are divided into two nodes at eac non-terminal node, and finally, a ierarcical tree is constructed from five terminal nodes. To validate te generalization ability, 300 samples for eac class tat are not used in training process were clusterd, and te discrimination rate for 20 independent trials was 98.5 ± 0.64%. It can be found tat te proposed metod can estimate te number of classes and acieve ig classfication rate. 4 Conclusion In tis paper, to deal wit te discrimination problem of ambiguous teacer signals, a ierarcical clustering metod was proposed. In tis metod, entoropy of te LLGMN s outputs at eac node are used as te termination criterion, and unnecessary splits in te structure of te classification tree can be avoided, so tat te proposed metod can make an interpretable and reasonable partition of te training data according solely to its statistical caracteristics. In future works, we would like to carry out discrimination experiments on various data and to examine te influence of te parameters to te clustering result. Furtermore, we would like to establis an improved metod tat determines te value of teresolds suc as α T automatically. References 1. Hiraiwa, A., Simoara, K., Tokunaga, Y.: EMG Pattern Analysis and Classification by Neural Network. IEEE International Conference on Syst., Man and Cybern., (1989) Tsuji, T., Fukuda, O., Icinobe, H., Kaneko, M.: A Log-linearized Gaussian Mixture Network and its Application to EEG Pattern Classification. IEEE Trans. on System, Man and Cybernetics-Part C: Applications and Reviews, Vol. 29., No. 1. (1999) Anderberg, M.R.: Cluster Analysis for Applications. Academic Press, New York (1974) 4. Ward, J.H.: Hierarcical Grouping to Optimize an Objective Function. Journal of te American Statistical Association, Vol. 58., No (1963) Koonen, T.: Self-organization and Associative Memory. Tird Edition, Springer- Verlag, Berlin (1994) 6. Terasima, M., Siratani, F., Yamamoto, K.: Unsupervised Cluster Segmentation Metod Using Data Density Histogram on Self-organizing Feature Map. IEICE Transactions on Information and Systems, PT. 2, Vol. J79., No. 7., (1996) (in Japanese)