An Ensemble of Adaptive Neuro-Fuzzy Kohonen Networks for Online Data Stream Fuzzy Clustering

Transcription

1 An Enseble of Adative euro-fuzzy Kohonen etworks for Online Data Strea Fuzzy Clustering Zhengbing Hu School of Educational Inforation Technology Central China oral University Wuhan China Eail: Yevgeniy V. Bodyanskiy Kharkiv ational University of Radio Electronics Kharkiv Ukraine Eail: Oleksii K. Tyshchenko and Olena O. Boiko Kharkiv ational University of Radio Electronics Kharkiv Ukraine Eail: Abstract A new aroach to data strea clustering with the hel of an enseble of adative neuro-fuzzy systes is roosed. The roosed enseble is fored with adative neuro-fuzzy self-organizing Kohonen as in a arallel rocessing ode. Their learning rocedure is carried out with different araeters that define a nature of cluster borders blurriness. Clusters quality is estiated in an online ode with the hel of a odified artition coefficient which is calculated in a recurrent for. A final result is chosen by the best neuro-fuzzy self-organizing Kohonen a. Index Ters Coutational Intelligence Data Strea Processing euro-fuzzy Syste Fuzzy Clustering Machine Learning. I. ITRODUCTIO Multidiensional data clustering is coon in Data Mining tasks. Such alication areas as Text Mining and Web Mining have becoe really widesread lately. A traditional aroach to solving this sort of tasks assues that each vector of a rocessed sequence ay only belong to a single class. Although it s a ore natural case when each secific observation ay be attributed to several classes at the sae tie with different ebershi levels. This situation is a subect under study for fuzzy cluster analysis [ ]. In this aroach the ost effective and silest ethods are robabilistic fuzzy clustering rocedures based on otiization of soe obective functions. Initial data for a fuzzy clustering roble is a sale of observations which consists of diensional n feature vectors X x x xk x R and a result of this clustering rocedure is a artition of the initial data set into overlaing classes with soe ebershi levels 0 u k of the k th feature vector to the th cluster. Thus the overwheling aority of the well-known fuzzy clustering algoriths is designated for a batch ode rocessing which eans that a sale volue can t be changed while the data are rocessed. There s a wide class of tasks to be solved only with the hel of the Data Strea Mining [3-6] aroach when data are fed and rocessed in an online ode. This task is rather tyical for Web Mining when inforation is fed in a real tie ode directly fro the Internet. Self-organizing as (SOMs) by Kohonen roved its efficiency in clustering tasks. Their efficiency is defined by their coutational silicity and their ability to work in a real tie ode for sequential data rocessing. These neural networks are learnt with the hel of self-learning rocedures based on the rinciles Winner takes all (WTA) and Winner takes ore (WTM). It s reviously assued that a structure of rocessed data ilies that fored clusters don t utually intersect which eans that it s ossible to build a searating hyer-surface which clearly distinguish different classes during a learning rocedure of a neural network. Recurrent odifications of the fuzzy clustering algoriths (which ake it ossible to solve a task in an online ode) were introduced for sequential data rocessing in [7 8]. It should be noted that the introduced rocedures are structurally close to the Kohonen self-learning rule according to the rincile «Winner Takes More». It allows

2 introducing a so-called «fuzzy clustering Kohonen network» [9] which ossesses a nuber of advantages coaring to a conventional self-organizing a. The well-known and ost coonly used fuzzy clustering algoriths can t be called fuzzy in the full sense because their results are significantly defined by a value of a secial araeter (also known as a fuzzifier which is chosen eirically). A case when belongs to an interval fro to corresonds to a transition fro cris borders which are obtained with the hel of the K-eans rocedure to their colete blurriness when all observations belong to all clusters with the sae ebershi level. We should note that in ost cases that corresonds to the fuzzy C-eans rocedure (FCM) by Bezdek [0]. There ay be a situation while rocessing real-world data when one obect belongs to different classes at the sae tie and these classes utually intersect (overla). Conventional SOMs don t take into consideration this occasion but this roble can be considered with the hel of fuzzy clustering techniques. The reainder of this aer is organized as follows: Section describes fuzzy clustering techniques with a variable fuzzifier. Section 3 describes an enseble s architecture of adative neuro-fuzzy Kohonen networks. Section 4 gives soe details on ossibilistic fuzzy clustering with a variable fuzzifier. Section 5 resents a realworld alication to be solved with the hel of the roosed fuzzy clustering aroach. Conclusions and future work are given in the final section. II. FUZZY CLUSTERIG WITH A VARIABLE FUZZIFIER Algoriths based on goal functions are considered to be strict fro a atheatical oint of view aong all clustering rocedures. They solve a task of their otiization under different a riori assutions. The ost coonly used rocedure in this situation is the robabilistic aroach which is based on a goal function s iniization under constraints () k ( ) E u c u k x k c u () k 0 u k (3) where u k [0] is a ebershi level of a vector x( k ) to the -th class c is a rototye of the -th cluster is a non-negative fuzzification araeter (a fuzzifier) which actually deterines a level of borders blurriness between clusters k. A result of this clustering rocedure is a U u which is also -atrix called a fuzzy artition atrix. We should notice that eleents of the atrix U due to the constraint () ay be considered as robabilities that data vectors belong to soe definite clusters. Because of this fact rocedures based on the iniization () are called robabilistic fuzzy clustering algoriths. A nuber of clusters is set beforehand and can t be changed during coutation rocedures. Introducing the Lagrange function ( ) Lu k c k u k xk c k k u k k (4) (here ( ) k is an undeterined Lagrange ultilier) and solving the Karush-Kuhn-Tucker syste of equations we can get a solution in the for

3 u c k x k c x k c l l u ( ) k k x k u k k l l k x k c (5) which coincides with the Fuzzy C-Means algorith (FCM) by J.Bezdek (when ). And when its results are close to results of the well-known conventional cris clustering algorith (Hard K-Means HKM). As an alternative to rocedures that use a fuzzifier Klawonn and Hoener [] offered an obective function for fuzzy robabilistic clustering ( ) ( ) ( ) (6) k E u c u k u k x k c with the constraints () and (3) where 0 is an adustable araeter which defines a nature of the obtained solution. Introducing the Lagrange function Lu k c ku ( ) u xk c k u k k l k and solving the Karush-Kuhn-Tucker syste of equations we coe to a solution u k xk c u k k k u k k 0 Lu k c k 0 L u k c k u k u k x k c 0 Lu k c k c u c k x k c k k l u k ( ) u x. u k ( ) u x k c l (7) It s easy to notice that when this rocedure coincides with FCM. Thus the rocedure (7) can t be used for solving Data Strea Mining tasks because it can t rocess inforation in an online ode. Therefore an adative odification of the exression (7) was introduced in []

4 u k ( ) c k c k u ku k xk c x k c k (8) l x k cl where is a learning rate araeter. It s easy to notice that the second recurrent exression (8) is the Kohonen self-learning rule according to the rincile «Winner Takes More» with a neighborhood function u k u k. III. A ESEMBLE OF ADAPTIVE EURO-FUZZY KOHOE ETWORKS Although a value of the araeter in the forulas (7) and (8) lies in a uch narrower range than a fuzzifier but there are currently no foral rules how to choose it and to tune it. Therefore while solving a concrete task in a batch ode this task is usually reeatedly solved with the hel of the exression (7) with different values (fro a very sall quantity to ). It s clear that such an aroach can t solve tasks effectively in an online ode. It ight be exedient to use an idea of an enseble of arallel working clustering rocedures [3 4] in this situation where each clustering rocedure works with a different fro others value. This enseble can be easily ileented with the hel of adative neuro-fuzzy Kohonen networks [5]. They are two-layer architectures where K rototyes are clarified in the Kohonen coetitive layer (which contains neurons ) and ebershi levels M are calculated in the outut layer (which contains neurons ). There is an architecture of the two-layer adative neuro-fuzzy Kohonen network in Fig.. There is also an enseble fored by such networks in Fig.. A self-learning algorith for the th enseble s eber q in an enseble that contains q neuro-fuzzy networks can be written down in the for Fig.. An adative neuro-fuzzy Kohonen network (FSOM) Fig.. An architecture of an enseble of adative neuro-fuzzy Kohonen networks

5 c k c k u k u k x k c u k xk c l xk cl (9) wherein the Kohonen layer is tuned with the hel of the first ratio (9) and the outut layer calculates ebershi levels u k for each incoing observation xk. Classification quality rovided by each enseble eber ay be estiated with the hel of any fuzzy clustering index []. Wherein one of the silest and ost effective indexes is the so-called artition coefficient (PC) which is a ean value of squared ebershi levels of all observations to each cluster: k PCk u. k (0) This coefficient has a clear hysical sense: the better clusters are exressed the higher the value PC is (a liit is PC ) its iniu ( PC ) is reached if data belong to all clusters evenly. But this henoenon is obviously a worthless solution. This coefficient is convenient in the fraework of the roosed syste because it allows online calculation. It should be noticed that the exression (0) is closely related to the traditional FCM. This coefficient ust be odified for a considered case in the for k k PC k u u which coincides with the exression (0) when. So clustering a data strea that is fed in an online ode is solved with the hel of arallel working adative neuro-fuzzy Kohonen networks which differ fro each other only by a value of the araeter. Thus the network s results are alied to a axiu value PC k as a final result at any articular tie oint. IV. POSSIBILISTIC FUZZY CLUSTERIG WITH A VARIABLE FUZZIFIER Basic robles that arise in fuzzy clustering have to do with constraints on a su of ebershi values (they ust be equal to ). That s why algoriths that use the constraint () are called robabilistic fuzzy clustering algoriths. An existence of this constraint leads to the fact that an observation which doesn t belong to any class gets the sae ebershi levels for all classes. One can avoid this drawback if the ideas of ossibilistic fuzzy clustering are used [6]. These ideas are based on iniization of an obective function u c u k E u c x k k l k where a scalar araeter 0 defines a distance where a ebershi level takes on a value of 0.5 which eans that if then x k c 0.5. u k Introducing an obective function [ 5] siilarly to (6) ( ) ( ) ( ) ( ) E u c u k u k x k c u k k k

6 and iniizing it by u c we coe to a odified ossibilistic rocedure in the for u c k xk c x k c u k k u k x k u k k u k k u k u k k u k u k x k c. () It s interesting to note that this ratio for a rototye calculation in the forulas (7) and () coletely coincides. The exressions () can be written down for an adative case in the for k xkck k xk c k k u k c k c kku ku kxkc k k k u u x c k u u k As one can see the second recurrent ratio () is the WTM self-learning rule by Kohonen with the neighborhood function u k u k. Although the ossibilistic algorith () is a little ore colicated fro a coutational oint of view than the robabilistic rocedure (8) its advantage is the fact that new clusters ay be detected with the hel of the ossibilistic aroach during online data rocessing. If a ebershi level of a new incoing observation xk to all classes turns out to be lower than soe redefined threshold then we can assue that there s a new th cluster and its initial rototye coordinates are xk c 0. The algorith () can be used as a learning rocedure for the two-layer FSOM (Fig.). Let s notice that an enseble of clustering neural networks with the hel of the ossibilistic aroach was introduced in [7 8] but its unwieldiness iedes its usage while rocessing data streas. Silicity of the enseble s (Fig.) nuerical ileentation akes it ossible to rocess data in a real tie ode. V. EXPERIMETS We have taken real-world data for our exerient. A data set describes students knowledge status about the subect of Couter Science. The data set contains ultivariate characteristics. It contains 30 instances with 5 attributes for each observation. Seaking of attribute inforation the data set contains these attributes: - STG (a degree of study tie for goal obect aterials); - SCG (a degree of a user s reetition nuber for goal obect aterials); - STR (a degree of user s study tie for related obects with a goal obect); - LPR (a user s exa erforance for related obects with a goal obect); - PEG (a user s exa erforance for goal obects). We chose 3 attributes (STR LPR PEG) for the sake of visualization.. ()

7 Fig.3. The Result of Fuzzy Clustering (STR/PEG) Fig.4. The Result of Fuzzy Clustering (PEG/ STR/LPR) As it can be seen the roosed ethod akes it ossible to reresent clusters in a rather coact for. A level of cluster overlaing is rather high (and it will kee on growing when a feature vector s diensionality increases). An a araeter for the considered case belonged to an interval [0.3; 0.4]. Fig.5. The Result of Fuzzy Clustering (SCG/PEG)

8 As it can be seen fro Fig.3 there are 3 fuzzy clusters. Generally seaking students current knowledge level should be deterined with the hel of real values (STG SCG PEG STR LPR). It should be noted that there are soe oints in Fig.3 which are not likely to be in those regions. For exale a oint belongs to a green class although all neighbor oints in that region belong to soe other ( ink or yellow ) class. Probably this fact eans that a student didn t actually send uch tie on his exa rearation although he deonstrated a high level of erforance (his PEG index is high). There ay be another situation when a student sent uch tie on his rearation but his results aren t very good. So the clustering accuracy increases for fuzzy algoriths and worsens for cris ones with the growth of a sale size; the clustering accuracy lessens with the growth of diensionality of a feature sace. Fuzzy rocedures are referable for those clustering tasks when every obect belongs to several categories at the sae tie. VI. COCLUSIO The ethod for online fuzzy clustering ultidiensional data sequences to be rocessed in a real tie ode is roosed. The task is solved with the hel of an enseble of adative neuro-fuzzy Kohonen networks which differ fro each other by a araeter s value that accounts for fuzziness of the received results. The roosed rocedure has rather sile coutational ileentation and akes it ossible to organize arallel couting to accelerate the syste s rocessing seed (because it can rocess data in an online ode and this fact is really iortant for Web Mining and Data Strea Mining). The results ay be successfully used in a wide class of Data Strea Mining Dynaic Data Mining Teoral Data Mining tasks and esecially in such alied areas as Web Mining Text Mining Medical Data Mining etc. So the roosed enseble of adative neuro-fuzzy self-organizing Kohonen as has roved its efficiency for online Data Strea fuzzy clustering. A nuber of exerients deonstrated a high effectiveness of the roosed neuro-fuzzy syste esecially under conditions of clusters overlaing. ACKOWLEDGMET The authors would like to thank anonyous reviewers for their careful reading of this aer and for their helful coents. This scientific work was suorted by RAMECS and CCU6A005. REFERECES [] F. Hoener F. Klawonn R. Kruse and T. Runkler Fuzzy Clustering Analysis: Methods for Classification Data Analysis and Iage Recognition Chichester: John Wiley & Sons 999. [] R. Xu and D.C. Wunsch Clustering (IEEE Press Series on Coutational Intelligence) Hoboken: John Wiley & Sons 009. [3] H. Bouchachia and E. Balaguer-Ballester DELA: A Dynaic Online Enseble Learning Algorith in Proc. nd Euroean Syosiu on Artificial eural etworks Coutational Intelligence and Machine Learning (ESA 04) Ar [4] D. Leite P. Costa Jr. and F. Goide Evolving granular neural networks fro fuzzy data streas eural etworks vol [5] D. Leite R. Ballini Pyrao Costa Jr. and F. Goide Evolving fuzzy granular odeling fro nonstationary fuzzy data streas Evolving Systes vol.3() [6] D. Leite R. Ballini Pyrao Costa Jr. and F. Goide Evolving fuzzy granular odeling fro nonstationary fuzzy data streas Evolving Systes vol.3() [7] D. Kangin and P. Angelov Evolving clustering classification and regression with TEDA in Proc. International Joint Conference on eural etworks (IJC 05) July [8] R. Hyde and P. Angelov A new online clustering aroach for data in arbitrary shaed clusters in Proc. International Conference on Cybernetics (CYBCOF 05) June [9] R.D. Baruah P.P. Angelov and D. Baruah Dynaically evolving clustering for data streas in Proc. Evolving and Adative Intelligent Systes (EAIS 04) June [0] R. Rosa F.A.C. Goide D. Dovzan I. Skranc Evolving neural network with extree learning for syste odeling in Proc. Evolving and Adative Intelligent Systes (EAIS 04) June [] D. Dovzan and I. Skranc Recursive clustering based on a Gustafson-Kessel algorith Evolving Systes vol. () [] R.D. Baruah P.P. Angelov and D. Baruah Dynaically evolving fuzzy classifier for real-tie classification of data streas in Proc. IEEE Int. Conf. on Fuzzy Systes (FUZZ-IEEE 04) July [3] R.D. Baruah and P.P. Angelov Online learning and rediction of data streas using dynaically evolving fuzzy aroach in Proc. IEEE Int. Conf. on Fuzzy Systes (FUZZ-IEEE 03) July [4] A.P. Leos W.M. Cainhas and F.A.C. Goide Multivariable Gaussian Evolving Fuzzy Modeling Syste IEEE Trans. Fuzzy Systes vol.9() [5] R.D. Baruah and P.P. Angelov Evolving fuzzy systes for data streas: a survey Data Mining and Knowledge Discovery vol.(6) [6] M. Prataa S.G. Anavatti M.J. Er and E. Lughofer Class: An Effective Classifier for Streaing Exales IEEE Trans. Fuzzy Systes vol.3() [7] Ye. Bodyanskiy V. Kolodyaznhiy and A. Stehan Recursive fuzzy clustering algoriths in Proc. 0 th East West Fuzzy Colloqiu Set

9 [8] Ye. Bodyanskiy Coutational intelligence techniques for data analysis Lecture otes in Inforatics vol. P [9] Ye. Gorshkov V. Kolodyazhniy and Ye. Bodyanskiy ew recursive learning algoriths for fuzzy Kohonen clustering network in Proc. 7 th Int. Worksho on onlinear Dynaics of Electronic Systes June [0] J.C. Bezdek Pattern Recognition with Fuzzy Obective Function Algoriths.Y.: Plenu Press 98. [] F. Klawonn and F. Hoener What is fuzzy about fuzzy clustering? Understanding and iroving the concet of the fuzzifier Lecture otes in Couter Science vol [] B. Kolchygin and Ye. Bodyanskiy Adative fuzzy clustering with a variable fuzzifier Cybernetics and Syste Analysis vol. 49 no [3] A. Tochy B. Minaei-Bidgali A.K. Jain and W.F. Punch Adative clustering ensebles in Proc.7th Int. Conf. on Pattern Recognition ICPR 004 Aug [4] S. Vega-Pons and J. Ruiz-Shulcloer A survey of clustering enseble algoriths Int. J. Pattern Recognition and Artificial Intelligence vol. 5 no [5] Ye. Bodyanskiy B. Kolchygin and I. Pliss Adative neuro-fuzzy Kohonen network with variable fuzzifier Int. J. Inforation Theories and Alications vol. 8 no [6] R. Krishnaura and J.M. Keller A ossibilistic aroach to clustering Fuzzy Systes vol. no [7] B.V. Kolchygin Enseble of neuro-fuzzy Kohonen networks for adative clustering in Proc. nd Int. Sci. Conf. of Students and Young Scientists Theoretical and Alied Asects of Cybernetics ov [8] Ye.V. Bodyanskiy V.V. Volkova and A.S. Yegorov Clustering of docuent collections based on the adative selforganizing neural network Radio Electronics Inforatics Control vol.(0)