Unsupervised Neural Network Adaptive Resonance Theory 2 for Clustering

Transcription

1 3 rd Internatonal G raduate Conference on Engneerng, Scence and Humantes (IGCESH) School of Graduate Studes Unverst Teknolog Malaysa 4 November 00 Unsupervsed Neural Network Adaptve Resonance Theory for Clusterng Ana Musdholfah and Dfla Yustsa Department of Computer Scence and Electroncs Gadah Mada Unversty (UGM) Yogyakarta, Indonesa non.nana@gmal.com Abstract Clusterng s one of pattern reco gnton technques whch are often used fo r extract nfo rmaton from large amount dataset to get mo re beneft for the data o wner. Clusterng whch s an unsupervsed technque assgns the nput data nto clusters based on ther smlarty degrees. In ths paper, the Unsupervsed Neural Network Adaptve Resonance Theory s used for clusterng data. To evaluate the results, a technque called Coheson and Separaton s utlzed. Furthermo re, n o rder to valdate the algorthm, ths study uses patent data. The proposed ART algorthms and valdato n technques scale well and gan consderable perfo rmance due to the resulted cluster. Keyword: Clusterng, Unsupervsed Neural Network, Adaptve Resonance Theory. I. INTRODUCTIO N Pattern recognton s one of technques that place obects to classes or groups. The pattern recognton s called supervsed pattern recognton when there are tranng data wth labels n each formed classes. But, there are many cases wth no tranng data. For that condton, the man obectves of the pattern recognton s how to calculate the smlartes among obects and then group the smlar obects to same cluster. Ths process s called unsupervsed pattern recognton or clusterng. In clusterng, there are no predefned classes and no examples. Obects are grouped based on self-smlarty. From ths practce perspectves, clusterng play a mportant part n data mnng applcatons, such as scentfc data exploratons, extractng nformaton, mnng, spatal databases applcatons, Web analyss, CRM, marketng, medcal dagnoss, and bology computatons. Many clusterng algorthms have been developed, such as K-means [], densty-based parttonng, grd based, and unsupervsed neural network []. Unsupervsed neural networks (UNN) have ablty to cluster multvarate data. The number of clusters s decded by UNN and drectly depends on data nputs (Clare dan Cohen, 00). SOM ( Self-Organzng Maps) s an UNN technque that commonly used. Another technque, Adaptve Resonance Theory (ART) has some types, such as Adaptve Resonance Theory (ART ) that s desgned to bnary nputs, and Adaptve Resonance Theory (ART ) that s desgned to contnuous (numerc) values [3]. In ths paper, ART s chosen, because ths algorthm can accept nput n numerc form that s usually used many data. Changng of data symbols to numerc form s also smpler than to another form, such as bnary form. Besdes that, Fausett [3] states that ART was desgned to stable networks. II. Preprocessng data s predecessor step before clusterng. Ths step s needed to form or arrange nput n order that can be used to next steps [4]. In preprocessng, t s needed to convert the data after got the data, because the data may be stll has varous formats or types, such as texts or symbolc data. Otherwse, n many neural networks, the nput should be n certan type, such as bnary type or contnuous type., ka f Nf = f, ka Maxf PREPROCESSING DATA The steps of convertng symbolc or text data to numerc data are () an nteger code s assgned for each symbol; () and then, change them to be bnary or contnuous data. To normalzng the contnuous data, we need to smple analyzng the data,.e., calculate the number of varaton for each attrbutes, and then defne the maxmal values for each feature. After that, by usng the maxmum values and smple equaton, normalze the values of each features. () > Maxf f Maxf Where F, f, Maxf and Nf s feature / feld, features values, Maxmum value for F that can be receved and normalzed values of F, respectvely. III. IN SUPERVISED NEURAL NETWORK (UNN) Artfcal Neural Networks s a system that process nformaton lke human neuron characterstcs. Artfcal neural network was developed as a mathematc model generalzaton of neural bology based on assumpton:. Informaton processng s done to many smple elements, that s called neuron.. Sgnal s send between the neurons by connecton lnk.

2 3 rd Internatonal G raduate Conference on Engneerng, Scence and Humantes (IGCESH) School of Graduate Studes Unverst Teknolog Malaysa 4 November Each the connecton lnk has sutable weght that wll strengthen or weaken the sgnal. 4. Each neuron use an actvaton functon (usually t s non lner) that s operated to sum of the nputs that s receved to decdng ther output sgnals. And then the output sgnal wll be compared to a lmt threshold. An artfcal neural network s decded by three terms [3]:. Connecton patterns between the neurons, t s called network archtecture,. Methods to decde the connecton s weght, t s called learnng algorthms, 3. Actvaton functon. The method to decde the connectons weght s one of specal characterstc from varous neural networks. Generally, such learnng dvde two types,.e., supervsed and unsupervsed [3]. Typcal neural network wth supervsed learnng s often used. Ths network learns used many vectors or patterns, and each vectors s connected to sutable output vectors. Ths process s often known as supervsed learnng/tranng [3]. Whereas the neural network type wth unsupervsed learnng s a self-organzng network. Ths network can group nput vectors that have smlartes wthout usng tranng data to decde what knds of the me mbers n one group, or whch output vectors that an nput should onts. The neural network then modfed weghts so that the most smlar nputs wll be n same output unt (cluster). Ths neural network also produce example vector for each clusters that produced. The networks that can be categorzed as unsupervsed neural network are Kohenen Network (or t s often called Self-Organzng Map ( SOM), and Adaptve Resonance Theory [3]. IV. ADAPTIVE RESONANCE THEORY (ART-) One of type of ART s ART, whch s specal desgned for clusterng of bnary vectors. Whereas another types, s ART, receved vector nput by contnuous values. The network s clusterng nput based on unsupervsed learnng. When each a pattern (n vector format) s represented, a sutable cluster wll be chosen and ts weght wll be adapted untl the cluster unt can earn pattern that was nputted before. The weght of a cluster unt s represented n vector code form or exemplar form for patterns n that cluster. F layer consst of 6 unt type (.e.: W, X, U, V, P, and Q). There are n unts for each the unt types, where n s dmenson of an nput pattern. There s only one extra unt for each unt types, whch s shown n fgure. An extra unt between W unt and X unt receves sgnals from all of W unts and sends the sgnals to all of X unts. Another extra unt between P unt and Q unt has same task. As s an extra unt before, an extra unt between unt V and U unt has same task. Both X unt and Q unt connect to V unt. The symbols of connecton path between the unts n F layers n fgure, ndcate sgnal transformaton from a type unt to next type unt. But, connecton between P unt n F layer and Y n F layer represent weght that multply transmtted sgnals by the path. F unt actvaton that wn, s d, where 0< d<. The symbol ---> ndcate normalzaton (Fausett, 994). The learnng algorthms use some facts,.e.:. Reset cannot happen durng resonance (step 8). A new wnner unt cannot be chosen whle resonance. But, ths algorthm dd not use yet facts that:. Usually, n slow learnng use N_IT and step 0th can be passed away.. In fast learnng, for fsrt pattern that be kearned by a cluster, u wll pararel wth t by learnng cycles and equlbrum weght wll be: t J = u ; bj = u. d d (4) Many another mpossble stop condtons are:. Repeat step 8 untl weght change under certan tolerance. For slow learnng, repeat step untl lm of certan tolerance. Whereas for fast learnng, repea step untl placng the patterns of cluster unt do not change from one epoch to next epoch.. Step 3 untl form a learnng tral that s a representaton of a pattern). It s better to reference performance of a learnng tral for each nput pattern as a pattern. The pattern s not needed to be presente wth same order for each epoch. Update actvaton F durng resonance phase n phase learnng mode cause value u, become dfferent durng learnng s gong on (Fausett, 994). Fgure. Typcal archtecture o f ART [3]

3 3 rd Internatonal G raduate Conference on Engneerng, Scence and Humantes (IGCESH) School of Graduate Studes Unverst Teknolog Malaysa 4 November 00 Step 0. Intates parameters: a,b,?,c,d,e,a,? Step. Do step - as many as N_EP Step. For each nput vectors s, do step 3-. Step 3. Update unt actvaton F : s u = 0, x e + s w = s, q p = 0, = 0, v = f ( x Update actvaton F agan: v u w = s + au, e + v p = u, w x e + w p q = v = f ( x ) + bf ( q e + p Step 4. calculate sgnal for F unt y = b Step 5. f reset=true, do step 6-7 Step 6. fnd F unt ( Y J ) wth bgst sgnal. (decde J untl y J = y for =,...,m.) Step 7. Check reset condton: p v u =. p = u + dt, J e + v u + cp r = e + u + c p If r < r e, theny J = - ( nhbt J) (reset=true; back to step 5); If r r e, then w w = s + au, x e + w p q =, v = f ( x ) + bf ( q e + p Reset =false; go to step 8. Step 8. Do step 9- as many as N_IT Step 9. Update weght for wnner unt J: t J = a d u + { + ad (d-) } t J, b J = a d u + { + ad (d- ) } b J, Step 0. Update actvaton F Fgure. Learnng Algorthm of ART The cluster valdaton technque can be dvded three types,.e., unsupervsed, supervsed, and relatve. Unsupervsed cluster valdaton technque measures as well as clusterng structures wthout see external nformaton aspects. One of the technques s Sum Squared Error (SSE). Unsupervsed measurements usually are dvded two classes,.e., cluster coheson that decd how close the obects n same cluster, and cluster separaton that decde how separate among the clusters [6]. A. Sum Square Error (SSE) k = x C dst( c, x) (5) whch k s the number of clusters, x s an obect n cluster, C s cluster-, c s C s centrod, and dst s Eucldean dstance among two obects [6]. B. Coheson and Separaton Cluster valdaton of a set wth K cluster, can be defned as a sum of weghted ndvdual cluster valdaton, overall valdty = V. CLUSTER VALIDATION One of type of ART s ART, whch s specal desgned for clusterng of bnary vectors. Whereas another types, s ART, receved vector nput by contnuous values. The network s clusterng nput based on unsupervsed learnng. When each a pattern (n vector format) s represented, a sutable cluster wll be chosen and ts weght wll be adapted untl the cluster unt can earn pattern that was nputted before. The weght of a clus er unt s represented n vector code form or exemplar form for patterns n that cluster. The man clusterng problem s how to decde the number of optmal cluster that sutable to the data set. Sometmes the clusters results are not represented to the real data structure. Hence, the quanttatve measures are needed to evaluate the result of clusterng algorthms need, next we called cluster valdaton [5]. SSE calculates error of each pont of data, then compute total of error square. Errors n ths context can be a Eucldean dstance to closest centrod. Formally, SSE can be defned: SSE = K = w valdty( C ) (6) where w s weght of cluster-, that depend of the cluster valdaton measurement technque. Valdaton functons can be coheson, separaton, or combnaton of coheson and separaton [6]. Coheson of the cluster can be defned as the sum of closeness of the cluster to the centrod. Separaton between two clusters can be measured by the closeness of the cluster prototype and the centrod. The llustraton can be shown n fgure 3 whch the cluster centrod s assgned by +. 3

4 3 rd Internatonal G raduate Conference on Engneerng, Scence and Humantes (IGCESH) School of Graduate Studes Unverst Teknolog Malaysa 4 November 00 Learnng algorthm of the artfcal neural network used learnng algorthm ART. Ths experment used slow learnng and fast learnng. (a) Coheson. (b) Separaton Fgure 3. Coheson and separaton o f the clusters Equaton 7 represents how to defne the coheson, and equatons 8 and 9 represent how to calculate the separaton. In the equaton, c s centrod of cluster C and c s centrod of all the obects. Equaton 7 s SSE of the the cluster, f Eucldean dstance s a proxmty functon [6]. coheson( C ) = proxmty( x, c x C separaton( C, C ) = proxmty( c, c ) separaton( C ) = proxmty( c, c) (9) (7) (8) ) x C TABLE I. Cluster measurements proxmty c, c) THE M EASUREM ENT OF C LUST ER VALIDAT ION Cluster weghts Types proxmty( x, c ) Prototypebased Coheson ( m (umlah obek dalam cluster ke) Prototypebased Separaton Coheson and separaton can be combned to a measurement of all cluster valdaton by usng the weghted sum, equaton 6. But, what weght that used need to be decded. There are so many weghts that can be used. [6] suggest two types of weght that can be used for valdaton purpose (shown n Table ). If the proxmty s measured by Eucldean dstances, the smple measurement of separaton between clusters s sum of squares between groups (SSB) that s sum of dstance square of a cluster centrod, c, wth mean all of ponts of data ( c). The total of SSB from all cluster defned as equaton 0, where c s mean of cluster- and c s mean of all clusters. The mportant thng of the results s how to mnmze the SSE (coheson) and or how to maxmze the SSB (separaton). (0) Total VI. SSB = K = m dst( c, c) THE ART- IMPLEMENTATION In ths paper, the ART was mplemented to cluster the daly medcal records. Next, the cluster result s valdated wth approprate measurement. The man process n ths mplementaton s clusterng. Whereas preprocessng process s used to study case data preparaton before clusterng process. A. Artfcal Neural Network Model Artfcal Neural Network ART archtecture can be shown n fgure 4. n ths paper, each unt types (P,Q,U,V,W,X) n nput layers ( F ) maxmal s 0 unts. Whereas layers F, that s cluster layers, the number of y unt s flexble. 4 Fgure 4. Artfcal Neural Network ART archtecture Desgn B. Clusterng In ths process, there are two steps of learnng used UNN ART and cluster valdaton. Clusterng proses s done many tmes agree wth Max_m and Mn_m parameters. It s done many tmes because to get the best clusterng result, that has optmal SSE total and SSB total. VII. ANALYSIS OF EXPERIMENTAL RESULT Ths study utlzes the daly medcal record of Local Government Clnc. The data has some attrbutes, but n ths experment only use fve attrbutes nclusng Dates, Address, Age, Sex, and medcal dagnoss. From the experment results, has shown that f the clusters that have smallest total SSE then they have bggest total SSB too. Besdes that, all of the experments have shown that the best clusterng begns n certan m values untl upper lmt of the range, and also n the same clusterng results. VIII. CONC LUSION Unsupervsed neural network ART (ART-) algorthm has been mplemented. From the expermental result, the algorthm acheved clusters wth smallest total SSE and bggest total SSE. The proposed ART

5 3 rd Internatonal G raduate Conference on Engneerng, Scence and Humantes (IGCESH) School of Graduate Studes Unverst Teknolog Malaysa 4 November 00 algorthms and valdaton technques scale well and ga n consderable performance due to the resulted cluster. 3 v v - - v 3 4 v v v - v 4 5 v v v v v TABLE III. THE RESULT OF THE EXPERIMENTS Experment Attrbutes Ranges Dagnose Date, Dagnose 3 Date, Age, Dagnose 4 Date, Age, S ex, Dagnose Best Result (Based on Total SSE and Total SSB) m w th smallest SSE T he num ber of cluster m wth bggest SS B T he num ber of cluster ACKNO WLEDGMENT (HEADING 5) Ths work s supported by a research grant from Mnstry of Natonal Educaton (Mendknas). The authors gratefully acknowledge many helpful comments by revewers n mprovng the publcaton. REFERENCES TABLE II. Fgure 5. Flowchart of clusterng process. THE EXPERIM ENT S AN D ATRIBUTES COMBINATONS Experment D menson The Atrb utes Date Address Age Sex Dagno sa - v - - v [] T.K. Anderson, Kernel densty estmato n and K-means clusterng to profle road accdent hotspots, Accdent Analyss and Prevento n, Vol. 4/3, pp , 009. [] P. Berkhn, Survey o f Clusterng Data Mnng Technques, Accrue Software, Inc., San Jose, Calforna, 00. [3] L. V. Fausett, Fundamentals of Neural Networks: Archtectures, Algorthms, and Applcatons, Prentce-Hall, 994. [4] A. Nachev and I. G anchev, Data Mnng for Browsng Patterns n Weblo g Data by ART Neural Networks, Internatonal Journal of Informaton Theores & Applcatons, Vol. 0. [5] M. Halk d, Y. Batstak s, et al.. "O n clusterng valdaton technq ues", Journal of Intellgent Informaton Systems, Vol 7/, pp , 00. [6] P.N. Tan, M. Stenbach, et al, Introducton to data mnng, Addso n Wesley,