Learning Semantics-Preserving Distance Metrics for Clustering Graphical Data

Transcription

1 Learnng Semantcs-Preservng Dstance Metrcs for Clusterng Graphcal Data Aparna S. Varde, Elke A. Rundenstener Carolna Ruz Mohammed Manruzzaman,3 Rchard D. Ssson Jr.,3 Department of Computer Scence Center for Heat Treatng Excellence 3 Department of Mechancal Engneerng Worcester Polytechnc Insttute (WPI) Worcester, MA USA (aparna rundenst ruz manruzz ssson)@wp.edu ABSTRACT In mnng graphcal data the default Eucldean dstance s often used as a noton of smlarty. However ths does not adequately capture semantcs n our targeted domans, havng graphcal representatons depctng results of scentfc experments. It s seldom known a-pror what other dstance metrc best preserves semantcs. Ths motvates the need to learn such a metrc. A technque called LearnMet s proposed here to learn a domanspecfc dstance metrc for graphcal representatons. Input to LearnMet s a tranng set of correct clusters of such graphs. LearnMet teratvely compares these correct clusters wth those obtaned from an arbtrary but fxed clusterng algorthm. In the frst teraton a guessed metrc s used for clusterng. Ths metrc s then refned usng the error between the obtaned and correct clusters untl the error s below a gven threshold. LearnMet s evaluated rgorously n the Heat Treatng doman whch motvated ths research. Clusters obtaned usng the learned metrc and clusters obtaned usng Eucldean dstance are both compared aganst the correct clusters over a separate test set. Our results show that the learned metrc provdes better clusters. Categores and Subject Descrptors I..6 [Artfcal Intellgence]: Learnng Parameter learnng. General Terms Algorthms, Expermentaton. Keywords Dstance metrc, clusterng, semantc graphcal mnng.. INTRODUCTION Expermental results n scentfc domans are often represented graphcally. Such graphs depct the functonal behavor of process parameters hence ncorporatng semantcs. In ths paper, we use the term graph to mean such a graphcal representaton. In mnng graphs wth technques such as clusterng [9] the measure for comparson s typcally Eucldean dstance [see e.g. 7] whch often poses problems. For example, n the doman of Heat Treatng of Materals whch motvated ths work, graphs are used to plot the heat transfer versus temperature durng the rapd coolng of a materal [see e.g. 3]. Crtcal regons on these graphs represent sgnfcant physcal phenomena n the doman. Any algorthm (based for nstance on Eucldean dstance) that consders two graphs as smlar (relatve to other graphs) although ther crtcal regons dffer s consdered semantcally ncorrect [3,4]. Lkewse n several scentfc domans, there could be sgnfcant features on graphs. Knowledge of these features and ther relatve mportance may at best be avalable n a subjectve form, but not as a metrc. Ths motvates the development of a technque to learn dstance metrcs that capture the semantcs of the graphs. Hnneburg et al [6] propose a learnng method to fnd the relatve mportance of dmensons for n-dmensonal objects. However, ther focus s on dmensonalty reducton and not on doman semantcs. In [6] they learn whch type of poston-based dstance s applcable for the gven data startng from the formula of Mahalanobs dstance. However they do not deal wth graphcal data and semantcs. Kem et al. [8] overvew varous dstance types for smlarty search over multmeda databases. However no sngle dstance measure encompassng several types s proposed. Lnear regresson [] and neural networks [] could possbly be used for learnng a doman-specfc dstance metrc for graphs. However these technques do not acheve accuracy acceptable n our targeted domans [4]. Genetc algorthms [5] f used for feature selecton n graphs also gve the problem of nsuffcent accuracy. Ths s due to lack of doman knowledge [4]. Fourer transforms [4] f used to represent the graphs do not preserve the crtcal regons n the doman due to the nature of the transform [4]. Hence they are not accurate enough n capturng semantcs. Accuracy n ths context s measured by evaluatng the effectveness of a gven metrc n mnng unseen data [4]. We propose an approach called LearnMet to learn a dstance metrc for graphs ncorporatng doman semantcs. The nput to LearnMet s a tranng set wth correct (.e., gven by doman experts) clusters of graphs over a subset of the expermental data. The steps of our approach are: () guess ntal metrc guded by doman knowledge; () use that metrc for clusterng wth an arbtrary but fxed clusterng algorthm; (3) evaluate accuracy of obtaned clusters by comparng them wth correct clusters; (4) adjust metrc based on error between obtaned and correct clusters, f error s below threshold or f executon tmes out then termnate and go to step (5), else go to step (); and (5) once

2 termnated, output the metrc gvng error below threshold, or mnmum error so far as the learned metrc. LearnMet s evaluated usng a dstnct test set of correct clusters of graphs provded by experts. The learned metrc s used to cluster the graphs n the test set. The obtaned clusters are compared wth correct clusters n the test set. The closer the obtaned clusters match the correct clusters, the lower the error. The clusters obtaned usng the default noton of Eucldean dstance, are also compared wth the correct clusters. The dfference n the error denotes the effectveness of the learned metrc. The LearnMet metrc consstently outperformed the Eucldean metrc n our experments. LearnMet s also evaluated by ntegratng t wth the AutoDomanMne system desgned for computatonal estmaton of process parameters n scentfc experments [3]. Ths s explaned n Secton 4.. DISTANCE METRICS We descrbe dstance metrc types relevant to our domans. Let A (A,A...A n ) and B (B,B...B n ) be two n-dmensonal objects. Poston-based Dstances: They refer to absolute poston of objects [see e.g. 7]. Examples are: Eucldean Dstance: As-the-crow-fles dstance between objects. D Eucldean = n = ( A B ) Manhattan Dstance: Cty-block dstance between objects. n DManhat tan = A B = Statstcal Dstances: Ths refers to dstances based on statstcal features [see e.g. ]. Examples are: Mean Dstance: Dstance between mean values of the objects: D Mean = Mean( A) Mean( Maxmum Dstance: Dstance between maxmum values: D Max = Max( A) Max( In addton to these, we ntroduce the concept of Crtcal Dstance for graphs as applcable to our targeted domans [4]. Crtcal Dstances: Gven graphs A and B, a Crtcal Dstance s a dstance metrc between crtcal regons of A and B where a crtcal regon represents the occurrence of a sgnfcant physcal phenomenon. They are calculated n a doman-specfc manner. Fgure : Crtcal Dstance Examples One example of crtcal dstance as shown n Fgure s the Ledenfrost dstance [3,4] n Heat Treatng. Ths s for Ledenfrost Ponts [see e.g. 3] whch denote the breakng of the vapor blanket around a part n heat treatment. It s calculated as D LF = ( ATLF BTLF ) + ( AhLF BhLF ) where T LF s the temperature at Ledenfrost Pont and h LF s the heat transfer coeffcent at that pont [0]. Another crtcal dstance s the Bolng Pont dstance D BP [3,4]. Ths s the dstance between ponts on the graphs correspondng to the Bolng Ponts [see e.g. 3] of the respectve coolng meda n heat treatment. D temperature and h BP s heat transfer coeffcent at Bolng Pont. BP = ( ATBP BTBP ) + ( AhBP BhBP ) where T BP s 3. THE LEARNMET STRATEGY To gve detals of LearnMet, a dstance metrc s frst defned. Dstance Metrc n LearnMet: A LearnMet dstance metrc D s a weghted sum of components, where each component can be a poston-based, a statstcal, or a crtcal dstance metrc. The weght of each component s a numercal value ndcatng ts relatve mportance n the doman. Thus a LearnMet dstance metrc s of the form D = w Dc w Dc where each Dc m m s a component, w s ts weght, and m s number of components. As requred n our applcaton doman, D should be a metrc so that clusterng algorthms requrng the noton of smlarty to be a dstance metrc can be used; ndexng structures such as B+ trees for metrcs can be appled; and prunng n smlarty search can be performed usng trangle nequalty. Condtons for D to be a metrc are stated as a theorem below. Theorem : If each component Dc s a dstance metrc and each weght 0 m w then D = = w Dc s a dstance metrc,.e., t satsfes the metrc propertes. The proof of ths theorem s straghtforward and can be found n [4]. In our targeted domans, condtons n Theorem are satsfed. Snce our graphs nvolve nterval-scaled varables and the fundamental dstance types applcable to these are metrcs [see e.g. 7], ths s suffcent to say that each Dc s a metrc. Also, we consder only non-negatve weghts. The LearnMet steps are dscussed n the subsectons below. 3. Intal Metrc Step Doman experts are asked to dentfy components (.e., dstance metrcs) applcable to the graphs that wll serve as buldng blocks for the learnng of a new metrc. If the experts know the relatve mportance of the components, ths nformaton s used to assgn ntal weghts. An Intal Weght Heurstc s proposed. Intal Weght Heurstc: Assgn ntal weghts to the components n the LearnMet dstance metrc based on the relatve mportance of the components on the graphs n the doman.

3 If the components relatve mportance s unknown, random weghts are assgned. Intal weghts are on a scale of to 0. : INITIAL METRIC STEP GIVEN: DOMAIN EXPERT INPUT ON DISTANCE TYPES FOR EACH DISTANCE TYPE ASSIGN A COMPONENT TO D IF RELATIVE IMPORTANCE OF COMPONENTS IS AVAILABLE THEN USE INITIAL WEIGHT HEURISTIC ELSE ASSIGN RANDOM WEIGHTS TO COMPONENTS 3. Clusterng Step Usng D as the dstance metrc, k clusters are constructed usng an arbtrary but fxed clusterng algorthm (e.g., k-means [0]), where k s the number of clusters n the tranng set. : CLUSTERING STEP GIVEN: A TRAINING SET CONSISTING OF A COLLECTION OF GRAPHS AND A CORRECT k-clustering OF THEM SELECT AN ARBITRARY BUT FIXED CLUSTERING ALGORITHM SET NUMBER OF CLUSTERS TO k ( CONSTANT) CLUSTER GRAPHS USING D = wdc w m Dc m 3.3 Cluster Evaluaton Step The clusters obtaned from the algorthm,.e., the predcted clusters, are evaluated by comparng them wth correct clusters n the tranng set,.e., the actual clusters. An example of predcted and actual clusters s shown n Fgure. Ideally, the predcted clusters should match the actual clusters. Any dfference between predcted and actual clusters s consdered an error. To compute ths error, we consder pars of graphs and ntroduce the followng notaton. True/False Postve/Negatve Pars of Graphs: Gven a par of graphs I and J, we say that: (I,J) s a True Postve (TP) par f I and J are n the same actual cluster and n the same predcted cluster. (I,J) s a True Negatve (TN) par f I and J are n dfferent actual clusters and n dfferent predcted clusters. (I,J) s a False Postve (FP) par f I and J are n dfferent actual clusters but n the same predcted cluster. (I,J) s a False Negatve (FN) par f I and J are n the same actual cluster but n dfferent predcted clusters. Fgure ncludes examples of each of these knds of pars: (g,g ) s a true postve par; (g,g 3 ) s a true negatve par; (g 3,g 4 ) s a false postve par; and (g 4,g 6 ) s a false negatve par. The error measure of nterest to us s falure rate whch s defned below. Success and s: Let TP, TN, FP and FN denote the number of true postve, true negatve, false postve and false negatve pars respectvely. Also let SR denote the Success Rate and FR = ( SR) denote the, as defned below: TP + TN SR = TP + FP + TN + FN FP + FN FR = TP + FP + TN + FN Fgure : Predcted and Actual Clusters In our doman, false postves and false negatves are equally undesrable. Hence, our defnton of falure rate weghs them equally. Gven a number G of graphs n the tranng set, the total number of pars of graphs s G C = G! (!( G )! ). Thus, for 5 graphs there are 300 pars, for 50 graphs, 5 pars, etc. We defne an epoch n LearnMet as one run of all ts steps. That s, a complete tranng cycle. Overfttng: To avod overfttng n LearnMet, we use an approach analogous to ncremental gradent descent [, 4]. Instead of usng all pars of graphs for evaluaton, a subset of pars s used called ppe or pars per epoch. In each epoch, a dstnct combnaton of pars s used for evaluaton and weght adjustments. Thus there s enough randomzaton n every epoch If ppe = 5, then we have a total of C5 =.95 0 dstnct pars for learnng [, 4]. Thus n each epoch 5 dstnct pars can be used. Ths stll gves a large number of epochs wth dstnct pars for learnng. Ths ncremental approach reduces the tme complexty of the algorthm and helps avod overfttng. Determnng the best ppe value s an optmzaton problem. Also n LearnMet, the random seed s altered n the clusterng algorthm n dfferent epochs as an addtonal method to avod overfttng. Ideally, the error.e., falure rate n an epoch should be zero. However, n practce a doman-specfc error threshold t s used. Error Threshold: A doman-specfc error threshold t s the extent of error allowed per epoch n the doman, where error s measured by falure rate. Dstance between a Par of Graphs: The dstance D(g a,g b ) between a par of graphs g a and g b s the weghted sum of components n the graphs usng metrc D. Thus, D g, g ) = w Dc ( g, g ) w Dc ( g, g ) ( a b a b m m a b Gven ths, consder FN pars, e.g., (g 4,g 5 ) and (g 4,g 6 ). These pars are n the same actual cluster. However they are predcted to be n dfferent clusters. Snce predcted clusters are obtaned wth the metrc D, the (average) dstance D(g a,g b ) for these pars s greater than t should be. Conversely, for FP pars n dfferent actual, same predcted clusters, e.g., (g 3,g 4 ), the (average) dstance D(g a,g b ) s smaller than t should be.

4 Average FN and FP Dstances D_FN and D_FP: FN D _ FN = (/ FN ) j = D _ FP = (/ FP) FP j = D ( g g a, b ) where (g a,g b ) denotes FN pars. D( g a, g b ) where (g a,g b ) denotes FP pars. 3: CLUSTER EVALUATION STEP SET ppe TO THE DESIRED NUMBER OF PAIRS OF GRAPHS FROM THE TRAINING DATASET TO BE CONSIDERED IN AN EPOCH RANDOMLY SELECT ppe PAIRS OF GRAPHS SET THE ERROR THRESHOLD t IDENTIFY THE TP, TN, FP, FN FROM ppe PAIRS CALCULATE FAILURE RATE FR IF (FR < t) THEN RETURN CLUSTERING IS ACCURATE ELSE CALCULATE D_FN,, D_FP 3.4 Weght Adjustment Step If the result of the evaluaton does not ndcate that the clusterng s accurate, then the dstances D_FN and D_FP are used to make weght adjustments to reduce the error n clusterng. Consder the error n FN pars. To reduce the average error D_FN, the weghts of one or more components n the metrc used to calculate the dstance n the present epoch s decreased. For ths we propose the FN Heurstc. FN Heurstc: Decrease the weghts of the components n the metrc D n proporton to ther contrbutons to the dstance D_FN. That s, for each component: D _ FNc w ' = w where D _ FNc = D _ FN for Dc alone D _ FN Conversely, consder FP pars. To reduce ther error, we ncrease D_FP. Ths s done by ncreasng the weghts of one or more components n the metrc usng the FP Heurstc. FP Heurstc: Increase the weghts of the components n the metrc D n proporton to ther contrbutons to the dstance D_FP. That s, for each component: D _ FPc w '' = w + where D _ FPc = D _ FP for Dc alone D _ FP Combnng these two adjustments: Weght Adjustment Heurstc: D _ FNc w ''' = max(0, w ( ) D _ FN + D _ FPc ( )) D _ FP Thus D ''' = w ''' Dc w m ''' Dcm The new metrc D obtaned after weght adjustments s lkely to mnmze the error due to the FN and FP type pars. If the weght of a component becomes negatve, t s converted to zero as we consder only non-negatve weghts. Clusterng s done wth ths new metrc. If the resultng error s below the threshold, then a confrmatory test usng the same metrc to cluster for more epochs s performed, and then ths step s complete. 4: WEIGHT ADJUSTMENT STEP IF CLUSTERING IS ACCURATE OR MAX EPOCHS REACHED THEN GO TO 5: FINAL METRIC STEP ELSE APPLY WEIGHT ADJUSTMENT HEURISTIC TO GET D GO TO : CLUSTERING STEP 3.5 Fnal Metrc Step If the weght adjustment termnates because the error s below the threshold then the metrc n the last epoch s consdered accurate and t s returned as output. However f termnaton occurs because the maxmum number of epochs s reached, then the most reasonable metrc to be output s the one correspondng to the epoch wth the mnmum error among all epochs. 5: FINAL METRIC STEP IF (FR < t) THEN RETURN METRIC D ELSE FIND EPOCH WITH MINIMUM RETURN CORRESPONDING METRIC D Convergence: LearnMet s not guaranteed to converge or to yeld an optmal dstance metrc. However, thorough expermental evaluaton n our applcaton doman has shown consstent convergence to errors below the requred threshold. 4. EXPERIMENTAL EVALUATION 4. Evaluaton of LearnMet. Expermental Parameters: A tranng set of 300 pars of graphs n Heat Treatng s obtaned from correct clusters of 5 graphs gven by experts. A dstnct test set of 300 pars of graphs s derved from 5 graphs gven by experts. We select ppe = whch yelds C 5 =.95 0 total dstnct combnatons of pars. Thus 5 dstnct pars are used n each epoch. Experts gve an error threshold of 0%,.e., 0. for estmaton. We use the same threshold for clusterng. Intal components n the metrc are gven by experts. Two dstnct assgnments of ntal weghts are gven by two dfferent experts [4]. The correspondng two metrcs are denoted by DE and DE respectvely. A thrd ntal metrc EQU s obtaned by assgnng equal weghts to all components. Several experments are run by assgnng random weghts to components n the ntal metrc [4]. We present two experments wth randomly generated metrcs called RND and RND. See Table. Table : Intal Metrcs n LearnMet Experments Table : Learned Metrcs and Number to Learn

5 Observatons durng Tranng: Table shows the metrc learned n each experment wth number of epochs. Fgures 3 to 7 depct the behavor of LearnMet durng tranng. Experments EQU, RND and RND take longer to converge than DE and DE. However they all converge to approxmately the same D Error over Tranng Set Fgure 3: Experment DE Error over Tranng Set Observatons durng Testng: The learned metrc n each experment s used to cluster graphs n the test set. These clusters are compared wth correct clusters over the test set. Eucldean dstance (ED) s also used to cluster the graphs and the clusters are compared wth the correct clusters. The observatons for all the experments are shown n Fgure 8. Ths fgure depcts the accuracy (success rate) for each metrc over the test set. Clusterng accuraces of the learned metrcs are hgher. Metrc ED RND RND EQU DE DE Clusterng Accuracy over Test Set Accuracy (Success Rate) Accuracy Fgure 4: Experment DE Error over Tranng Set Fgure 5: Experment EQU Fgure 8: Test Set Observatons 4. Evaluaton wth System Integraton Purpose of Integraton: LearnMet has been developed manly for the AutoDomanMne system, whch performs computatonal estmaton of process parameters [3]. AutoDomanMne estmates the graphcal result of a scentfc experment gven ts nput condtons, usng exstng data to predct future trends. Ths technque clusters graphs from exstng experments and sends clusterng output to a decson tree classfer e.g., ID3/J4.8 [,5] to learn the clusterng crtera. For each graph, the nput condtons of ts correspondng experment and the cluster n whch t was placed are used to construct the decson tree. The tree dentfes the combnaton of nput condtons that characterze each cluster. A representatve graph s also selected per cluster. When nput condtons of a new experment are submtted, the tree s traversed to predct the cluster. The representatve graph of that cluster s the estmated graph for that experment. Error over Tranng Set Fgure 6: Experment RND Error over Tranng Set Fgure 9a: Evaluaton wth AutoDomanMne Step Fgure 7: Experment RND

6 Fgure 9b: Evaluaton wth AutoDomanMne Step Evaluatng LearnMet wth AutoDomanMne: LearnMet s evaluated by measurng the accuracy of the AutoDomanMne estmaton wth and wthout the learned metrcs. Ths process s llustrated n Fgures 9a and 9b. The estmaton obtaned from clusterng usng the learned metrcs s compared wth that from clusterng usng Eucldean dstance. Another crteron for comparson s D = DEucldean + DLF + D whch s called the BP AutoDomanMne metrc denoted as ADM [3,4]. Observatons wth AutoDomanMne: The average estmaton accuracy over 0 experments, each usng 0-fold cross valdaton, s shown n Fgure 0. Accuracy wth each metrc output from LearnMet s hgher than that wth Eucldean dstance. Accuraces wth the learned metrcs are also hgher than the accuracy wth the AutoDomanMne metrc. 9% 8% 7% 6% 5% 4% 3% % % % Estmaton Accuracy wth Dfferent Metrcs used n Clusterng ED ADM DE DE EQU RND RND Average Estmaton Accuracy Fgure 0: Evaluaton Results wth AutoDomanMne 5. CONCLUSIONS A technque called LearnMet s proposed to learn a domanspecfc dstance metrc for mnng graphs. LearnMet compares clusters of graphs obtaned from a state-of-the-art clusterng algorthm wth correct clusters gven by experts. An ntal metrc s guessed and refned wth every round of clusterng to gve a fnal metrc wth error below a threshold. LearnMet s evaluated rgorously n the Heat Treatng doman that nspred ts development. Addtonal evaluatons wll be conducted n related domans. Ongong research ncludes determnng a good number of pars per epoch; usng normalzed weghts; consderng scalng factors n weght adjustments; refnng thresholds; and assgnng sutable components to the ntal metrc wthout expert nput. ACKNOWLEDGMENTS Ths work s supported by the Center for Heat Treatng Excellence and by DOE ITP Award DE-FC-07-0ID497. REFERENCES [] Anscombe, F.J. Graphs n Statstcal Analyss, n Amercan Statstcan, 7, 7-, 973. [] Bshop, C.M. Neural Networks for Pattern Recognton, Oxford Unversty Press, England, 996. [3] Boyer, H., and Cary, P. Quenchng and Control of Dstorton, ASM Internatonal, OH, USA, 989. [4] Fourer, J. The Analytcal Theory of Heat (translated by A. Freeman) Dover Publcatons Inc., NY, USA, 955. [5] Fredberg, R.M. A Learnng Machne: Part I, n IBM Journal,, -3, 958. [6] Hnneburg, A., Aggarwal, C., and Kem, D. What s the Nearest Neghbor n Hgh Dmensonal Spaces, n Proceedngs of VLDB Caro, Egypt. August 000. [7] Han, J., and Kamber, M. Data Mnng: Concepts and Technques, Morgan Kaufmann Publshers, CA, USA. 00. [8] Kem, D., and Bustos, B. Smlarty Search n Multmeda Databases, n Proceedngs of ICDE Boston, MA, USA, March 004. [9] Kaufman, L., and Rousseeuw, P. Fndng Groups n Data: An Introducton to Cluster Analyss, John Wley, NY, USA [0] MacQueen, J. B. Some Methods for Classfcaton and Analyss of Multvarate Observatons, n Proceedngs of 5-th Berkeley Symposum on Mathematcal Statstcs and Probablty. :8-97. Berkeley, CA, USA, 967. [] Petrucell, J., Nandram, B., and Chen, M., Appled Statstcs for Engneers and Scentsts, Prentce Hall, NJ, USA, 999. [] Qunlan, J. R. Inducton of Decson Trees, n Machne Learnng, : [3] Varde, A., Rundenstener, E., Ruz, C., Manruzzaman, M., and Ssson Jr., R. Data Mnng over Graphcal Results of Experments wth Doman Semantcs, n Proceedngs of ACM SIGART's nd Internatonal Conference on Intellgent Computng and Informaton Systems ICICIS Caro, Egypt, March 005. [4] Varde, A., Rundenstener, E., Ruz, C., Manruzzaman, M., and Ssson, Jr., R. The LearnMet Approach for Learnng a Doman-Specfc Dstance Metrc to preserve the Semantc Content n Mnng Graphcal Data, TR#: CS-CHTE , WPI, Worcester, MA, USA, June 005. [5] Wtten, I., and Frank, E., Data Mnng: Practcal Machne Learnng Algorthms wth Java Implementatons, Morgan Kaufmann Publshers, CA, USA, 000. [6] Xng, E., Ng., A., Jordan, M., and Russell, S. Dstance Metrc Learnng wth Applcaton to Clusterng wth Sde Informaton, n Proceedngs of NIPS-03, Vancouver, Canada, December 003.