Pattern Extraction, Classification and Comparison Between Attribute Selection Measures

Transcription

1 Subrata Pramak et al. / (IJCSIT) Iteratoal Joural of Computer Scece ad Iformato Techoes, Vol. 1 (5), 010, Patter Extracto, Classfcato ad Comparso Betwee ttrbute Selecto Measures Subrata Pramak, Md. Rashedul Islam, Md. Jamal Udd Departmet of Computer Scece & Egeerg, Uversty of Rajshah Rajshah-605, Bagladesh bstract I ths research, we have compared three dfferet attrbute selecto measures algorthms. We have used ID3 algorthm, C4.5 algorthm ad CRT algorthm. ll these algorthms are decso tree based algorthm. We have got the accuracy of three dfferet algorthms ad we observed that the accuracy of ID3 algorthm s greater tha C4.5 algorthm. But the accuracy of CRT algorthm s greater tha other two algorthms. We have also calculated the tme complexty of three dfferet algorthms. To compare these algorthms, we have used heart dsease dataset whch s collected from UCI mache learg repostory. Keywords Patter extracto, Classfcato, Decso Tree, ID3 algorthm, C4.5 algorthm, CRT algorthm. I. INTRODUCTION Today s world s a dgtal world. s the world populato s creasg rapdly, the eed ad usage of storg eormous amout of wde rages of data to databases have creased rapdly recet years. Data-mg s ecessary to extract hdde useful kowledge from large datasets a gve applcato. Ths usefuless relates to the user goal, other words oly the user ca determe whether the resultg kowledge aswers hs goal. Data mg refers to extractg or mg kowledge from large amouts of data. There are several tools of data mg. It s ecessary to choose useful data mg tools to acqure useful kowledge. lso data mg tools should be hghly teractve ad partcpatory. Decso tree ducto s a greedy algorthm that costructs decso tree a top-dow recursve dvde-ad-coquer maer. decso tree s a tree whch each brach ode represets a choce betwee a umbers of alteratves, ad each leaf ode represets a decso. For extractg rules, formato ga measure s used to select the test attrbute at each ode the tree. Classfcato s oe of the most popular tasks data mg. Classfcato volves the assgmet of a object to oe of several pre-specfed categores. Classfcato of data wthout ay terpretato of the uderlyg model could reduce the trust of users the system. Data mg s a very demadg research feld amog researchers ow-a-days. I our research we have performed Patter extracto, classfcato ad comparso betwee ttrbute selecto measures. We foud that some research has bee doe ths feld before, but ther accuracy ot good eough ad there are spaces where mprovemet ca be acheved. II. OVERVIEW OF THIS WORK t frst, we have dvded the data set to fve fold by usg fve fold cross valdato method. By applyg fve fold cross valdato method, we get fve fold trag ad testg dataset whch s show Fg. 1. Data set Fve Fold Cross Valdato Method Trag ad Testg Data Set Fg. 1 Steps for gettg Trag ad Testg Dataset The a classfcato algorthm s appled to the trag dataset to extract patter. The extracted patter s appled to the testg dataset. The we get the classfed dataset whch s show Fg.. Trag dataset Learg lgorthm Extracted Patter Testg dataset Classfed dataset Fg. Basc steps of ths work 371

2 Subrata Pramak et al. / (IJCSIT) Iteratoal Joural of Computer Scece ad Iformato Techoes, Vol. 1 (5), 010, III. PTTERN EXTRCTION By aalyzg a large amout of data a patter or rule s extracted. Rules are the verbal equvalet of a graphcal decso tree, whch specfes class membershp based o the herarchcal sequece of decsos. Each rule a set of decso rules therefore geerally takes the form of a Hor clause where class membershp s mpled by a cojucto of cotget observato [1]. typcal rule structure s gve below: IFcodto1 NDcodto ND ND codto THEN CLSS= class IV. CLSSIFICTION Data classfcato s two-step process [1, ]. I the frst step, a classfer s bult descrbg a predetermed set of data classes or cocepts. Ths s the learg step (or trag phase), where a classfcato algorthm bulds the classfer by aalyzg or learg from a trag set made up of database tuples ad ther assocated class labels. tuple, X, s represeted by a -dmesoal attrbute vector, X=(x1,x, xn), depctg N measuremets made o the tuple from N database attrbutes respectvely, 1,..N. Each tuple, X, assumed to be to a predefed class as determed by aother database attrbute called the class label attrbute. The class label attrbute s dscrete-valued ad uordered. It s categorcal that each value serves as a category or class. The dvdual tuples makg up the trag set are referred to as trag tuples ad are selected from the database uder aalyss. I ths cotext of classfcato, data tuples ca be referred to as samples, examples, staces, data pots or objects [1]. Because the class label of each trag tuple s provded, ths step s also kow as supervsed learg (.e. the learg of the classfer s supervsed that t s told to whch class each trag tuple bes). It costructs wth usupervsed learg (or clusterg), whch the class label of each trag tuple s ot kow, ad the umber or classes to be leared may ot be kow advace [1, ]. V. TTRIBUTE SELECTION MESURES There are may algorthms to select the attrbute whch decdes whch attrbute wll be the top of the tree. I ths research, we have used the followg attrbute selecto measures:. ID3 (Iteratve Dchotomser) lgorthm The ID3 techque [1,, 9] to buldg a decso tree s based o formato theory ad attempts to mmze the expected umber of comparsos. The basc dea of the ducto algorthm s to ask questos whose aswers provde the most formato. The frst questo dvdes the search space to two large search domas, whle the secod performs lttle dvso of the space. The basc strategy used by ID3 s to choose splttg attrbutes wth the hghest formato ga frst. The amout of formato assocated wth a attrbute value s related to the probablty of occurrece. Let ode N represets or hold the tuples of partto D. The attrbute wth the hghest formato ga s chose as the splttg attrbute for ode N. Ths attrbute mmzes the formato eeded to classfy the tuples the resultg parttos ad reflects the least radomess or mpurty these parttos. To calculate the ga [1,, 9] of a attrbute, at frst we calculate the etropy of that attrbute by the followg formula: Etropy ( S ) p p (1) 1 where, p s the probablty that a arbtrary tuple S bes to class C ad estmated by C, D. fucto D to the base s used, because the formato s ecoded bts. Etropy(S) s just the average amout of formato eeded to detfy the class label of the tuple S. Now ga of a attrbute s calculated by the formula: [1] S Ifo ( S ) Etropy ( S ) () 1 S where, S ={ S, S,... S }= parttos of S accordg to 1 values of attrbute = umber of attrbutes S = umber of cases the partto S = total umber of cases S Iformato ga s defed as the dfferece betwee the orgal formato requremet ad ew requremet. That s, [1] Ga S ( ) Etropy ( S ) Ifo ( S ) (3) I other words, Ga() tell us how much would be gaed by brachg o. It s the expected reducto the formato requremet caused by kowg the value of. The attrbute wth hghest formato ga s chose as the splttg attrbute at ode N. B. C4.5 lgorthm The C4.5 algorthm s Qula s [1,, 7, 8] exteso of hs ow ID3 algorthm for geeratg decso trees. Just as wth CRT, the C4.5 algorthm recursvely vsts each decso ode, selectg the optmal splt, utl o further splts are possble. The steps of C4.5 algorthm [1,, 7] for growg a decso tree s gve below: choose attrbute for root ode. Create brach for each value of that attrbute Splt cases accordg to braches Repeat process for each brach utl all cases the brach have the same class questo that, how a attrbute s chose as a root ode? t frst, we calculate of the ga rato of each attrbute. The root ode wll be that attrbute whose ga rato s maxmum. Ga rato s calculated by the formula: [7, 8] Ga ( ) (4) GaRato ( ) SpltIfo ( ) where, s a attrbute whose ga rato wll be calculated. The attrbute wth the maxmum ga rato s selected as the 37

3 Subrata Pramak et al. / (IJCSIT) Iteratoal Joural of Computer Scece ad Iformato Techoes, Vol. 1 (5), 010, splttg attrbute. Ths attrbute mmzes the formato eeded to classfy the tuples the resultg parttos. Such a approach mmzes the expected umber of tests eeded to classfy a gve tuple ad guaratees that a smple tree f foud. To calculate the ga of a attrbute, at frst we calculate the etropy of that attrbute by the followg formula: [7, 8] Etropy ( S ) p p (5) 1 where, P s the probablty that a arbtrary tuple S bes to class C ad estmated by C, D /D. fucto to the base s used, because the formato s ecoded bts. Etropy(S) s just the average amout of formato eeded to detfy the class label of the tuple S Now ga of a attrbute s calculated by the formula: [7, 8] S Ga ( ) Etropy ( S ) Etropy ( S ) (6) S 1 where, S ={ S, S,... S }= parttos of S accordg to 1 values of attrbute = umber of attrbutes. S = umber of cases the partto S S = total umber of cases S The ga rato dvdes the ga by the evaluated splt formato. Ths pealzes splts wth may outcomes. [7, 8] SpltIfo S S ( ) (7) S 1 S The splt formato s the weghted average calculato of the formato usg the proporto of cases whch are passed to each chld. Whe there are cases wth ukow outcomes o the splt attrbute, the splt formato treats ths as a addtoal splt drecto. Ths s doe to pealze splts whch are made usg cases wth mssg values. fter fdg the best splt, the tree cotues to be grow recursvely usg the same process. C. CRT lgorthm The CRT algorthm [1, ] measures the mpurty of D, a data partto or set of trag tuples, as m P 1 G ( D ) 1 (8) where, P s the probablty that a tuple D bes to class C ad s estmated by C, D / D. The sum s computed over m classes. The CRT algorthm [1, ] cosders a bary splt for each attrbute. Let s frst cosder the case where s a dscrete valued attrbute havg v dstct values, { a a,...,. a 1 v }, occurrg D. To determe the best splt o, we exame all of the possble subsets that ca be formed usg kow value of. each subset, S a, ca be cosdered as a bary test of attrbute of the form S. Gve a tuple, ths test s satsfed f the value of for the tuple s amog the values lsted S. If has v v possble values, the there are possble subsets. Whe cosderg a bary splt, we compute a weghted sum of the mpurty of each resultg partto. For example, f a bary splt o parttos D to D 1 ad D, the g dex of D gve that parttog s [1] D1 D G ( D) G ( D1 ) G ( D ) (9) D D For each attrbute, each of the possble bary splts s cosdered. For a dscrete-valued attrbute, the subset that gves the mmum g dex for the attrbute s selected as ts splttg subset. For cotuous-valued attrbutes [], each possble spltpot must be cosdered. The strategy s smlar to the descrbed for formato ga, where the mdpot betwee each par of adjacet values s take as a possble splt-pot. The pot gvg the mmum g dex for a gve attrbute s take as the splt-pot of that attrbute. Recall that for a possble splt-pot of, D s the set of tuples D satsfyg 1 splt-pot, ad D s the set of tuples D satsfyg >splt-pot. The reducto mpurty that would be curred by a bary splt o a dscrete-valued or cotuous-valued attrbute s [1] G ( ) G ( D ) G ( D ) (10) The attrbute that maxmzes the reducto mpurty s selected as the splttg attrbute. Ths attrbute ad ether ts splttg subset or splt-pot together form the splttg crtero. VI. RESULT ND DISCUSSION To expermet the research cocept o a represetatve dataset, a system s developed usg Matlab7, whch s powerful tool for complex calculato ad hgh-level programmg.. Expermetal Data I our research as expermetal data we have used a bomedcal dataset for detectg heart dsease ad t s collected from the UCI Mache Learg Repostory. The repostory database s freely avalable ad ca be obtaed from the followg lk: The heart dsease data set of 70 patets s used ths expermet. Ths dataset cotas 13 attrbutes ad a class varable wth two possble values, whch are show Table-I the followg page.. Ths data cotas some attrbutes (such as age, restg blood pressure, Serum cholesterol mg/dl, Maxmum heart rate acheve, ST depresso duced by exercse relatve to rest ad The slope of the peak exercse ST segmet) whch 373

4 Subrata Pramak et al. / (IJCSIT) Iteratoal Joural of Computer Scece ad Iformato Techoes, Vol. 1 (5), 010, cotas cotuous values. So, before usg ths dataset ths expermet, those cotuous valued attrbutes are dvded to rages. TBLE I DESCRIPTION OF THE FETURES IN THE HERT DISESE DTSET Fg- shows obtaed orgal dataset. The resultat trasformed dataset s show Fg-3. Both the data sets are trasformed to Matlab readable text fles. Fg-: The orgal obtaed dataset. B. Patter Extracto Sgfcat patters are extracted whch are useful for uderstadg the data patter ad behavour of expermetal dataset. The followg patter s extracted by applyg CRT attrbute selecto measures algorthm. Heart_dsease(absece):- Thal=fxed_defect,Number_Vessels=0, Cholestoral = Coverage = 4 samples Heart_dsease(presece):- Thal=ormal, Number_Vessels=0, Old_Peak=0-1.5, Max_Heart_Rate= , Cholestoral= Coverage = 7 samples Heart_dsease(absece):- Thal=ormal, Number_Vessels=0, Old_Peak=0-1.5, Max_Heart_Rate= , Cholestoral=14-301, Rest=0, Pressure= Coverage = 5 samples C. Classfcato Ths system successfully classfes heart dsease dataset used ths research. Whe test data does ot match wth oe of the patters the class attrbute of ths sample s labelled as uclassfed whch meas that ths system faled to classfy the dataset. By applyg the extracted patter o testg dataset I have got the followg classfed dataset. Total data tem: 54(54) Coverage: 54 Mss Class=13 ccuracy: % D. Comparso betwee ttrbute Selecto Measures We have mplemeted the ID3, C4.5 CRT algorthm ad tested them o our expermetal dataset. The results are show o Table-II ad Fg-4. TBLE III COMPRTIVE RESULTS OF BTRIBUTE SELECTION MESURES Fg-3: The trasformed dataset for learg. 374

5 Subrata Pramak et al. / (IJCSIT) Iteratoal Joural of Computer Scece ad Iformato Techoes, Vol. 1 (5), 010, ccuracy Comparso betwee ttrbute Selecto Measures Fold1 Fold Fold3 Fold4 Fold5 verage ID3 C4.5 CRT Fg. 4 Comparso betwee attrbute selecto measures whch shows the accuracy of each algorthms From our results we observed that amog the attrbute selecto measures C4.5 performs better tha the ID3 algorthm, but CRT performs eve better both respect of accuracy ad tme complexty. I a prevous research [11], the authors used the ID3 algorthm of decso tree ducto methods ad the average accuracy of ths research was %. We have doe ths research ad we have foud % accuracy wth the CRT algorthm whch s greater tha prevous research. VII. CONCLUSION I our research we choose three attrbute selecto measure algorthms to have a comparatve study amog them. s expermetal data we have used the bomedcal heart dsease dataset. To classfy dataset, we have mplemeted three dfferet algorthms. mog these algorthms, CRT algorthm always geerates a bary decso tree. That meas the decso tree geerated by CRT algorthm has exactly two or o chld. But the decso tree whch s geerated by other two algorthms may have two or more chld. lso, respect of accuracy ad tme complexty CRT algorthm performs better tha the other two algorthms. REFERENCES [1] Jawe Ha ad Mchele Kamber, Data Mg Cocepts ad techques, d ed., Morga Kaufma Publshers, Sa Fracsco, C, 007. [] Margaret H. Duham, Data Mg Itroductory ad dvaced Topcs, Publshed by Pearso Educato (Sgapur) Pte. Ltd. Delh, Ida, 004. [3] D. E. Johso, F. J. Oles, T. Zhag ad T. Geotz, Decso Tree Based Symbolc Rule Iducto System for Text Categorzato, IBM Systems Joural, Vol. 41, No 3, 00 [4]. Juozapavcus ad V. Rapsevcus, Clusterg through Decso Tree Costructo Geoy, Nolear alyss: Modelg ad Cotrol, 001, vol. 6, No, [5] hmed Sulta l-hegam, Classcal ad Icremetal Classfcato Data Mg Process, IJCSNS Iteratoal Joural of Computer Scece ad Network Securty, VOL.7 No.1, December 007. [6] Kusr, Sr Hartat, Implemetato of C4.5 algorthm to evaluate the cacellato possblty of ew studet applcats at stmk amkom yogyakarta. Proceedgs of the Iteratoal Coferece o Electrcal Egeerg ad Iformatcs Isttut Teko Badug, Idoesa Jue 17-19, 007. [7] Jaso R. Beck, Mara E. Garca, Mgyu Zhog, Mchael Georgopoulos, Georgos agostopoulos Backward djustg Strategy for the C4.5 Decso Tree Classfer, Beck, et al [8] Qula, J. R., C4.5: Programs for Mache Learg, Sa Mateo, C: Morga Kaufma Publcatos, [9] Qula, J. R., Iducto of Decso Trees, Mache Learg, 1:1, Bosto: Kluwer, cademc Publshers, 1986, [10] Rajeev Rastog, Kyuseok Shm, PUBLIC: Decso Tree Classfer that Itegrates Buldg ad Prug Bell Laboratores. [11] Coushk hmed, Utpala Nada Chowdhury, Md. Nazmul Huda, Extracto of Rules from Chemcal Compoud Data usg LEPH, Natoal Coferece o Electrocs, Iformato ad Telecommucato, 007, Bagladesh. 375