Classification. Outline. 8.1 Statistical Learning Theory Formulation. 8.3 Methods for Classification. 8.2 Classical Formulation.

Transcription

1 lassfcaton Learnng From Data hater 8 lassfcaton Inut samle, 2,, d s classfed to one and onl one of J grous. oncerned wth relatonsh between the classmembersh label and feature vector. Goal s to estmate the mang decson rule usng labeled tranng data,. Outlne 8. Statstcal Learnng Theor Formulaton 8.2 lasscal Formulaton 8.3 Methods for lassfcaton 8.4 Summar Two-class roblem Outut of the sstem Takes on values {, }. Learnng machne Imlement a set of ndcator functons f, ω. Loss functon L, f, w f f f f,, ω ω

2 Rsk Functonal R ω L, f, ω, dd Learnng s the roblem of fndng the functon f, ω that mnmzes the robablt of msclassfcaton usng onl the tranng data. lasscal formulaton of the classfcaton ondtonal denst, α *, ror robablt β * osteror robablt Imlementaton of methods usng SRM requres: Secfcaton of a nested structure on a set of ndcator aromatng functons. Mnmzaton of the emrcal rsk msclassfcaton error for a gven element of a structure. Estmaton of redcton rsk. Decson rule f, α * >, β * f otherwse Ales the ERM nductve rncle ndrectl to frst estmate the denstes. oncetuall flawed n estmatng decson boundar va denst estmaton.

3 8. Statstcal Learnng Theor Formulaton Goal s to estmate an ndcator functon or decson boundar f, ω. Use SRM nductve rncle. S h S 2... S m h 2... h m h s V - dmenson of m S m onstructve methods should select an element of a structure S m f, ω and an ndcator functon. f š, ω m Bound on redcton rsk m m R ω R ω + Φ em / m n h SLT Formulaton ont d Bnar classfcaton roblem Goal s to mnmze the msclassfcaton error. ercetron algorthm Smle otmzaton rocedure for fndng f,ω* rovdng zero msclassfcaton error f tranng data s lnearl searable. ndcator functon s SLT Formulaton ont d Two strateges for mnmzng the bound Kee the confdence nterval fed and mnmze the emrcal rsk. Include all statstcal and neural network methods usng dctonar reresentaton. Wth hgh-dmensonal data, frst roject the data onto the low-dmensonal subsace.e, m features and then erform modelng n ths subsace. Kee the value of the emrcal rsk fed small and mnmze the confdence nterval. Requres a secal structure such that the value of the emrcal rsk s ket small for all aromatng functons. rerequste Weght udate rule When the data s not searable and/or the otmal decson boundar s nonlnear, does not rovde an otmal soluton.

4 SLT Formulaton ont d ML classfer an form fleble nonlnear decson boundares. Aromate the ndcator functon b a wellbehaved sgmod functon. Emrcal rsk functonal Sgmod actvatons of hdden unts enable the constructon of a fleble nonlnear decson boundar. Outut sgmod aromates the dscontnuous ndcator functon. SLT Formulaton ont d General rescrton for mlementng constructve methods Secf a fleble class of aromatng functons for constructng a nonlnear decson boundar. 2 hoose a nonlnear otmzaton method for selectng the best functon from class,.e., the functon rovdng the smallest emrcal rsk. 3 Select contnuous error functonal sutable for the otmzaton methods chosen n 2. 4 Select the best redctve model from a class of functons usng the frst strateg for mnmzng SLT bound. In neural networks, a common rocedure for classfcaton decsons s to use sgmod outut. f I [ s g, w *, V * θ ] Outut of the traned network s nterreted as an estmate of the osteror robablt. s g, w *, V * ˆ SLT Formulaton ont d lassfcaton methods use a contnuous error functonal that onl aromates the true one. A classfcaton method usng such aromaton wll be successful onl f mnmzng of the error functonal selected n 3 also mnmzes true emrcal rsk.

5 8.2 lasscal Formulaton Based on statstcal decson theor rovdes the foundaton for constructng otmal decson rules mnmzng rsk. Strctl ales onl when all dstrbutons are known. oncerned wth constructng decson rules. Estmate the requred dstrbutons from the data and use them wthn the framework of statstcal decson theor. lasscal Formulaton ont d When s not observed When s observed lasscal Formulaton ont d Msclassfcaton wth unequal costs { } otherwse f whenever s defned such that regon mnmzed f Rsk s rsk s overall the eected cost s, > < r d d d d q d d q If R R R R R R R R R

6 osteror dstrbutons Two strateges Estmate the ror robabltes and class condtonal denstes and lug them nto Baes rule. Estmate osteror denstes drectl usng tranng data from all the classes. Two aroaches for two strateges arametrc methods. Adatve fleble methods. arametrc Regresson Lnear regresson can be used for classfcaton for non-gaussan dstrbutons. Fsher lnear dscrmnant rovdes an aromaton for the osteror robablt, but based, rovdng an accurate classfcaton rule. Ma rovde a oor decson boundar. osteror ont d Drect Estmaton Estmaton of osteror denstes can be done usng regresson methods. For two-class case Adatve Regresson Accurate regresson does not guarantee a low classfcaton rsk. Use the data to estmate the condtonal eectaton. For two-class roblems

7 8.3 Methods for lassfcaton 8.3. Regresson-Based Methods Methods based on contnuous numercal otmzaton can be cast n the form of multleresonse regresson. Most oular aroach to classfcaton Tree-Based Methods Based on a greed otmzaton strateg. ART s an eamle. ML lassfers Use -of-j outut encodng and sgmod outut unts. ractcal hnts and mlementaton ssues re-scalng of nut varables Scale nut data to the range [-.5,.5]. Tcall re-scaled to zero mean, unt varance. Hels to avod remature saturaton and seeds u tranng. Alternatve target outut values Tranng oututs are set to values. and.9. Avod long tranng tme and etremel large weghts durng tranng Nearest-Neghbor and rotote Methods Local methods for classfcaton. Estmate the decson boundar locall. Nearest-neghbors classfcaton, Kohonen s learnng vector quantzaton Intalzaton Network arameters are ntalzed to small random values. hoce of ntalzaton range has subtle regularzaton effect. Stong rules Durng tranng Tranng should roceed as long as the decreasng contnuous loss functon reduces the emrcal msclassfcaton error. Earl stong used as a form of comlet control model selecton.

8 Multle local mnma Man factor comlcatng emrcal rsk mnmzaton as well as model selecton. Use the msclassfcaton error rather than squared error loss durng model selecton. Learnng rate and momentum term Affects local mnma found b backroagaton tranng. Otmal choce of these s roblem-deendent. RBF ont d haracterstcs Imlements local decson boundares. For fed values of bass functon arameters, w k are estmated va lnear least squares. omlet can be determned b the number of bass functons m. ossble to use resamlng technques to estmate redcton rsk to erform model selecton. Tcall use normalzed bass functons Allows to be nterreted as a te of denst mture model. RBF lassfers Use multle outut regresson to buld a decson boundar. Form of dscrmnant functons Tree-Based Methods Adatvel slt nut sace nto dsjont regons n order to construct a decson boundar. Based on a greed otmzaton rocedure. Slttng rocess can be reresented as a bnar tree. Followng the growth of the tree, runng occurs as a form of model selecton. Growng and runng strateg rovdes better classfcaton accurac than just growng alone.

9 Tree-Based Methods runng crtera rovdes an estmate of the redcton rsk whle the growng crtera roughl reflects emrcal rsk. Resultng classfer has a bnar tree reresentaton. ART ont d Q t Q t, 2 t, n t t n rtera to be met,, j t J t n j t n t Q s at ts mamum onl for robabltes /J,, /J. Q s at ts mnmum onl for robabltes,,,,,,,,,,,,,,. Q s a smmetrc functon of ts arguments. ART oular aroach to construct a bnar-treebased classfer. Greed search emlos a recursve arttonng strateg. ost functon Measure node murt. Gve a measurement of how homogeneous a node t s wth resect to the class labels of the tranng data n the regon of node t. ART ont d Eamles Gn and entro functons are used for ractcal mlementaton. Gn and entro functons do not measure the classfcaton rsk drectl.

10 ART ont d Two dffcultes when usng the emrcal msclassfcaton cost There are cases where msclassfcaton cost does not decrease for an canddate slt, leadng to earl haltng n a oor local mnmum. The msclassfcaton cost does not favor slts that tend to rovde a lower msclassfcaton cost n future slts. ART ont d Decrease n murt Varable k and the slt ont v are selected to mamze the decrease n node murt. ART ont d runng erformed after growng s comleted. Imlements model selecton. Based on mnmzng enalzed emrcal rsk. erformed n a greed search strateg. Ever ar of sblng leaf nodes s recombned n order to fnd a ar that reduces the followng. R em en R + λ T R em : msclassfcaton rate for the tranng data T : number of termnal nodes

11 ART ont d rocedure Intalzaton Root node conssts of whole nut sace. Estmate roorton of the classes va jtn j /n. Tree growng Reeat untl stong crteron s satsfed. erform ehaustve search over all vald nodes n the tree, all slt varables, and all vald knot onts. Incororate the daughters nto the tree that results n the largest decrease n the murt usng gn or entro cost functon. Nearest-Neghbor and rotote Methods Goal of local methods for classfcaton onstructon of local decson boundares. In SLT vew, local methods for classfcaton follow the framework of local rsk mnmzaton. In classcal decson theor, local methods are nterreted as local osteror denst estmaton. ART ont d Tree runng Reeat untl no more runng occurs. erform ehaustve search over all sblng leaf nodes n the tree, measurng the change n model selecton crteron. Delete the ar that leads to the largest decrease of model selecton crteron. Dsadvantages Senstve to coordnate rotatons Because ART arttons the sace nto as-orented subregons. an be allevated b slttng on lnear combnatons of nut varables. Nearest-Neghbor lassfcaton lassfes an object based on the class of the k data onts nearest to the estmaton ont. Nearness s most commonl measured usng eucldean dstance metrc n -sace. Local decson rule s constructed usng the rocedure of local rsk mnmzaton.

12 Nearest ont d Eamle : two-class roblem Emrcal rsk Learnng Vector Quantzaton LVQ Outlne Use vector quantzaton methods to determne ntal locatons of m rotote vectors. Assgn class labels to these rotote. Adjust the locatons usng a heurstc strateg. Reduce the emrcal msclassfcaton rsk. LVQ, LVQ2, LVQ3 b Kohonen. Nearest ont d Reasons for the success ractcal roblems often have a low ntrnsc dmensonalt even though the ma have man nut varables. Effect of the curse of dmensonalt s not as severe due to the nature of the classfcaton roblem. Accurate estmates of condtonal robabltes are not necessar for accurate classfcaton. LVQ ont d LVQ A stochastc aromaton method. Gven k, k : a data ont, c j k : rotote center, and rotote labels w j. Determne the nearest rotote center to the data ont. Udate the locaton of the nearest rotote. If correctl classfed Else kk+

13 Emrcal omarsons Mture of Gaussans Rle, 994 Emrcal omarsons2 Lnearl Searable roblem Tranng data sets are generated accordng to the dstrbuton N,I, R. Ten tranng sets are generated, and each data set contans 2 samles. redcton rsk s estmated for each ndvdual classfer usng a large test set 2 samles.

14 Emrcal omarsons3 Waveform Data Frst used n Breman et al nut varables that corresond to 2 dscrete tme samles taken from a randoml generated waveform. Waveform s generated usng a random lnear combnaton of two out of three ossble comonent waveforms. lass: lass 2: lass 3: j j j u u u h h h 2 j j j u u u h h h j j j + ε + ε + ε j j j Ten tranng sets are generated, and each data set contans 3 samles. redcton rsk s estmated usng a large test set 2 samles. Summar Understandng classfcaton methods requres clear searaton between concetual rocedure based on the SRM nductve rncle and ts techncal mlementaton. oncetual rocedure Two necessar thngs Mnmze the emrcal classfcaton error va nonlnear otmzaton. Estmate accuratel future classfcaton error model selecton.

15 Summar ont d Techncal mlementaton omlcated b the dscontnuous msclassfcaton error functonal. revents drect mnmzaton of the emrcal rsk All ractcal methods use a sutable contnuous loss functon rovdng aromaton for msclassfcaton error. In model selecton, should use classfcaton error loss. Accurate estmaton of a osteror robabltes s not necessar for accurate classfcaton. Good robablt estmates are not necessar for good classfcaton; smlarl, low classfcaton error does not ml that the corresondng class robabltes are beng estmate even remotel accuratel - Fredman 997 Summar ont d SLT s elanaton for emrcal evdence on wellbehavng of smle methods Smle classfcaton methods ma not requre nonlnear otmzaton, so the emrcal classfcaton error s mnmzed drectl. Often smle methods rovde the same level of emrcal msclassfcaton error n the mnmzaton ste as more comle methods. No need to use more comle methods. lassfcaton roblems are nherentl less senstve than regresson to otmal model selecton.