Machine Learning: Algorithms and Applications

Size: px

Start display at page:

Download "Machine Learning: Algorithms and Applications"

Emery Smith
5 years ago
Views:

1 /03/ Mache Learg: Algorthms ad Applcatos Florao Z Free Uversty of Boze-Bolzao Faculty of Computer Scece Academc Year 0-0 Lecture 3: th March 0 Naïve Bayes classfer ( Problem defto A trag set X, where each trag stace x s represeted as a -dmesoal attrbute vector: (x, x,..., x A pre-defed set of classes: C={c, c,..., c m } Gve a ew stace z, whch class should z be classfed to? We wat to fd the most probable class for stace z c MAP = arg max P( c cmap = argmax P( c z, z,..., c C c MAP = argmax c!c P(z, z,..., z c * P(c P(z, z,..., z c C (by Bayes theorem c MAP = argmax P(z, z,..., z c " P(c c!c (P(z,z,...,z s the same for all classes

2 /03/ Naïve Bayes classfer ( Assumpto Naïve Bayes classfer. The attrbutes are codtoally depedet gve the classfcato P( z, z,..., z c = P( z j c j= Naïve Bayes classfer fds the most probable class for z c NB = argmax c!c " P(c * P(z j c j= Naïve Bayes classfer - Algorthm The learg (trag phase (gve a trag set For each class c C Estmate the pror probablty: P(c For each attrbute value z j, estmate the probablty of that attrbute value gve class c : P(z j c The classfcato phase For each class c C, compute the formula " P(c! P(z j c Select the most probable class c * c * = argmax c!c j= # P(c " P(z j c j=

3 /03/ Naïve Bayes classfer Example ( Rec. ID Age Icome Studet Credt_Ratg Buy_Computer Youg Hgh No Far No Youg Hgh No Excellet No 3 Medum Hgh No Far Yes 4 Old Medum No Far Yes 5 Old Low Yes Far Yes 6 Old Low Yes Excellet No 7 Medum Low Yes Excellet Yes 8 Youg Medum No Far No 9 Youg Low Yes Far Yes 0 Old Medum Yes Far Yes Youg Medum Yes Excellet Yes Medum Medum No Excellet Yes 3 Medum Hgh Yes Far Yes 4 Old Medum No Excellet No Wll a youg studet wth medum come ad far credt ratg buy a computer? Naïve Bayes classfer Example ( Represetato of the problem z = (Age=Youg,Icome=Medum,Studet=Yes,Credt_Ratg=Far Two classes: c (buy a computer ad c (ot buy a computer Compute the pror probablty for each class P(c = 9/4 P(c = 5/4 Compute the probablty of each attrbute value gve each class P(Age=Youg c = /9; P(Age=Youg c = 3/5 P(Icome=Medum c = 4/9; P(Icome=Medum c = /5 P(Studet=Yes c = 6/9; P(Studet=Yes c = /5 P(Credt_Ratg=Far c = 6/9; P(Credt_Ratg=Far c = /5 3

4 /03/ Naïve Bayes classfer Example (3 Compute the lkelhood of stace z gve each class For class c P(z c = P(Age=Youg c *P(Icome=Medum c *P(Studet=Yes c * P(Credt_Ratg=Far c = (/9*(4/9*(6/9*(6/9 = For class c P(z c = P(Age=Youg c *P(Icome=Medum c *P(Studet=Yes c * P(Credt_Ratg=Far c = (3/5*(/5*(/5*(/5 = 0.09 Fd the most probable class For class c P(c *P(z c = (9/4*(0.044 = 0.08 For class c P(c *P(z c = (5/4*(0.09 = Cocluso: The perso z (a youg studet wth medum come ad far credt ratg wll buy a computer! Naïve Bayes classfer Issues ( What happes f o trag staces assocated wth class c have attrbute value x j? E.g., the buy computer example, o youg studets bought computers P(x j c = (c j,x j /(c j =0, ad hece: Soluto: use a Bayesa approach to estmate P(x j c ( c, x j + mp P( x j c = ( c + m (c : umber of trag staces assocated wth class c (c,x j : umber of trag staces assocated wth class c that have attrbute value x j p: a pror estmate for P(x j c Assume uform prors: p=/k, f attrbute f j has k possble values m: a weght gve to pror P(c!" P(x j c = 0 To augmet the (c actual observatos by a addtoal m vrtual samples dstrbuted accordg to p j= 4

5 /03/ Naïve Bayes classfer Issues ( P(x j c <, for every attrbute value x j ad class c So, whe the umber of attrbute values s very large lm j= P( x j c = 0 Soluto: use a logarthmc fucto of probablty c NB = argmax c!c c NB * $ '- log& P(c "# P(x j c, + %& j= ( /. = arg max log P( c + log P( x c C j= j c Naïve Bayes classfer Summary Oe of the most practcal learg methods Based o the Bayes theorem Parameter estmato for Naïve Bayes models uses the maxmum lkelhood estmato Computatoally very fast Trag: oly oe pass over the trag set Classfcato: lear the umber of attrbutes Despte ts codtoal depedece assumpto, Naïve Bayes classfer shows a good performace several applcato domas Whe to use? A moderate or large trag set avalable Istaces are represeted by a large umber of attrbutes Attrbutes that descrbe staces are codtoally depedet gve classfcato 5

6 /03/ Lear regresso Lear regresso Itroducto Goal: to predct a real-valued output gve a put stace A smple-but-effectve learg techque whe the target fucto s a lear fucto The learg problem s to lear (.e., approxmate a real-valued fucto f f: X Y X: The put doma (.e., a -dmesoal vector space R Y: The output doma (.e., the real values doma R f: The target fucto to be leared (.e., a lear mappg fucto f ( x = w0 + w x + w x w x = w0 + w x (w,x R Essetally, to lear the weghts vector w = (w 0, w, w,, w = 6

7 /03/ Lear regresso Example What s the lear fucto f(x? x f(x E.g., f(x = x f(x x Lear regresso Trag / test staces For each trag stace x=(x,x,...,x X, where x R The desred (target output value c x ( R The actual output value y x = w Here, w are the system s curret estmates of the weghts 0 + = w x The actual output value y x s desred to (approxmately be c x For a test stace z=(z,z,...,z To predct the output value By applyg the leared target fucto f 7

8 /03/ Lear regresso Error fucto The learg algorthm requres to defe a error fucto To measure the error made by the system the trag phase Defto of the trag square error E Error computed o each trag example x: E( x = ( cx yx = 0 cx w = w x Error computed o the etre trag set X: E = " E(x = " (c x # y x = $ c # w # w x ' "& " % x 0 ( x!x x!x x!x = Least-square lear regresso Learg the target fucto f s equvalet to learg the weghts vector w that mmzes the trag square error E Why the ame of the approach s Least-Square Lear Regresso Trag phase Italze the weghts vector w (small radom values Compute the trag error E Update the weghts vector w accordg to the delta rule Repeat utl covergg to a (locally mmum error E Predcto phase For a ew stace z, the (predcted output value s: f ( = w* 0 + w* z where w*=(w* 0,w*,..., w* s the leared weghts vector = 8

9 /03/ The delta rule To update the weghts vector w the drecto that decreases the trag error E η s the learg rate (.e., a small postve costat To decde the degree to whch the weghts are chaged at each trag step Istace-to-stace update: w w + η(c x -y x x Batch update: w! w +! $ ( c x " y x x Other ames of the delta rule LMS (least mea square rule Adale rule Wdrow-Hoff rule x#x LSLR_batch(X, η for each attrbute w a tal (small radom value whle ot CONVERGENCE for each attrbute delta_w 0 for each trag example x X compute the actual output value y x for each attrbute delta_w delta_w + η(c x -y x x for each attrbute w w + delta_w ed whle retur w 9

10 /03/ Batch vs. cremetal update The prevous algorthm follows a batch update approach Batch update At each trag step (cycle, the weghts are updated after all the trag staces are putted to the system - Frst, the error s computed cumulatvely o all the trag staces - The, the weghts are updated accordg to the overall (cumulated error Icremetal update At each trag step, the weghts are updated mmedately after each trag stace s putted to the system - The dvdual error s computed for the trag stace - The weghts are updated mmedately accordg to the dvdual error LSLR_cremetal(X, η for each attrbute w a tal (small radom value whle ot CONVERGENCE for each trag example x X compute the actual output value y x for each attrbute w w + η(c x -y x x ed whle retur w 0

11 /03/ Trag termato codtos I the LSLR_batch ad LSLR_cremetal learg algorthms, the trag process termates whe the codtos dcated by CONVERGENCE are met The (trag termato codtos are typcally defed based o some kd of system performace measure Stop, f the error s less tha a threshold value Stop, f the error at a learg step s greater tha that at the prevous step Stop, f the dfferece betwee the errors at two cosecutve steps s less tha a threshold value Stop, f... Nearest eghbor learer

12 /03/ Nearest eghbor learer Itroducto ( Some alteratve ames Istace-based learg Lazy learg Memory-based learg Nearest eghbor learer Gve a set of trag staces Just store the trag staces Not costruct a geeral, explct descrpto (model of the target fucto based o the trag staces Gve a test stace (to be classfed/predcted Exame the relatoshp betwee the test stace ad the trag staces to assg a target fucto value Nearest eghbor learer Itroducto ( The put represetato Each stace x s represeted as a vector a - dmesoal vector space X R x = (x,x,,x, where x ( R s a real umber We cosder two learg tasks Nearest eghbor learer for classfcato To lear a dscrete-valued target fucto The output s oe of pre-defed omal values (.e., class labels Nearest eghbor learer for predcto To lear a cotuous-valued target fucto The output s a real umber

13 /03/ Nearest eghbor learer Example earest eghbor Assg z to c 3 earest eghbors Assg z to c 5 earest eghbors Assg z to c class c class c test stace z k-nearest eghbor classfer Algorthm For the classfcato task Each trag stace x s represeted by The descrpto: x=(x,x,,x, where x R The class label: c ( C, where C s a pre-defed set of class labels Trag phase Just store the trag staces set X = {x} Test phase. To classfy a ew stace z For each trag stace x X, compute dstace betwee x ad z Compute the set NB( the eghbourhood of z The k staces X earest to z accordg to a dstace fucto d Classfy z to the majorty class of the staces NB( 3

14 /03/ k-nearest eghbor predctor Algorthm For the regresso task (.e., to predct a real output value Each trag stace x s represeted by The descrpto: x=(x,x,,x, where x R The output value: y x R (.e., a real umber Trag phase Just store the trag examples set X Test phase. To predct the output value for ew stace z For each trag stace x X, compute dstace betwee x ad z Compute the set NB( the eghbourhood of z The k staces X earest to z accordg to a dstace fucto d Predct the output value of z: y z = k x NB( y x Oe vs. More tha oe eghbor Usg oly a sgle eghbor (.e., the trag stace closest to the test stace to determe the classfcato s subject to errors E.g., ose (.e. error the class label of a sgle trag stace Cosder the k (> earest trag staces, ad retur the majorty class label of these k staces The value of k s typcally odd to avod tes For example, k=3 or k=5 4

15 /03/ Dstace fucto ( The dstace fucto d Play a very mportat role the earest eghbor learg approach Typcally defed before, ad fxed through, the trag ad test phases.e., ot adjusted based o data Choce of the dstace fucto d Geometry dstace fuctos, for cotuous-valued put space (x R Hammg dstace fucto, for bary-valued put space (x {0,} Dstace fucto ( Geometry dstace fuctos Mahatta dstace Eucldea dstace Mkowsk (p-orm dstace Chebyshev dstace d( = d( = d d = x z ( x z = / p p ( = x z = / p p ( = lm x z p = = max x z 5

16 /03/ Dstace fucto (3 Hammg dstace fucto For bary-valued put space E.g., x=(0,,0,, d( = = Dfferece( x, z, f ( a b Dfferece( a, b = 0, f ( a = b Attrbute value ormalzato The Eucldea dstace fucto Assume that a stace s represeted by 3 attrbutes: Age, Icome (per moth, ad Heght ( meters x = (Age=0, Icome=000, Heght=.68 z = (Age=40, Icome=300, Heght=.75 The dstace betwee x ad z d( = [( ( ( ] / The dstace s domated by the local dstace (dfferece o the Icome attrbute Because the Icome attrbute has a large rage of values To ormalze the values of all the attrbutes to the same rage Usually the value rage [0,] s used d( = ( x z E.g., for every attrbute : x = x /max_value_of_attrbute_ = 6

17 /03/ Attrbute mportace weght The Eucldea dstace fucto d( = = All the attrbutes are cosdered equally mportat the dstace computato Dfferet attrbutes may have dfferet degrees of fluece o the dstace metrc To corporate attrbute mportace weghts the dstace fucto w s the mportace weght of attrbute : ( x z How to acheve the attrbute mportace weghts? By the doma-specfc kowledge (e.g., dcated by experts the problem doma By a optmzato process (e.g., usg a separate valdato set to lear a optmal set of attrbute weghts d( = = w ( x z Dstace-weghted NN learer ( Cosder NB( the set of the k trag staces earest to the test stace z Each (earest stace has a dfferet dstace to z Should these (earest staces fluece equally to the classfcato/predcto of z? No! test stace z To weght the cotrbuto of each of the k eghbors accordg to ther dstace to z Larger weght for earer eghbor! 7

18 /03/ Dstace-weghted NN learer ( Let s deote by v a dstace-based weghtg fucto Gve a dstace d( the dstace of x to z v( s versely proportoal to d( For the classfcato task: " $ c( = argmax # v( "!(c j,c(x!(a, b = # c j!c x!nb( %$ v(! f (x For the predcto task: f ( = # x"nb( x"nb( v( Select a dstace-based weghtg fucto #,f (a = b 0,f (a! b v( = v( α + d( = σ v( = e α + [ d( ] d ( Lazy learg vs. Eager learg Lazy learg. The target fucto estmato (.e., geeralzato s postpoed utl the test stace s troduced E.g., Nearest eghbor learer, Locally weghted regresso Estmate (.e., approxmate the target fucto locally ad dfferetly for each test stace.e., performed at the classfcato/predcto tme Compute may local approxmatos of the target fucto Typcally take loger tme to aswer queres, ad requre more memory space Eager learg. The target fucto estmato s completed before ay test stace s troduced E.g., Lear regresso, Support vector maches, Neural etworks, etc. Estmate (.e., approxmate the target fucto globally for the etre stace space.e., performed at the trag tme Compute a sgle (global approxmato of the target fucto 8

19 /03/ Nearest eghbor learer Whe? Istaces are represeted as vectors R The dmesoalty of the put space s ot large A large set of trag staces s avalable Advatages No trag s eeded (.e., just store the trag staces Scale well wth a large umber of classes Not eed to lear classfers for classes k-nn (k >> learer s robust to osy data Classfcato/predcto s performed cosderg k earest eghbors Dsadvatages Dstace fucto must be carefully chose Computatoal cost ( tme ad memory at the classfcato/predcto tme May be msled by rrelevat attrbutes 9

CS 2710 Foundations of AI Lecture 22. Machine learning. Machine Learning

CS 2710 Foundations of AI Lecture 22. Machine learning. Machine Learning CS 7 Foudatos of AI Lecture Mache learg Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Mache Learg The feld of mache learg studes the desg of computer programs (agets) capable of learg from past eperece