Our Learning Problem, Again

Transcription

1 Noparametric Desity Estimatio Matthew Stoe CS 520, Sprig 2000 Lecture 6 Our Learig Problem, Agai Use traiig data to estimate ukow probabilities ad probability desity fuctios So far, we have depeded o describig the fuctios i a kow parametric form Today, we relax that assumptio 1

2 Let s Start with a Obvious Idea Nearest-eighbor classificatio Algorithm: Start with poits of traiig data: = { x,,x } 1 Give test poit x Fid traiig poit x closest to x Assig x the same category as x How Well Does This Work? Hard to say uless you have a lot of data But suppose data is o object Label of x is class ω ; true label of x is ω Correct aswer if ω = ω: What s P(ω = ω)? p(ω x ) i geeral p(ω x) as x becomes closer to x 2

3 How Well Does This Work? cotiued Probability of error at x is therefore: c i = 1 1 P( ω x) P( ω x) i i.e., wrog i all cases except those where x ad x happe to agree. I priciple, best you could do is: 1 P( ω i x) i.e., guess most likely i How Well Does This Work? some perspective Ayoe who s ayoe gets 95% accuracy Whe Bayes error is 5% 1 P( ω i x) Limit earest-eighbor error is ~9% 1 P( ω x) P( ω x) c i = 1 i Could be better, if distributios are favorable Could be worse, because you do t have ifiite data i 3

4 How Well Does This Work? some perspective Surprisigly good (sice it s so easy) But it may ot be eough for your task Classifyig sequeces At 7 elemets, Bayes could get 2/3 right Nearest eighbor is just gettig 1/2 right A Possible Improvemet K-Nearest Neighbor Classificatio Start with poits of traiig data: = { x,,x } 1 Give test poit x Fid k traiig poits X closest to x Assig x the most frequet category of X 4

5 K-Nearest Neighbor Good poits: More likely data ca overcome rare evets I earest eighbor, each rare data poit traslates ito a ball of likely mistakes I 3-earest eighbor, you eed two rare data poits together to get a ball of likely mistakes Ca get better ad better the more poits vote K-Nearest Neighbor Bad poits: Need tos more data Oly if voters are close to x does vote provide good desity iformatio about x Oly by cosiderig lots of voters do you coverge o a accurate likelihood for x 5

6 Returig to Desity Estimatio Have t we chaged the problem? K-earest eighbor is a classifier Maximum likelihood builds a distributio Wat to get a distributio for KNN Compare approaches Mix KNN ad other ifo probabilistically Returig to Desity Estimatio Basic earest eighbor idea works To fid p( x, ωi ) Place a cell of volume V aroud x Capture k samples, of which i are i Calculate i k p(x,ω i ) = V p( x, ωi ) p( x ωi ) = P( ω ) i ω i 6

7 Returig to Desity Estimatio Basic earest eighbor idea works Well, almost Real probabilities should itegrate to oe (Although you do t always eed real ω i probabilities to build discrimiat fuctios) Volume V varies as a fuctio of x so you may have trouble across the whole space i k p(x,ω i ) = V Sample-based Desity Estimatio We ll ow cosider a close variat of KNN that represets the desity more coveietly Parze Widows 7

8 Parze Widows Treat each sample as cotributig a small Gaussia desity that peaks aroud it ad drops off quickly Use parameter h (dummy for variace σ) to cotrol drop off Desity aroud u is: ϕ( x; u) = 1 ( x u) ( x u) exp 2 2πh 2h T Parze Widows Overall desity for data = { x,,x } 1 is 1 p( x) = ϕ( x; xi ) i = 1 8

9 Cute Implemetatio 3 Layer Network: Layer Oe: Iputs Each ode gets a feature of the patter that you re classifyig Patter is ormalized to have uit legth Cute Implemetatio 3 Layer Network: Layer Two: Patters Each ode gets a ormalized traiig vector w; o iput x it computes z = w T x 9

10 Cute Implemetatio 3 Layer Network: Layer Two: Patters The ode will output likelihood compoet e 2 2 ( z 1) σ Cute Implemetatio 3 Layer Network: Layer Three: Categories Oe ode per class 2 2 Sums iput from ( patter z 1) σ e odes for traiig data i the class 10

11 Kid of Neural Network Easy to trai Add a ew patter ode for each sample Easy to iterpret probabilistically Approximates arbitrary iput distributios (usig samples) Outputs Bayes optimal classificatio give its assumed distributio of iputs 11