GRADIENT DESCENT. An aside: text classification. Text: raw data. Admin 9/27/16. Assignment 3 graded. Assignment 5. David Kauchak CS 158 Fall 2016

Transcription

1 Adi Assiget 3 graded Assiget 5! Course feedback GRADIENT DESCENT David Kauchak CS 158 Fall 2016 A aside: text classificatio Text: ra data Ra data labels Ra data labels Features? Chardoay Chardoay Piot Grigio Piot Grigio Zifadel Zifadel 1

2 Feature exaples Feature exaples Ra data labels Features Ra data labels Features Chardoay Clito said piot repeatedly last eek o tv, piot, piot, piot Chardoay Clito said piot repeatedly last eek o tv, piot, piot, piot Piot Grigio piot clito said califoria across tv rog (1, 1, 1, 0, 0, 1, 0, 0, ) capital Piot Grigio piot clito said califoria across tv rog (4, 1, 1, 0, 0, 1, 0, 0, ) capital Zifadel Occurrece of ords Zifadel Frequecy of ord occurreces This is the represetatio e re usig for assiget 5 Decisio trees for text Decisio trees for text Each iteral ode represets hether or ot the text has a particular ord heat heat is a coodity that ca be foud i states across the atio heat buschl far buschl far export coodity export coodity agriculture agriculture 2

3 Decisio trees for text Pritig out decisio trees The US vies techology as a coodity that it ca export by the buschl. heat buschl far export coodity agriculture heat buschl export far coodity agriculture (heat (buschl predict=ot heat (export predict=ot heat predict=heat)) (far (coodity (agriculture predict=ot heat predict=heat) predict=heat) predict=heat)) Soe ath today (but do t orry!) Liear odels A strog high-bias assuptio is liear separability:! i 2 diesios, ca separate classes by a lie! i higher diesios, eed hyperplaes A liear odel is a odel that assues the data is liearly separable 3

4 Liear odels A liear odel i -diesioal space (i.e. features) is defie by +1 eights: I to diesios, a lie: 0 = 1 f f 2 + b (here b = -a) I three diesios, a plae: 0 = 1 f f f 3 + b I -diesios, a hyperplae 0 = b + Perceptro learig algorith repeat util covergece (or for soe # of iteratios): for each traiig exaple (f 1, f 2,, f, label): predictio = b + if predictio * label 0: // they do t agree for each : = + *label b = b + label Which lie ill it fid? Which lie ill it fid? Oly guarateed to fid soe lie that separates the data 4

5 Liear odels Perceptro algorith is oe exaple of a liear classifier May, ay other algoriths that lear a lie (i.e. a settig of a liear cobiatio of eights) Goals: - Explore a uber of liear traiig algoriths - Uderstad hy these algoriths ork Perceptro learig algorith repeat util covergece (or for soe # of iteratios): for each traiig exaple (f 1, f 2,, f, label): predictio = b + if predictio * label 0: // they do t agree for each i : i = i + f i *label b = b + label A closer look at hy e got it rog Model-based achie learig * f 1 +1* f 2 = 0 * 1+1* 1= 1 (-1, -1, positive) We d like this value to be positive sice it s a positive value 1. pick a odel - e.g. a hyperplae, a decisio tree, - A odel is defied by a collectio of paraeters What are the paraeters for DT? Perceptro? did t cotribute, but could have decrease cotributed i the rog directio decrease 0 -> > 0 Ituitively these ake sese Why chage by 1? Ay other ay of doig it? 5

6 Model-based achie learig 1. pick a odel - e.g. a hyperplae, a decisio tree, - A odel is defied by a collectio of paraeters Model-based achie learig 1. pick a odel - e.g. a hyperplae, a decisio tree, - A odel is defied by a collectio of paraeters DT: the structure of the tree, hich features each ode splits o, the predictios at the leaves perceptro: the eights ad the b value 2. pick a criterio to optiize (aka objective fuctio) What criteria do decisio tree learig ad perceptro learig optiize? Model-based achie learig 1. pick a odel - e.g. a hyperplae, a decisio tree, - A odel is defied by a collectio of paraeters 2. pick a criterio to optiize (aka objective fuctio) - e.g. traiig error 3. develop a learig algorith - the algorith should try ad iiize the criteria - soeties i a heuristic ay (i.e. o-optially) - soeties exactly Liear odels i geeral 1. pick a odel 0 = b + These are the paraeters e at to lear 2. pick a criterio to optiize (aka objective fuctio) 6

7 Soe otatio: idicator fuctio Soe otatio: dot-product!# 1 if x = True %# 1[ x] = " & $# 0 if x = False '# Coveiet otatio for turig T/F asers ito ubers/couts: beers _ to_ brig _ for _ class = 1[ age >= 21] age class Soeties it is coveiet to use vector otatio We represet a exaple f 1, f 2,, f as a sigle vector, x Siilarly, e ca represet the eight vector 1, 2,, as a sigle vector, The dot-product betee to vectors a ad b is defied as: a b = a j b j Liear odels 1. pick a odel These are the paraeters e at to lear 2. pick a criterio to optiize (aka objective fuctio) 0 = b + 1[ y i + b) 0] What does this equatio say? 0/1 fuctio 1[ y i + b) 0] hether or ot the predictio ad label agree, true if they do t total uber of istakes, aka 0/1 - distace fro hyperplae - sig is predictio 7

8 Model-based achie learig 1. pick a odel 0 = b + 2. pick a criteria to optiize (aka objective fuctio) 1[ y i + b) 0] 3. develop a learig algorith [ ] argi,b 1 y i + b) 0 Fid ad b that iiize the 0/1 (i.e. traiig error) Miiizig 0/1 [ ] argi,b 1 y i + b) 0 Ho do e do this? Ho do e iiize a fuctio? Why is it hard for this fuctio? Fid ad b that iiize the 0/1 Miiizig 0/1 i oe diesio Miiizig 0/1 over all 1[ y i + b) 0] 1[ y i + b) 0] Each tie e chage such that the exaple is right/rog the ill icrease/decrease Each e feature e add (i.e. eights) adds aother diesio to this space! 8

9 Miiizig 0/1 More aageable fuctios [ ] argi,b 1 y i + b) 0 Fid ad b that iiize the 0/1 This turs out to be hard (i fact, NP-HARD!) Challege: - sall chages i ay ca have large chages i the (the chage is t cotiuous) - there ca be ay, ay local iia - at ay give poit, e do t have uch iforatio to direct us toards ay iia What property/properties do e at fro our fuctio? More aageable fuctios Covex fuctios Covex fuctios look soethig like: - Ideally, cotiuous (i.e. differetiable) so e get a idicatio of directio of iiizatio - Oly oe iia Oe defiitio: The lie seget betee ay to poits o the fuctio is above the fuctio 9

10 Surrogate fuctios Surrogate fuctios For ay applicatios, e really ould like to iiize the 0/1 0/1 : l(y, y') =1[ yy' 0] A surrogate fuctio is a fuctio that provides a upper boud o the actual fuctio (i this case, 0/1) We d like to idetify covex surrogate fuctios to ake the easier to iiize Ideas? Soe fuctio that is a proxy for error, but is cotiuous ad covex Key to a fuctio: ho it scores the differece betee the actual label y ad the predicted label y Surrogate fuctios Surrogate fuctios 0/1 : l(y, y') =1[ yy' 0] 0/1 : l(y, y') =1[ yy' 0] Squared : l(y, y') = (y y') 2 Hige: l(y, y') = ax(0,1 yy') Expoetial: l(y, y') = exp( yy') Hige: l(y, y') = ax(0,1 yy') Expoetial: l(y, y') = exp( yy') Squared : l(y, y') = (y y') 2 Why do these ork? What do they pealize? y-y 10

11 Model-based achie learig 1. pick a odel 2. pick a criteria to optiize (aka objective fuctio) 3. develop a learig algorith argi,b 0 = b + use a covex surrogate fuctio Fid ad b that iiize the surrogate Fidig the iiu You re blidfolded, but you ca see out of the botto of the blidfold to the groud right by your feet. I drop you off soehere ad tell you that you re i a covex shaped valley ad escape is at the botto/iiu. Ho do you get out? Fidig the iiu Oe approach: gradiet descet Partial derivatives give us the slope (i.e. directio to ove) i that diesio Ho do e do this for a fuctio? 11

12 Oe approach: gradiet descet Oe approach: gradiet descet Partial derivatives give us the slope (i.e. directio to ove) i that diesio Approach:! pick a startig poit ()! repeat: " pick a diesio " ove a sall aout i that diesio toards decreasig (usig the derivative) Partial derivatives give us the slope (i.e. directio to ove) i that diesio Approach:! pick a startig poit ()! repeat: " pick a diesio " ove a sall aout i that diesio toards decreasig (usig the derivative) Gradiet descet Gradiet descet! pick a startig poit ()! repeat util does t decrease i ay diesio: " pick a diesio " ove a sall aout i that diesio toards decreasig (usig the derivative) = η d d ()! pick a startig poit ()! repeat util does t decrease i ay diesio: " pick a diesio " ove a sall aout i that diesio toards decreasig (usig the derivative) = η d d () What does this do? learig rate (ho uch e at to ove i the error directio, ofte this ill chage over tie) 12

13 Soe aths Gradiet descet d d = d d = d y i + b) d = y i x ij! pick a startig poit ()! repeat util does t decrease i ay diesio: " pick a diesio " ove a sall aout i that diesio toards decreasig (usig the derivative) = +η y i x ij What is this doig? Expoetial update rule = +η y i x ij for each exaple x i : = +ηy i x ij Does this look failiar? Perceptro learig algorith! repeat util covergece (or for soe # of iteratios): for each traiig exaple (f 1, f 2,, f, label): predictio = b + if predictio * label 0: // they do t agree for each : = + *label b = b + label = +ηy i x ij or = + x ij y i c here c = η 13

14 The costat c = η The costat c = η learig rate label predictio Whe is this large/sall? label predictio If they re the sae sig, as the predicted gets larger there update gets saller If they re differet, the ore differet they are, the bigger the update Perceptro learig algorith! Oe cocer repeat util covergece (or for soe # of iteratios): for each traiig exaple (f 1, f 2,, f, label): predictio = b + if predictio * label 0: // they do t agree for each : Note: for gradiet descet, e alays update = + *label b = b + label argi,b We re calculatig this o the traiig set We still eed to be careful about overfittig! = +ηy i x ij or = + x ij y i c here c = η The i,b o the traiig set is geerally NOT the i for the test set Ho did e deal ith this for the perceptro algorith? 14

15 Suary Model-based achie learig: - defie a odel, objective fuctio (i.e. fuctio), iiizatio algorith Gradiet descet iiizatio algorith - require that our fuctio is covex - ake sall updates toards loer es Perceptro learig algorith: - gradiet descet - expoetial fuctio (odulo a learig rate) 15