CS246: Mining Massive Datasets Jure Leskovec, Stanford University

Size: px

Start display at page:

Download "CS246: Mining Massive Datasets Jure Leskovec, Stanford University"

Amelia Hill
5 years ago
Views:

1 CS46: Mnng Massve Datasets Jure Leskovec, Stanford Unversty

/19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, http://cs46.stanford.edu Perceptron: y = sgn( x Ho to fnd parameters?

2 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, Perceptron: y = sgn( x Ho to fnd parameters? Start th 0 = 0 Pck tranng examples x t one by one Predct class of x t usng current t y = sgn( t x t If y s correct (.e., y t = y No change: t1 = t If y s rong: Adjust t t1 = t y t x t s the learnng rate parameter x t s the tth tranng example y t s true tth class label ({1, 1} y t x t t1 t x t, y t =1

/19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, http://cs46.stanford.

3 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 3 Overfttng: Regularzaton: If the data s not separable eghts dance around Medocre generalzaton: Fnds a barely separatng soluton

4 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 4 Want to separate from usng a lne Data: Tranng examples: (x 1, y 1 (x n, y n Each example : x = ( x (1,, x (d x (j s real valued y { 1, 1 } Inner product: x = d j=1 (j x (j Whch s best lnear separator (defned by?

5 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 5 A B C Dstance from the separatng hyperplane corresponds to the confdence of predcton Example: We are more sure about the class of A and B than of C

6 Margn: Dstance of closest example from the decson lne/hyperplane The reason e defne margn ths ay s due to theoretcal convenence and exstence of generalzaton error bounds that depend on the value of margn. /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 6

7 Remember: Dot product A B = A B cos θ A = A (j /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 7 d j=1

8 Dstance from a pont to a lne A (x A (1, x A ( H L Let: Lne L: xb = (1 x (1 ( x ( b=0 = ( (1, ( Pont A = (x A (1, x A ( Pont M on a lne = (x M (1, x M ( (0,0 M (x 1, x d(a, L = AH = (AM = (x A (1 x M (1 (1 (x A ( x M ( ( = x A (1 (1 x A ( ( b = A b Remember x (1 M (1 x ( M ( = b snce M belongs to lne L /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 8

9 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 10 Predcton = sgn(x b Confdence = ( x b y For th datapont: γ = x b y Want to solve: max mn Can rerte as max, γ s. t., y ( x b

10 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 11 Maxmze the margn: Good accordng to ntuton, theory (VC dmenson & practce max, s. t., y ( x b xb=0 γ s margn dstance from the separatng hyperplane Maxmzng the margn

11 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 1 Separatng hyperplane s defned by the support vectors Ponts on / planes from the soluton If you kne these ponts, you could gnore the rest If no degeneraces, d1 support vectors (for d dm. data

12 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 13 Problem: Let x b y = γ then x b y = γ Scalng ncreases margn! Soluton: Work th normalzed : γ = x b y x x 1 Let s also requre support vectors x j to be on the plane defned by: x j b = ±1 d = (j j=1

13 Want to maxmze margn γ! What s the relaton beteen x 1 and x? x 1 = x γ We also kno: x 1 b = 1 x b = 1 So: x 1 b = 1 x γ x b γ 1 b = 1 = 1 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 14 x x 1 1 Note:

14 We started th max, s. t., y ( x b mn s. t., 1 y ( x b But can be arbtrarly large! We normalzed and... max max Then: 1 mn mn 1 Ths s called SVM th hard constrants 1 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 15 x x 1

15 If data s not separable ntroduce penalty: mn 1 s. t., y ( x C b 1 Mnmze ǁǁ plus the number of tranng mstakes Set C usng cross valdaton Ho to penalze mstakes? All mstakes are not equally bad! (#number of mstakes /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 16

16 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 17 Introduce slack varables mn, b, 0 s. t., 1 y ( x If pont x s on the rong sde of the margn then get penalty C n 1 b 1 j For each datapont: If margn 1, don t care If margn < 1, pay lnear penalty

17 mn s. t., 1 y ( x C (#number of b 1 What s the role of slack penalty C: C=: Only ant to, b that separate the data C=0: Can set to anythng, then =0 (bascally gnores the data (0,0 mstakes /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 18 small C good C bg C

18 penalty /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 19 SVM n the natural form arg mn, b 1 C SVM uses Hnge Loss : 0/1 loss n max 0,1 ( x Margn 1 Emprcal loss L (ho ell e ft tranng data Regularzaton parameter mn, b 1 s. t., y y C ( x n 1 b b Hnge loss: max{0, 1z} z y ( x b

19 Announcement: HW s graded. We sorted t alphabetcally nto several ples. Please don t mess the ples. /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 0

20 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 1 mn, b 1 s. t., y ( x C Want to estmate and b! Standard ay: Use a solver! Solver: softare for fndng solutons to common optmzaton problems Use a quadratc solver: Mnmze quadratc functon Subject to lnear constrants Problem: Solvers are neffcent for bg data! n 1 b 1

21 Want to estmate, b! Alternatve approach: Want to mnmze f(,b: Ho to mnmze convex functons f(z? Use gradent descent: mn z f(z Iterate: z t1 z t f (z t /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, n d j j j d j j b x y C b f 1 1 ( ( 1 ( 1 ( 0,1 max, ( n b b x y s t C 1 (,.. mn 1 1, f(z z

22 Want to mnmze f(,b: Compute the gradent (j.r.t. (j /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 3 n j j j y x L C b f j 1 ( ( (, (, ( ( else 1 ( f 0, ( ( ( j j x y b x y y x L n d j j j d j j b x y C b f 1 1 ( ( 1 ( 1 ( 0,1 max, ( Emprcal loss L(x y

23 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 4 Gradent descent: Iterate untl convergence: For j = 1 d f (, b Evaluate: ( j ( j Update: (j (j (j j C n 1 L( x, y ( j Problem: Computng (j takes O(n tme! n sze of the tranng dataset learnng rate parameter C regularzaton parameter

24 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 5 Stochastc Gradent Descent Instead of evaluatng gradent over all examples evaluate t for each ndvdual tranng example ( j L( x, y ( j, C ( j Stochastc gradent descent: Iterate untl convergence: For = 1 n For j = 1 d Evaluate: (j, Update: (j (j (j, We just had: n j L( x, y ( j C ( j ( 1

25 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 6 Example by Leon Bottou: Reuters RCV1 document corpus Predct a category of a document One vs. the rest classfcaton n = 781,000 tranng examples (documents 3,000 test examples d = 50,000 features One feature per ord Remove stopords Remove lo frequency ords

26 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 7 Questons: (1 Is SGD successful at mnmzng f(,b? ( Ho quckly does SGD fnd the mn of f(,b? (3 What s the error on a test set? Standard SVM Fast SVM SGD SVM Tranng tme Value of f(,b Test error (1 SGDSVM s successful at mnmzng the value of f(,b ( SGDSVM s super fast (3 SGDSVM test set error s comparable

27 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 8 SGD SVM Conventonal SVM Optmzaton qualty: f(,b f ( opt,b opt For optmzng f(,b thn reasonable qualty SGDSVM s super fast

SGD on full dataset vs. Batch Conjugate Gradent on a sample of n tranng examples Theory says: Gradent descent converges n lnear tme k. Conjugate gradent converges n k.

28 SGD on full dataset vs. Batch Conjugate Gradent on a sample of n tranng examples Theory says: Gradent descent converges n lnear tme k. Conjugate gradent converges n k. Bottom lne: Dong a smple (but fast SGD update many tmes s better than dong a complcated (but slo BCG update a fe tmes k condton number /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 9

29 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 30 Need to choose learnng rate and t 0 t L( x, y t 1 t t C t t0 Leon suggests: Choose t 0 so that the expected ntal updates are comparable th the expected sze of the eghts Choose : Select a small subsample Try varous rates (e.g., 10, 1, 0.1, 0.01, Pck the one that most reduces the cost Use for next 100k teratons on the full dataset

30 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 31 Sparse Lnear SVM: Feature vector x s sparse (contans many zeros Do not do: x = [0,0,0,1,0,0,0,0,5,0,0,0,0,0,0, ] But represent x as a sparse vector x =[(4,1, (9,5, ] Can e do the SGD update more effcently? L( x, y C Approxmated n steps: L( x, y C ( 1 cheap: x s sparse and so fe coordnates j of ll be updated expensve: s not sparse, all coordnates need to be updated

31 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 3 Soluton 1: = s v Represent vector as the product of scalar s and vector v Then the update procedure s: To step update procedure: (1 ( L( x, y C ( 1 (1 v = v ηc L x,y ( s = s(1 η Soluton : Perform only step (1 for each tranng example Perform step ( th loer frequency and hgher

32 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 33 Stoppng crtera: Ho many teratons of SGD? Early stoppng th cross valdaton Create valdaton set Montor cost functon on the valdaton set Stop hen loss stops decreasng Early stoppng Extract to dsjont subsamples A and B of tranng data Tran on A, stop by valdatng on B Number of epochs s an estmate of k Tran for k epochs on the full dataset

33 Idea 1: One aganst all Learn 3 classfers vs. {o, } vs. {o, } o vs. {, } Obtan: b, b, o b o Ho to classfy? Return class c arg max c c x b c /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 34

34 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 35 Learn 3 sets of eghts smoultaneously For each class c estmate c, b c Want the correct class to have hghest margn: y x b y 1 c x b c c y, (x, y

35 Optmzaton problem: To obtan parameters c, b c (for each class c e can use smlar technques as for class SVM SVM s dely perceved a very poerful learnng algorthm /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 36 c c y y n c b b x b x C 1 mn 1 c 1, y c 0,,

36 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 37 Ne settng: Onlne Learnng Allos for modelng problems here e have a contnuous stream of data We ant an algorthm to learn from t and sloly adapt to the changes n data Idea: Do slo updates to the model All our methods SVM and Perceptron make updates f they msclassfy an example So: Frst tran the classfer on tranng data. Then for every example from the stream, f e msclassfy, update the model (usng small learnng rate

37 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 38 Protocol: User comes and tell us orgn and destnaton We offer to shp the package for some money ($10 $50 Based on the prce e offer, sometmes the user uses our servce (y = 1, sometmes they don't (y = 1 Task: Buld an algorthm to optmze hat prce e offer to the users Features x capture: Informaton about user Orgn and destnaton Problem: Wll user accept the prce?

38 Model hether user ll accept our prce: y = f(x; Accept: y =1, Not accept: y=1 Buld ths model th say Perceptron or Wnno The ebste that runs contnuously Onlne learnng algorthm ould do somethng lke User comes User s represented as an (x,y par here x: Feature vector ncludng prce e offer, orgn, destnaton y: If they chose to use our servce or not The algorthm updates usng just the (x,y par Bascally, e update the parameters every tme e get some ne data /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 39

39 We dscard ths dea of a data set Instead e have a contnuous stream of data Further comments: For a major ebste here you have a massve stream of data then ths knd of algorthm s pretty reasonable Don t need to deal th all the tranng data If you had a small number of users you could save ther data and then run a normal algorthm on the full dataset Dong multple passes over the data /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 40

40 /19/013 Jure Leskovec, Stanford CS46: Mnng Massve Datasets, 41 An onlne algorthm can adapt to changng user preferences For example, over tme users may become more prce senstve The algorthm adapts and learns ths So the system s dynamc

CS246: Mining Massive Datasets Jure Leskovec, Stanford University

CS246: Mining Massive Datasets Jure Leskovec, Stanford University CS246: Mnng Massve Datasets Jure Leskovec, Stanford Unversty http://cs246.stanford.edu 2/17/2015 Jure Leskovec, Stanford CS246: Mnng Massve Datasets, http://cs246.stanford.edu 2 Hgh dm. data Graph data