Three Embedded Methods

Size: px

Start display at page:

Download "Three Embedded Methods"

Samantha Allen
5 years ago
Views:

1 Embedded Methods

3 Review Wrappers Evaluation via a classifier, many search procedures possible. Slow. Often overfits. Filters Use statistics of the data. Fast but potentially naive. Embedded Methods A classifier which selects features naturally as part of learning. Returns a feature set and a classifier Much less prone to overfitting.

4 Three Embedded Methods Random Forests for Feature Ranking Recursive Feature Elimination with SVM Adaboost for Feature Selection

5 Three Embedded Methods Random Forests for Feature Ranking Recursive Feature Elimination with SVM Adaboost for Feature Selection

6 Background: Decision Trees Recursive partitioning of space. At each node, a new feature is selected. Trees implicitly select good features with mutual information.

7 Background: Random Forests For t = 1:T Take a bootstrap from the dataset. Build a decision tree, but at each node... - choose random subset of the available features - select the best split from this subset. end Combine trees by majority voting. No pruning. Subset size usually chosen as M = log 2 (M + 1).

8 Background: Random Forests Trees can end up very different...

9 Random Forests : Average Information Gain Method Build a Random Forest as usual. For each feature X k For each tree t If X k was used, examine the split Record the information-gain gain(k, t) End End End Feature score: J(X k ) = 1 T T t=1 gain(k, t) (Averaged gain in the forest)

10 Random Forests : Permutation Test Method {D in, D out } = bootstrap(d) Build a tree using D in. Calculate error e on this data. Permute feature X k, giving D perm out. Calculate error e perm on permuted data. ( ) Feature score: J(X k ) = e perm e. (increase of error) Average this score over all trees in the forest.

11 Random Forests for Feature Ranking RF Very accurate, rarely overfits, if ever. Can be parallelized for speed. RF for Feature Ranking Handles mixed data (continuous/categorical) Naturally multi-variate (not univariate) Used everywhere in bio-informatics.

12 Three Embedded Methods Random Forests for Feature Ranking Recursive Feature Elimination with SVM Adaboost for Feature Selection

13 Background: Support Vector Machines

14 Background: Support Vector Machines This boundary has maximum margin.

15 Background: Support Vector Machines A linear SVM has a decision function: f(x) = w T x b. where the weights/bias are found by a constrained minimisation: L(α) = 1 2 wt w N i=1 λ i{y i (w T x i b) 1} First term proportional to 1 margin. So smaller 1 2 wt w... means a bigger margin. Second term avoids the trivial minimum of w i = 0, i.

16 Background: Support Vector Machines Feature X2 doesn t seem to be helping us much...

17 Background: Support Vector Machines For a point on the decision boundary, we have: M f(x) = w T x b = w i x i b = 0 In our 2-D example: i=1 f(x) = w 1 x 1 + w 2 x 2 b = 0 w 1 x 1 + x 2 = b w 2 w 2 x 2 = w 1 x 1 + b w 2 w 2 This follows the geometry of the line, y = mx + c, with gradient m = ( w 1 w 2 )

18 Background: Support Vector Machines w 1 = m = w 1 w 2 w 2 = b = 1 Gradient m > 1 means feature x 1 is most useful. Gradient m < 1 means feature x 2 is most useful.

19 Recursive Feature Elimination 10. Train a linear SVM. 20. Remove the feature with smallest weight w i 30. Goto 10. Or, remove many at once - depending on computational power. Looks like a wrapper. Generally referred to as an embedded method. The boundaries are not strict : filter/wrapper/embedded... Gene selection for cancer classification using support vector machines I. Guyon, J.Weston, B.Barnhill, and V.Vapnik, Mach. Learning, Vol 46 (1), 2002.

20 Three Embedded Methods Random Forests for Feature Ranking Recursive Feature Elimination with SVM Adaboost for Feature Selection

21 Background: Adaboost (Freund & Schapire, 1996) Classic algorithm - MANY MANY extensions. Basic idea: 10. Train a classifier h t, on dataset D t. 20. Find examples where h t makes errors - emphasize these examples, giving D t Increment t, goto 10. Dataset Weighted Bootstrap Learner 1 Weighted Bootstrap Learner 2 Weighted Bootstrap Learner m Weighted Vote

22 Background: Adaboost Input: Training set D = {(x i, y i )} N 1, where y i { 1, +1} Define a uniform distribution D 1 (i) over elements of D. for t = 1 to T do Train a model h t using distribution D t. Calculate ɛ t = ( P Dt (h) t (x) y) Set α t = 1 2 ln 1 ɛ t ɛ t Update D t+1 (i) D t (i) e αtyiht(xi) end for For a new testing ( point (x, y ), T ) H(x ) = sign t=1 α th t (x )

23 Boosting for Feature Selection... A decision stump... Base your weak learners on single features. The order in which Adaboost uses them, gives the feature ranking.

24 Viola & Jones Face Detector P.Viola & M.Jones, Robust Real-Time Face Detection, International Journal of Computer Vision, vol 57 (2004) - Major breakthrough in computer vision research. - First object-detection scheme to work in real-time. - Uses Adaboost to select simple features.

25 Viola & Jones Face Detector A B C D Figure 5: The first and second features selected by AdaBoost. The two features are sho and then overlayed on a typical training face in the bottom row. The first feature measure intensity between the region of the eyes and a region across the upper cheeks. The feature observation that the eye region is often darker than the cheeks. The second feature comp in the eye regions to the intensity across the bridge of the nose. le features shown relative to the enclosing detection window. The sum of the Output of feature is thresholded sum of pixels in white area white rectangles are subtracted from the sum of pixels in the grey rectangles. shown subtract in (A) and the (B). sum Figure of(c) pixels showsina grey three-rectangle area. feature, and (D) a directly increases computation time. 4 The Attentional Cascade

26 Viola & Jones Face Detector

The Hardest Question in Pattern Recognition Ok, I ve learnt all this stuff. Now, which one will be best for my data?!! Speed Filters are fast. Wrappers are slow. Embedded methods are in the middle.

27 The Hardest Question in Pattern Recognition Ok, I ve learnt all this stuff. Now, which one will be best for my data?!! Speed Filters are fast. Wrappers are slow. Embedded methods are in the middle. Data Efficiency (i.e. if we have few samples) Uni-variate filters probably best. With larger feature-sample ratio, try RELIEF / JMI / CMIM depending on the data-type. If data/computation are no problem, try a wrapper...but it is likely to overfit.

28 Conclusions Day 2, finito! On day 3, we will study why/when FS works. On day 4, we study cause/effect in feature selection, and probabilistic frameworks to analyse/understand FS.

Learning to Detect Faces. A Large-Scale Application of Machine Learning

Learning to Detect Faces. A Large-Scale Application of Machine Learning Learning to Detect Faces A Large-Scale Application of Machine Learning (This material is not in the text: for further information see the paper by P. Viola and M. Jones, International Journal of Computer