Ulas Bagci

Transcription

1 CAP5415-Computer Vision Lecture 14-Decision Forests for Computer Vision Ulas Bagci 1

2 Readings Slide Credits: Criminisi and Shotton Z. Tu R.Cipolla 2

3 Common Terminologies Randomized Decision Forests Random Forests Decision Forests 3

4 The Abstract Forest Model Regression Semi-supervised learning Classification Decision Forests Density estimation Manifold learning Active learning 4

5 General Idea S Training Data Multiple Data Sets S1 S2 Sn Multiple Classifiers C1 C2 Cn Combined Classifier H 5

6 The Abstract Forest Model Recognizing the type of a scene captured in a photograph can be cast as a classification task. The desired output is a discrete label Sky, dog, cat, indoor, outdoor, etc. 6

7 The Abstract Forest Model Recognizing the type of a scene captured in a photograph can be cast as a classification task. The desired output is a discrete label: Sky, dog, cat, indoor, outdoor, etc. Predicting the price of a house as a function of its distance from a good school may be thought of as a regression problem. In this case the output is a continuous variable. 7

8 The Abstract Forest Model Recognizing the type of a scene captured in a photograph can be cast as a classification task. The desired output is a discrete label: Sky, dog, cat, indoor, outdoor, etc. Predicting the price of a house as a function of its distance from a good school may be thought of as a regression problem. In this case the output is a continuous variable. Interactive image segmentation may be thought of as a semisupervised problem, where the user s brush strokes define labeled data and the rest of image pixels provide already available unlabeled data. Can be applied to both regression and classification problems! 8

9 The Abstract Forest Model Recognizing the type of a scene captured in a photograph can be cast as a classification task. The desired output is a discrete label: Problems Sky, dog, cat, related indoor, to outdoor, the automatic etc. or semi-automatic analysis of complex data such as photographs, videos, medical scans, text or genomic data can all be categorized into a relatively small set of prototypical machine learning tasks. Predicting the price of a house as a function of its distance from a good school may be thought of as a regression problem. In this case the output is a continuous variable. Interactive image segmentation may be thought of as a semisupervised problem, where the user s brush strokes define labeled data and the rest of image pixels provide already available unlabeled data. 9

10 Decision Tree Basics DT has been around for a number of years. 10

11 Decision Tree Basics DT has been around for a number of years. W 1 W 2 W

12 Decision Tree Basics DT has been around for a number of years. Their recent revival is mostly due to the discovery that higher accuracy on previously unseen data (i.e., generalization) 12

13 Decision Tree Basics A tree is a special type of graph: nodes and edges. 13

14 Decision Tree Basics A tree is a special type of graph: nodes and edges. Nodes->internal nodes (split): circle and Terminals (leaf):square 14

15 Decision Tree Basics A tree is a special type of graph: nodes and edges. Nodes->internal nodes (split): circle and Terminals (leaf):square -Tree cant contain loops -all nodes have one incoming edge (except root) 15

16 Decision Tree Basics Solve a complex problem by running simpler tests! 16

17 Decision Tree Basics Solve a complex problem by running simpler tests! Decision Tree: simpler tests are organized in a tree structure! 17

18 Decision Tree Basics Solve a complex problem by running simpler tests! Decision Tree: simpler tests are organized in a tree structure! Decision 18

19 Decision Tree Basics 19

20 Decision Tree Object Detection 20

21 Decision Tree Image Segmentation 21

22 Decision Tree Basics-Notations A data point feature v =(x 1,x 2,...,x d ) where x denotes in CV applications: v may be pixel, and x may be responses of a chosen filter-bank at that pixel! 22

23 Decision Tree Basics-Notations A data point feature v =(x 1,x 2,...,x d ) where x denotes in CV applications: v may be pixel, and x may be responses of a chosen filter-bank at that pixel! In theory, dimensionality of d can be very large! In practice, only some of the features are being used, d << d 23

24 Decision Tree Basics-Notations Set of tests: split functions (i.e., weak learner, test function) 24

25 Decision Tree Basics-Notations Set of tests: split functions (i.e., weak learner, test function) Formulate a test function at a split node j as a function of its binary outputs: h(v, j ):R d T! {0, 1} 25

26 Decision Tree Basics-Notations Set of tests: split functions (i.e., weak learner, test function) Formulate a test function at a split node j as a function of its binary outputs: h(v, j ):R d T! {0, 1} Split parameters Domain of Split parameters 26

27 Decision Tree Basics-Notations Set of tests: split functions (i.e., weak learner, test function) Formulate a test function at a split node j as a function of its binary outputs: h(v, j ):R d T! {0, 1} Split parameters Domain of Split parameters Training point/set: is a data point for which the attributes (we are seeking for) may be known and used to compute tree parameters: Example: a training set would be a set of photos with associated indoor/outdoor labels. 27

28 Decision Tree Basics-Notations Entire training set. 28

29 Decision Tree Basics-Notations S j j j = argmax 2T I(S j, ) S L j S R j Subset of training points reaching node j: S j Subsets going to the children of node j: S L j S R j 29

30 Decision Tree Basics-Notations S j j j = argmax 2T I(S j, ) S L j S R j Subset of training points reaching node j: S j Subsets going to the children of node j: S L j S R j S j = S L j [ S R j 30

31 Decision Tree Basics-Notations S j j j = argmax 2T I(S j, ) S L j S R j Subset of training points reaching node j: S j Subsets going to the children of node j: S L j S R j S j = S L j [ S R j ; = S L j \ S R j 31

32 Decision Tree Basics-Notations S j Off-line phase : training On-line phase : testing j j = argmax 2T I(S j, ) S L j S R j At each node j, we learn the function that best splits into and S j S L j S R j 32

33 Decision Tree Basics-Notations S j Off-line phase : training On-line phase : testing j j = argmax 2T I(S j, ) S L j S R j At each node j, we learn the function that best splits into and S j S L j S R j à Maximization of objection function h. 33

34 Weak Learner Models The split functions play a crucial role both in training and testing. 34

35 Weak Learner Models The split functions play a crucial role both in training and testing. Let us start with a simple geometric parameterization for weak learner model: =(,, ) 35

36 Weak Learner Models The split functions play a crucial role both in training and testing. Let us start with a simple geometric parameterization for weak learner model: =(,, ) Where ϕ is filter (selector) function, selecting some features out of the entire vector v 36

37 Weak Learner Models The split functions play a crucial role both in training and testing. Let us start with a simple geometric parameterization for weak learner model: =(,, ) Where ϕ is filter (selector) function, selecting some features out of the entire vector v ψ defines the geometric primitive used to separate data (axisaligned hyperplane, ) 37

38 Weak Learner Models The split functions play a crucial role both in training and testing. Let us start with a simple geometric parameterization for weak learner model: =(,, ) Where ϕ is filter (selector) function, selecting some features out of the entire vector v ψ defines the geometric primitive used to separate data (axisaligned hyperplane, ) τ captures thresholds for the inequalities used in the binary test. 38

39 Ex: Data Separation h(v, ) =[ 1 > (v). > 2 ] where [.] indicator function. Linear separation Non-Linear separation h(v, ) =[ 1 > (v) T (v) > 2 ] 39

40 Entropy and Information Gain The tree training phase is driven by the statistics of the training set. 40

41 Entropy and Information Gain The tree training phase is driven by the statistics of the training set. Entropy (H) and information gain (I) are basic building blocks of the training object function! 41

42 Entropy and Information Gain The tree training phase is driven by the statistics of the training set. Information gain: reduction in uncertainty level by splitting data arriving at the node into multiple child subsets I = H(S) Entropy (H) and information gain (I) are basic building blocks of the training object function! X i2{l,r} S i S H(Si ) 42

43 Example: Information gain for classification Note that vertical split produces purer class distribution in the two child nodes! 43

44 Example: Information gain for clustering Note that vertical split produces better class separation in the two child nodes! 44

45 Example: Information gain for clustering Note that vertical split produces better class separation in the two child nodes! H(S) = Z y2y p(y)log(p(y))dy 45

46 Classification SVM: Support Vector Machines Guarantees maximum margin separation for two-class problems Boosting: building strong classifiers as linear combination of many weak classifiers 46

47 Classification SVM: Support Vector Machines Guarantees maximum margin separation for two-class problems Boosting: building strong classifiers as linear combination of many weak classifiers Despite the success of SVMs and boosting, these techniques do not extent naturally to multiple class problems. 47

48 Classification SVM: Support Vector Machines Guarantees maximum margin separation for two-class problems Boosting: building strong classifiers as linear combination of many weak classifiers Despite the success of SVMs and boosting, these techniques do not extent naturally to multiple class problems. Classification forests work unmodified with any number of classes. 48

49 Classification SVM: Support Vector Machines Tracking keypoints in Video Human Pose estimation problems Gaze estimation Boosting: building Anatomy strong detection classifiers in CT as linear combination Scans of many weak classifiers Semantic Segmentation of Despite the success of SVMs boosting, these techniques do not extent photos naturally to and multiple videos class problems. 3D delineation of brain Classification lesions forests work unmodified with any number of classes. Guarantees maximum margin separation for two-class 49

50 1. Given a training set S 2. For i = 1 to k do: The Algorithm 3. Build subset S i by sampling with replacement from S 4. Learn tree T i from S i 5. At each node: 6. Choose best split from random subset of F features 7. Each tree grows to the largest extend, and no pruning 8. Make predictions according to majority vote of the set of k trees. 50

51 Ex: Non-imaging data (Baseball salary) Salary is color-coded from low (blue/green) to high (yellow/red) Numbers of years that he has played, and the number of hits that he made in the previous year. 51

52 Ex: Non-imaging data (Baseball salary) 52

53 Ex: Non-imaging data (Baseball salary) R 1 = {X Years < 4.5} R 2 = {X Years 4.5, Hits 117.5} R 2 = {X Years 4.5, Hits < 117.5} 53

54 Ex: Non-imaging data (Baseball salary) 54

55 Trees versus Linear Models 55

56 DEMO mo/demoforest.html 56

57 Face Detection Viola and Jones 2001: Landmark paper in Computer vision 1. A large number of Haar features. 2. Use of integral images. 3. Cascade of classifiers (Random Forest) 4. Boosting. 57

58 Using Mutual Information to Build Decision Trees I M (X, Y )=H(X) H(X Y ) 58

59 Lecture 14: Decision Forests for Computer Vision Body Part Classification (BPC) in Kinect 59

60 Application: Efficient Human Pose Estimation from Single Depth Images (Shotton et al) Fast and reliable estimation of pose of the human body from images has been goal of computer vision for many years (i.e., gaming, human-computer interaction, security, tele-presence, healthcare). From Microsoft kinect! BPC: body part classification 60

61 Application: Efficient Human Pose Estimation from Single Depth Images (Shotton et al) Synthetic and real data were used. When synthetic data is generated, then you have ground truth for evaluation! 61

62 Application: Efficient Human Pose Estimation from Single Depth Images (Shotton et al) 31 body parts were defined/labeled: LU/RU/LW/RW head, neck, L/R shoulder, LU/RU/LW/RW arm, L/R elbow, L/R wrist, L/R torso,. 62

63 Application: Efficient Human Pose Estimation from Single Depth Images (Shotton et al) Image Features: depth comparison features are extracted for a pixel position u=(u x,u y ): v(u) =(z(q 1 ),...,z(q i ),...,z(q d )) with 63

64 Application: Efficient Human Pose Estimation from Single Depth Images (Shotton et al) Image Features: depth comparison features are extracted for a pixel position u=(u x,u y ): v(u) =(z(q 1 ),...,z(q i ),...,z(q d )) with q i = u + i z(u) where z indicates depth, sigma denotes a 2D offset w.r.t the reference position u. 1/z assures that feature response is depth invariant 64

65 Application: Efficient Human Pose Estimation from Single Depth Images (Shotton et al) Yellow crosses indicate the reference image pixel u being classified. Red circles indicate the offset pixels (a) Two example features give a large depth difference response (b) The same two features at new image locations give a much smaller response 65

66 Application: Efficient Human Pose Estimation from Single Depth Images (Shotton et al) Recap: Selector function =(,, ) Fixed for all split: Weak learner function weak learners (v) =(v i,v j )withi, j 2 {1,...,d} =(1, 1) h(u; n )=[f(u; n, ) n ] f(u; n, ) = (v(u)). n: denotes any node in the tree If h evaluates to 0 -> path branches to the left, otherwise to right. Continue until reaching a leaf node. 66

67 BPC: Body Part Classification BPC predicts a discrete body part label at each pixel as an intermediate step towards predicting joint positions: p l (c) 1. Store Over body parts c at each leaf l. 2. For a given input pixel u, the tree is descended to reach leaf l=l(u) and the distribution is retrieved. The distributions are averaged together for all trees in the forest to give the final classification as: p(c u) = 1 T X l2l(u) p l (c) 67

68 BPC: Body Part Classification 68

69 Ex. Anatomy Detection Training: each bounding box includes one organ from CT images, and features are simply defined as absolute position of face of the box (6 faces, hence a vector of length 6) 69

70 Texton Forests for Image Categorization Johnson, Shotton, Cipolla Semantic texton forest (STF) are a form of random decision forests that can be employed to produce powerful low-level codewords for computer vision. 70

71 Texton Forests for Image Categorization Johnson, Shotton, Cipolla Semantic texton forest (STF) are a form of random decision forests that can be employed to produce powerful low-level codewords for computer vision. (a) Test image with ground truth, (b) a set of semantic textons, (c) a rough per-pixel classification, (d) categories 71

72 Texton Forests for Image Categorization Johnson, Shotton, Cipolla Semantic texton forest (STF) are a form of random decision forests that can be employed to produce powerful low-level codewords for computer vision. (a) Test image with ground truth, (b) a set of semantic textons, (c) a rough per-pixel classification, (d) categories Texton: fundamental microstructures in natural images (representation of small texture patches) 72

73 Texton Forests for Image Categorization (a) Semantic texton forest features. It simply includes raw image pixels within delta x delta Neighborhood (either raw values, summation of them, average, or absolute difference of pair Of pixels can be used as features). (b) Visualization of leaf nodes from one tree (distance delta=21 pixels). 73

74 Lecture 14: Decision Forests for Computer Vision Texton Forests for Image Categorization One texton map per tree categories Ground truth 74

75 Advantages / Disadvantages Advantages Improve predictive performance Other types of classifiers can be directly included Easy to implement No too much parameter tuning The combined classifier is not so transparent (black box) Not a compact representation Disadvantages 75

76 References Breiman, Pince, Computer Vision. Hasite et al., The Elements of Statistical Learning Criminisi and Shotton, Decision Forests, Springer. 76