Machine Learning for. Artem Lind & Aleskandr Tkachenko

Transcription

1 Machine Learning for Object Recognition Artem Lind & Aleskandr Tkachenko

2 Outline Problem overview Classification demo Examples of learning algorithms Probabilistic modeling Bayes classifier Maximum margin classification SVM

3 Object Recognition as a Classification Task

4 Typical Workflow 1. Preprocessing Images 2. Extracting features 3. Learning classifier 4. Assessing results

5 Object representation Vectors of quantitative descriptors (length, weight, area) raw pixel values String and trees of structural descriptors Capture spatial relationships between features

6 Object representation Vectors of quantitative descriptors (length, weight, area) raw pixel values String and trees of structural descriptors Capture spatial relationships between features

7 Classification demo: Iris dataset Sepal Petal

8 Classification demo: Iris dataset Two classes: Iris-versicolor, Iris-virginica Two features: sepal length, petal length

9 Iris dataset: feature vector representation

10 Classification demo: linear classification

11 Classification demo: linear classifier How good is my classifier?

12 Classifier evaluation Confusion Matrix a b <-- classified as 44 6 a = Iris-versicolor True positives False negatives False positives 8 42 b = Iris-virginica True Negatives

13 Classifier evaluation Confusion Matrix a b <-- classified as 44 6 a = Iris-versicolor True positives False negatives False positives 8 42 b = Iris-virginica True Negatives Precision = TP /(TP +FP ) Recall = TP /(TP +FN) Accuracy = (TP + TN) /( FP + FN) F-Measure = harmonic_mean(precision, Recall)

14 Classifier evaluation Confusion Matrix a b <-- classified as 44 6 a = Iris-versicolor 8 42 b = Iris-virginica Summary Correctly Classified Instances % Incorrectly Classified Instances % Total Number of Instances 100 Detailed Accuracy By Class TP Rate FP Rate Precision Recall F-Measure ROC Area Class Iris-versicolor Iris-virginica

15 Classifier evaluation Most training algorithms optimize Accuracy / Precision / Recall for the given data However, we want classifier to perform well on unseen data This makes algorithms and theory way more complicated. This makes validation somewhat more complicated.

16 Proper validation Split the data: Training set Test set Validation Crossvalidation -If the data is scarce

17 Common workflow Summary: Get a decent dataset Identify discriminative features Train your classifier on the training set Validate on the test set

18 Classifiers Probabilistic modeling Bayesclassifier Margin maximization Support Vector Machines

19 Bayesclassifier (for binary classification) Learning: Estimate from the training data P(Class X) Classifying: Bayes Decision Rule: Predict C1, if P( C1 X) > P(C2 X); otherwise C2

20 Bayesclassifier Predict C1, if P( C1 X) > P(C2 X);otherwise C2 P( X C1) P(C1) / P(X) > P(X C2)P(C2) / P(X) P( X C1) P(C1) > P(X C2)P(C2)

21 Bayesclassifier Predict C1, if P( X C1) P(C1) > P(X C2) P(C2);otherwise C2

22 Bayesclassifier If P( X Class)and P(Class)are known, then the classifier is optimal. In practice, distributions P(X Class) and P(Class)are unknown and need to be estimated from the data: P(Class): Assign equal probability to all classes Use prior knowledge P(C) = #examples_in_c/ #examples P(X Class): Some well-behaved distribution expressed in an analytical form. Parameters are estimated based on data for each class. The closer this assumption is to reality, the closer the Bayes classifier approaches the optimum.

23 Bayesclassifier for Gaussian patterns p( x c ) = N( µ, σ ) i i i ( x µ ) i 1 2 i p x c P c e P c i 2πσ 2σ ( i ) ( i ) = ( i ), = 1,2 µ and σ i 2 i i 2 are sample mean and variance for the class i.

24 Example from the book

25 Summary Advantages: Creates linear boundaries which are simple to compute. Easy to implement. Disadvantages: is based on a single prototype per class (class center) which is often insufficient in practice. Usually won t work well for too many classes.

26 Support Vector Machines

27 SVM: Maximum margin classification

28 SVM: Maximum margin classification

29 SVM: Maximum margin classification

30 SVM: Maximum margin classification

31 SVM: Maximum margin classification

32 SVM: Maximum margin classification Hyperplane: For any point x, its distance to the hyperplane: Assuming all points are classified correctly: The margin is then:

33 Maximizing the margin Find (w, b) such that Margin is maximal, can be shown to be equivalent with: This is doable using efficient optimization algorithms.

34 SVM: Non separable case

35 SVM: How to choose Cparameter? Cparameter penalizes training points within the margin Large C-value can lead to over-fitting. Cross-validation + Grid search Ad hock: Find C-values which give you zero errors on the training set.

36 Nonlinear case

37 Solution: a nonlinear map Instead of classifying points x, we ll classify points in a higher-dimensional space.

38 Kernel methods For nearly any linear classifier: Trained on a dataset The resulting vector w can be represented as: Which means:

39 Kernel methods

40 Kernel methods Function Kis called a kernel, it measures similarity between objects. The computation of is unnecessary. You can use any type of data. Your method is nonlinear. Any linear method can be kernelized. Kernels can be combined.

41 Basic kernels How to choose the kernel?

42 SVM: more that 2 classes

43 SVM in practice

44 SVM Summary SVMs are maximum margin classifiers. SVMs can handle non-separable data. SVMs are known to perform well in high dimensional problems with few examples. Depending on the kernel, SVMs can be slow during classification Kernelizable

45 References Massachusetts Institute of technology course: Pattern Recognition for Machine Vision A Practical Guide to Support Vector Classication Konstantin Tretyakov slides on datamining

46 Demo