Visuelle Perzeption für Mensch- Maschine Schnittstellen

Transcription

1 Visuelle Perzeption für Mensch- Maschine Schnittstellen Vorlesung, WS 2008 Dr. Rainer Stiefelhagen Dr. Edgar Seemann Interactive Systems Laboratories Universität Karlsruhe (TH) Edgar Seemann, Interactive Systems Laboratories, Universität Karlsruhe (TH)

2 Basics: Pattern Recognition WS 2008/09 Dr. Edgar Seemann Edgar Seemann,

3 Organisatorisches Gruppe 1: Christian Johner Mike Morante Patrick Mehl Gruppe 2: Thomas Stephan (Java) Steffen Braun (Java) Gruppe 3: Martin Wagner Hilke Kieritz Jan Hendrik Hammer Gruppe 4: Wenlei Wu Chengchao Qu Gruppe 5: Michael Weber Tomas Semela Dennis Kopcan Gruppe 6: Johann Korndoerfer Daniel Koester Daniel Putsch Gruppe 7 Benjamin Bartosch Thomas Lichtenstein frei: Igor Plotkin Adrian Genaid Gruppe 8 Florian Krupicka Mathias Luedtke Edgar Seemann,

4 Edgar Seemann, Question from last lecture multiplication of the two projection matrices: u v 1 = k u k v u 0 v 0 1 f f f x y z 1 u v 1 = k u f 0 u 0 0 k v f v x y z 1

5 Separate projections were defined as x y z x y z 1 u v = 1 fx fy fz z = k u k v f f f u 0 v 0 1 x y z 1 x y z 1 Edgar Seemann,

6 Termine (1) Termine Thema Introduction, Applications Basics: Image Processing Basics: Image Transformations, 2D Structure Basics: Pattern recognition Computer Vision: Tasks, Challenges, Learning, Performance measures Face Detection I: Color, Edges (Birchfield) Project 1: Intro + Programming tips Project 1: Questions Face Detection II: ANNs, SVM, Viola & Jones Face Recognition I: Traditional Approaches, Eigenfaces, Fisherfaces, EBGM Face Recognition II Head Pose Estimation: Model-based, NN, Texture Mapping, Focus of Attention Project 1: Student Presentations, Project 2: Intro People Detection I People Detection II People Detection III (Part-Based Models) Scene Context and Geometry I (Ground-Plane, Hoiem, Leibe) Edgar Seemann,

7 Today Why pattern recognition and what is it? Dimensionality Reduction Principle Component Analysis Classification Bayes Decision Theory k-nn Kd-trees Ball trees Edgar Seemann,

8 Overview Learn common patterns based on either a priori knowledge or on statistical information Why statistics? Manual definition of patterns often tedious or not obvious Important for the adaptability to different tasks/domains Finally: Try to mimic human learning / better understand human learning Edgar Seemann,

9 Pattern Recognition Sensor Representation pattern Supervised Data samples with associated class labels Unsupervised Feature Selection/ Extraction Data samples WITHOUT any labels Feature pattern Classifier Decision Semi-Supervised / Weakly-Supervised Learning Not a topic in this lecture Edgar Seemann,

10 Pattern Recognition Stages 1. Feature Extraction Which part of the data is most important 2. Classification/Learning How can we map the extracted features to our desired output (supervised learning)? = x y in [a,b,c, ] w Edgar Seemann,

11 Examples Speech recognition w Computer Vision You will see many examples in the course of this semester Edgar Seemann,

12 What s in the box? Parametric/Non-parametric distributions Support Vector Machines Neural Networks Decision Trees. Edgar Seemann,

13 Problems and Considerations Features How do I encode domain knowledge? Allow invariance (e.g. rotation in the case of images) see e.g. previous lecture Which part of the data can be discarded as it represents redundant information? How can we reduce the dimensionality? i.e. how can we make the problem as simple as possible, but as complex as necessary Edgar Seemann,

14 Curse of Dimensionality Generally: the more dimensions, the more difficult for a learning algorithm to extract patterns More dimensions = more degrees of freedom Decision boundary Class A Class B Defining a linear decision boundary: 2-dim: 3 degrees of freedom 3-dim: 5 degrees of freedom 4-dim: 7 degrees of freedom Edgar Seemann,

15 Dimensionality Reduction Reduce the dimensionality of the data, while retaining relevant information Popular techniques: Principal Component Analysis (PCA) Linear Discriminant Analysis (LDA) Multidimensional Scaling (MDS) Many of these techniques are readily available in MATLAB or Octave Edgar Seemann,

16 PCA Edgar Seemann,

17 First Steps: Sample Variance Given a set of samples with mean a' An unbiased variance estimator is defined as with the sequence with zero mean written in vectors ( a 1,..., a n ) 1 2 var( a) = ( a a') = i n 1 n 1 ( z 1,..., z n ) 1 n 1 T zz 2 z i Edgar Seemann,

18 First Steps: Covariance Given two sets of samples with means a', b' A covariance estimator is defined as cov( a, b) with sequences with zero mean written in vectors ( a 1,..., a n ) 1 = ( ai a')( bi b' ) n 1 ( c1,..., cn),( d1,..., dn) 1 cd n 1 1 n 1 = c i d i Edgar Seemann, T ( b 1,..., b n )

19 First Steps: Covariance Matrix Given n sets samples v i with means and d being the dimensionality A multi-dimensional covariance estimator is defined as Written in vectors ' v 1,...,v n v = i ( a 1,..., a d 1 cov( V ) = ( vi vi ')( v j v j ') n 1 1 n 1 VV Edgar Seemann, T ) T

20 Variance Properties Properties: var(a)=cov(a,a) cov(a,b)=0 iff a,b are completely uncorrelated cov(v) is a square-symmetric matrix (containing variances on the diagonal and covariances on the off-diagonals) var( v.. 1 )... cov( v.. 1, v n ) cov( v n, v 1 ).. var( v n ) Edgar Seemann,

21 PCA: Toy example Toy example: Try to understand the motion of a spring Data from three camera sensors (x,y-position for each camera, i.e. 6-dim. data) Edgar Seemann,

22 Noise and Redundancy Data contains noise Data is redundant 1-dimensional motion vs. 6-dimensional sample data Sensor data from the three cameras are highly correlated Edgar Seemann,

23 Change of basis Basis Change By changing the basis, we may represent the data almost perfectly with a lower number of dimensions Remember that the data has a mean of zero r 1 r 2 Edgar Seemann,

24 Change of Basis Basis change can be written as matrix multiplication Given the new basis vectors p 1,..., p n we can transform data samples x i in the following manner i.e. we are projecting x i onto the new basis vectors Edgar Seemann,

25 Reducing the co-variances Remember the covariance matrix: We have seen that a basis change may reduce the correlation between the different dimensions Goal of PCA var( v.. cov( v Make covariance matrix as diagonal as possible n 1 ), v 1 )..... cov( v 1.. var( v, v n ) n ) Edgar Seemann,

26 PCA Assumptions Basis (principal components) is orthogonal Linearity Change of basis is a linear operation For non-linear problems: kernel PCA Mean and variance are sufficient statistics This holds if data is distributed according to a Gaussian distribution Large variances have important dynamics Edgar Seemann,

27 Finally solving it Theorem: The covariance matrix is diagonalized by an orthogonal matrix of its eigenvectors That is the principal components of the underlying data are the eigenvectors The higher the eigenvalue the more variance is captured along the dimension Edgar Seemann,

28 Eigenspectrum and Energy Edgar Seemann,

29 PCA in practice Eigenvectors can be computed by solving the linear equation system Ax = λx In practice, we most often use a singular value decomposition (SVD) If the data samples are high dimensional, the covariance matrix can get extremely large Let the sample x i be images with a resolution of 1600x1200 then the covariance would be a x matrix Computationally, computing the eigenvectors would be very costly Edgar Seemann,

30 Algebra to the rescue Fortunately, it ain t that bad If we have a maximum of n data samples, we can also obtain only a maximum of n eigenvectors (one dimension for each sample point) i.e. the number of eigenvectors is restricted by both the sample dimension and the sample number Let s have a look at the following equations: A T AA AA Ax = λx T T Ax = Aλx ( Ax) = λ( Ax) λ is eigenvalue of A T A λ is eigenvalue of AA T We can compute the eigenvectors of A T A and multiply them by A Edgar Seemann,

31 Summary PCA can be used to reduce the number of dimensions Find the principal dimensions Principal dimensions are assumed to have a high variance discard non-informative dimensions (redundancy and noise) Extensions: Robust PCA (deal with occlusion) Incremental PCA Kernel PCA (non-linear) Edgar Seemann,

32 Pattern Classification Edgar Seemann,

33 Bayes Decision Theory Example: Character Recognition Classify characters into the classes C 1 =a and C 2 =b minimizing the probability of an incorrect classification Edgar Seemann,

34 Priors Prior probabilities represent the probability of a class to occur Often used to encode domain knowledge Edgar Seemann,

35 Conditional Probability Given a feature vector x x measuring e.g. the width or height of the character The conditional probability p( x Ck ) measures the probability of observing x when the character is of class C k Edgar Seemann,

36 Example Say x = 15 x = 15 What decision should be made? Edgar Seemann,

37 Example Say x = 25 x = 25 What decision should be made? Edgar Seemann,

38 Example Say x = 20 x = 20 What decision should be made? Edgar Seemann,

39 Bayes Theorem The a-posteriori probability ( x C p k defines the probability of a class given a specific feature vector x ) Edgar Seemann,

40 Decision boundary Decision boundary Edgar Seemann,

41 Decision Rule Choose C 1 iff this is equivalent to this is equivalent to p ( C1 x) > p( C2 x) p ( x C1) p( C1) > p( x C2) p( C2) p( x C p( x C 1 2 ) ) > p( C p( C 2 1 ) ) Special cases: C ) p( ) and p = p x C ) = p( x ) ( 1 C2 ( 1 C2 Edgar Seemann,

42 More Classes We can generalize this rule to multi-class problems (e.g. all letters from the alphabet) Choose the class with the highest a-posteriori probability Edgar Seemann,

43 Example Edgar Seemann,

44 Generalization with Loss-Function In some applications we may consider the loss of a misclassification E.g. medical applications Loss if classified as healthy despite a disease: Loss if classified as ill despite being healthy: We assume: λ ( healthy ill) >> λ( ill healthy) λ( healthy ill) λ( ill healthy) Edgar Seemann,

45 Expected Loss We have an observation x, but do not know the loss function λ for all x: λ( x) = α i? λ ( α i C ) = λ with α i the possible decision and C k the possible classes The expected loss can be computed in the following manner: k ik Edgar Seemann,

46 Example Let s look at a two class example: We want to minimize the expected loss, i.e. we choose α i when > Edgar Seemann,

47 Some simple math Edgar Seemann,

48 Linear Discriminant Functions Perceptron Algorithm Edgar Seemann,

49 Linear Discriminant Functions Separate two classes with a linear hyper plane T With w the normal vector of the hyper plane Examples: Perceptron Linear SVM T y( x) = w x + w 0 w 0 T w Edgar Seemann,

50 Perceptron Algorithmus[Rosenblatt 58] Preconditions: Data set is linear separable Goal Find a separating hyper plane Idea Iteratively improve solution Only update solution if the data sample of interest is classified incorrectly Edgar Seemann,

51 Perceptron Algorithm 1. Initialize w = 0 2. Classify a new data sample with the following inner product y(x) = sign(w T x) 3. If correct, then goto step 2 else and w = w - y(x)x 4. If no errors left, then done else goto step 2 Edgar Seemann,

52 Algorithm Visualized Edgar Seemann,

53 Why does this work? Given misclassified sample x with y(x)=1 Then: w = w - x If we now classify the sample again: y(x) = (w-x) T x = w T x x T x x T x is positive and therefore the next prediction for this data sample will be closer to the correct value Convergence: Does not converge if data is not separable Edgar Seemann,

54 Wait We did not consider w 0 What happens if separating hyper plane does not pass through (0,0) Small trick: Choose x to be n+1 dimensional with n+1 dimension always being 1, then Becomes T y( x) = w x + w y( x) = w T x 0 Edgar Seemann,

55 Instance-Based Learning Edgar Seemann,

56 Instance-based Learning Learning=storing all training instances Classification=assigning target function to a new instance Referred to as Lazy learning Instance Examples: Template-Matching K-Nearest Neighbor Edgar Seemann,

57 1-Nearest-Neighbor Features All instances correspond to points in an n-dimensional Euclidean space Classification done by comparing feature vectors of the different points Edgar Seemann,

58 3-Nearest-Neighbor Search for the 3 nearest neighbors Typically majority vote, when not all neighbors are from the same class Edgar Seemann,

59 3-Nearest-Neighbor Decision surface Described by the Voronoi diagram Voronoi diagram is the dual of the Delaunay triangulation (computable in O(nlogn)) Edgar Seemann,

60 When to Consider Nearest Neighbor? Lots of training data Less than 20 attributes (dimensions) per example Advantages: Training is very fast Learn complex target functions Don t lose information Disadvantages: Slow at query time Easily fooled by irrelevant attributes Edgar Seemann,

61 K-Nearest Neighbors Typically decision surface is not computed, but for each new test sample, we compute the nearest neighbors on the fly Probabilistic interpretation: estimate density in a local neighborhood There are efficient memory indexing techniques in order to retrieve the stored training examples kd-tree [Friedman et al. 1977] Ball-tree Edgar Seemann,

62 KD Tree for NN Search Each node contains Children information The tightest box that bounds all the data points within the node. Edgar Seemann,

63 NN Search by KD Tree Edgar Seemann,

64 NN Search by KD Tree Edgar Seemann,

65 NN Search by KD Tree Edgar Seemann,

66 NN Search by KD Tree Edgar Seemann,

67 NN Search by KD Tree Edgar Seemann,

68 NN Search by KD Tree Edgar Seemann,

69 NN Search by KD Tree Edgar Seemann,

70 Standard Kd-tree construction Choose splitting planes by cycling through the dimensions Example: Root node: split in x-direction Next level: split in y-direction Next level: split in z-direction The position of the splitting plane is chosen to be the median of the points (with respect to their coordinates in the axis being used) Typically generates a quite balanced tree Edgar Seemann,

71 Kd-tree construction There exist many (sometimes application specific) heuristics that try to speed up kd-tree creation E.g. split in the dimension of the highest variance Kd-trees can be built incrementally Useful for incremental learning (for example when exploring a new territory with a robot) Existing libraries: C++: libkdtree++ Edgar Seemann,

72 A kd-tree: level 1 Edgar Seemann,

73 A kd-tree: level 2 Edgar Seemann,

74 A kd-tree: level 3 Edgar Seemann,

75 A kd-tree: level 4 Edgar Seemann,

76 A kd-tree: level 5 Edgar Seemann,

77 A kd-tree: level 6 Edgar Seemann,

78 Complexity Building a static kd-tree from n points: O(n log n) O(log n) tree levels, O(n) median search Insertion into a balanced kd-tree: O(log n) Removal from a balanced kd-tree O(log n) Query of an axis-parallel range in a balanced kdtree: O(n 1-1/d +k) with k the number of the reported points, and d the dimension of the kd-tree. Edgar Seemann,

79 Problem of Dimensionality Imagine instances described by 20 attributes, but only 2 are relevant to target function Curse of dimensionality: nearest neighbor is easily mislead when high dimensional X Edgar Seemann,

80 Ball Trees Can be constructed in a similar fashion as kd-trees Ball trees have shown to be superior to kd-trees in many applications (though there is high variance and dataset dependence) The Proximity Project [Gray, Lee, Rotella, Moore 2005] Edgar Seemann,

81 A ball-tree: level 1 Edgar Seemann,

82 A ball-tree: level 2 Edgar Seemann,

83 A ball-tree: level 3 Edgar Seemann,

84 A ball-tree: level 4 Edgar Seemann,

85 A ball-tree: level 5 Edgar Seemann,

86 knn Discussion Highly effective inductive inference method for noisy training data and complex target functions Target function for a whole space may be described as a combination of less complex local approximations Learning is very simple Classification can be time consuming, if training data set is large: O(logn) Edgar Seemann,

87 End of Lecture Edgar Seemann,