Recognition. Computer Vision I. CSE252A Lecture 20

Transcription

1 Recognition CSE252A Lecture 20

2 Announcements HW 4 Due tomorrow (Friday) Final Exam Thursday, HW 3 to be returned at end of class. Final exam discussion at end of class. Stop me at 12:05!!

3 Dr.Kriegman, In lec7.pdf on page 2 where the BRDF with an apple is shown, it is stated: "ratio of incident irradiance to emitted radiance" The equation shows the ratio of emmitted radiance to incident irradiance instead, so the wording above should be reversed. -Fred

4 With assumptions in previous slide Bi-directional Reflectance Distribution Function ρ(θ in, φ in ; θ out, φ out ) BRDF Ratio of the emitted radiance to incident irradiance Function of Incoming light direction: θ in, φ in Outgoing light direction: θ out, φ out (θ in,φ in ) ^n (θ out,φ out ) ρ ( x θ, φ ; θ, φ ) CSE 252A, Winter 2006 ; = in in out out L i L ( ) o x; θout, φout ( x; θ, φ ) cosθ dω in in in

5 Motion wrap Normal Flow Aperture problem Lukas Kanade Tracking Last lecture

6 Recognition Given a database of objects and an image determine what, if any of the objects are present in the image.

7 Recognition Given a database of objects and an image determine what, if any of the objects are present in the image.

8 Bug Human/Felix Face Barbara Steele Camel Quadruped Problem: Recognizing instances Recognizing categories

9 Recognition Given a database of objects and an image determine what, if any of the objects are present in the image.

10 Object Recognition: The Problem Given: A database D of known objects and an image I: 1. Determine which (if any) objects in D appear in I 2. Determine the pose (rotation and translation) of the object Pose Est. (where is it 3D) Segmentation (where is it 2D) Recognition (what is it) WHAT AND WHERE!!!

11 Recognition Challenges Within-class variability Different objects within the class have different shapes or different material characteristics Deformable Articulated Compositional Pose variability: 2-D Image transformation (translation, rotation, scale) 3-D Pose Variability (perspective, orthographic projection) Lighting Direction (multiple sources & type) Color Shadows Occlusion partial Clutter in background -> false positives

12 Object Recognition Issues: How general is the problem? 2D vs. 3D range of viewing conditions available context segmentation cues What sort of data is best suited to the problem? Whole images Local 2D features (color, texture, 3D (range) data What information do we have in the database? Collection of images? 3-D models? Learned representation? Learned classifiers? How many objects are involved? small: brute force search large:??

13 A Rough Recognition Spectrum Appearance-Based Recognition (Eigenface, Fisherface) Shape Contexts Geometric Invariants Image Abstractions/ Volumetric Primitives Local Features + Spatial Relations Aspect Graphs 3-D Model-Based Recognition Function Increasing Generality

14 Sketch of a Pattern Recognition Architecture Image (window) Feature Extraction Feature Vector Classification Object Identity

15 Example: Face Detection Scan window over image. Classify window as either: Face Non-face Window Classifier Face Non-face

16 Pattern Classification Summary Supervised vs. Unsupervised: Do we have labels? Supervised Nearest Neighbor Bayesian Plug in classifier Distribution-based Projection Methods (Fisher s, LDA) Neural Network Support Vector Machine Kernel methods Unsupervised Clustering Reinforcement learning

17 Image as a Feature Vector x 2 x 1 x 3 Consider an n-pixel image to be a point in an n-dimensional space, x R n. Each pixel value is a coordinate of x.

18 Nearest Neighbor Classifier { R j } are set of training images. ID = arg min dist( R j j, I) I x 2 R 1 x 1 x 3 R 2

19 Comments Sometimes called Template Matching Variations on distance function (e.g. L 1, robust distances) Multiple templates per class- perhaps many training images per class. Expensive to compute k distances, especially when each image is big (N dimensional). May not generalize well to unseen examples of class. Some solutions: Bayesian classification Dimensionality reduction

20 Bayesian Classification Blackboard

21 Some loss may be inevitable: the minimum risk (shaded area) is called the Bayes risk

22 Finding a decision boundary is not the same as modeling a conditional density.

23 Plug-in classifiers Assume that distributions have some parametric form - now estimate the parameters from the data. Common: assume a normal distribution with shared covariance, different means; use usual estimates ditto, but different covariances; ditto Issue: parameter estimates that are good may not give optimal classifiers.

24 Example: Face Detection Scan window over image. Classify window as either: Face Non-face

25 Example: Finding skin Non-parametric Representation of CCD Skin has a very small range of (intensity independent) colors, and little texture Compute an intensity-independent color measure, check if color is in this range, check if there is little texture (median filter) See this as a classifier - we can set up the tests by hand, or learn them. get class conditional densities (histograms), priors from data (counting) Classifier is

26 Figure from Statistical color models with application to skin detection, M.J. Jones and J. Rehg, Proc. Computer Vision and Pattern Recognition, 1999 copyright 1999, IEEE

27 Receiver Operating Curve Figure from Statistical color models with application to skin detection, M.J. Jones and J. Rehg, Proc. Computer Vision and Pattern Recognition, 1999 copyright 1999, IEEE

28 Eigenfaces: linear projection An n-pixel image x R n can be projected to a low-dimensional feature space y R m by y = Wx where W is an n by m matrix. Recognition is performed using nearest neighbor in R m. How do we choose a good W?

29 Eigenfaces: Principal Component Analysis (PCA) Some details: Use Singular value decomposition, trick described in text to compute basis when n<<d

30 Singular Value Decomposition [ Repeated from Lecture 7] Any m by n matrix A may be factored such that A = UΣV T [m x n] = [m x m][m x n][n x n] U: m by m, orthogonal matrix Columns of U are the eigenvectors of AA T V: n by n, orthogonal matrix, columns are the eigenvectors of A T A Σ: m by n, diagonal with non-negative entries (σ 1, σ 2,, σ s ) with s=min(m,n) are called the called the singular values Singular values are the square roots of eigenvalues of both AA T and A T A & Columns of U are corresponding Eigenvectors Result of SVD algorithm: σ 1 σ 2 σ s

31 SVD Properties In Matlab [u s v] = svd(a), and you can verify that: A=u*s*v r=rank(a) = # of non-zero singular values. U, V give us orthonormal bases for the subspaces of A: 1st r columns of U: Column space of A Last m - r columns of U: Left nullspace of A 1st r columns of V: Row space of A 1st n - r columns of V: Nullspace of A For d<= r, the first d column of U provide the best d-dimensional basis for columns of A in least squares sense.

32 Thin SVD Any m by n matrix A may be factored such that A = UΣV T [m x n] = [m x m][m x n][n x n] If m>n, then one can view Σ as: ' 0 Where Σ =diag(σ 1, σ 2,, σ s ) with s=min(m,n), and lower matrix is (n-m by m) of zeros. Alternatively, you can write: A = U Σ V T In Matlab, thin SVD is:[u S V] = svds(a)

33 Performing PCA with SVD Singular values of A are the square roots of eigenvalues of both AA T and A T A & Columns of U are corresponding Eigenvectors n T T T And a a = [ a a a ][ a a L a ] = AA i= 1 Covariance matrix is: i i Σ = 1 1 n n 2 i= 1 L n 1 2 r r r ( x i μ)( x So, ignoring 1/n subtract mean image μ from each input image, create data matrix, and perform thin SVD on the data matrix. i r μ) T n

34 First Principal Component Direction of Maximum Variance Mean

35 Modeling Eigenfaces 1. Given a collection of n labeled training images, 2. Compute mean image and covariance matrix. 3. Compute k Eigenvectors (note that these are images) of covariance matrix corresponding to k largest Eigenvalues. (Or perform using SVD!!) 4. Project the training images to the k-dimensional Eigenspace. Recognition 1. Given a test image, project to Eigenspace. 2. Perform classification to the projected training images.

36 Eigenfaces: Training Images [ Turk, Pentland 01

37 Eigenfaces Mean Image Basis Images

38 Eigenfaces for a single individual, observed under variable lighting

39 Face detection using distance to face space Scan a window ω across the image, and classify the window as face/not face as follows: Project window to subspace, and reconstruct as described earlier. Compute distance between ω and reconstruction. Local minima of distance over all image locations less than some treshold are taken as locations of faces. Repeat at different scales. Possibly normalize windows intensity so that ω = 1.

40 Underlying assumptions Background is not cluttered (or else only looking at interior of object Lighting in test image is similar to that in training image. No occlusion Size of training image (window) same as window in test image.

41 Difficulties with PCA Projection may suppress important detail smallest variance directions may not be unimportant Method does not take discriminative task into account typically, we wish to compute features that allow good discrimination not the same as largest variance or minimizing reconstruction error.

42

43 Fisherfaces: Class specific linear projection P. Belhumeur, J. Hespanha, D. Kriegman, Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection, PAMI, July 1997, pp An n-pixel image x R n can be projected to a low-dimensional feature space y R m by y = Wx where W is an n by m matrix. Recognition is performed using nearest neighbor in R m. How do we choose a good W?

44 PCA & Fisher s Linear Discriminant Between-class scatter Within-class scatter Total scatter c ST = ( xk Where c is the number of classes μ i is the mean of class χ i χ i is number of samples of χ i.. c S B = χi ( μi μ)( μi μ) S W = i= 1 c i= 1 x χ i= 1 x χ k i k ( x i k μ )( x i μ)( x k k μ) T μ ) T i = T S B + S W χ 1 χ 2 μ 1 μ μ 2 If the data points are projected by y=wx and scatter of points is S, then the scatter of the projected points is W T SW

45 PCA & Fisher s Linear Discriminant PCA χ χ 2 1 FLD PCA (Eigenfaces) W PCA = arg maxw W Maximizes projected total scatter Fisher s Linear Discriminant W fld Maximizes ratio of projected between-class to projected within-class scatter W = arg max W W W T T T S S S T B W W W

46 Computing the Fisher Projection Matrix The w i are orthonormal There are at most c-1 non-zero generalized Eigenvalues, so m <= c-1 Can be computed with eig in Matlab

47 Fisherfaces W fld W = PCA W arg max W = W fldwpca Since S W is rank N-c, project training set to subspace T STW spanned by first N-c principal W components of the training set. T T Apply FLD to N-c W WPCASBWPCAW dimensional subspace yielding T c-1 dimensional feature space. = arg maxw W W T PCA S W W PCA W Fisher s Linear Discriminant projects away the within-class variation (lighting, expressions) found in training set. Fisher s Linear Discriminant preserves the separability of the classes.

48 PCA vs. FLD

49 Harvard Face Database 15 o 30 o 45 o 10 individuals 66 images per person Train on 6 images at 15 o Test on remaining images 60 o

50 Recognition Results: Lighting Extrapolation Error Rate degrees 30 degrees 45 degrees Light Direction Correlation Eigenfaces Eigenfaces (w/o 1st 3) Fisherface

51 Support Vector Machines Bayes classifiers and generative approaches in general try to model of the posterior, p( ω x) Instead, try to obtain the decision boundary directly potentially easier, because we need to encode only the geometry of the boundary, not any irrelevant wiggles in the posterior. Not all points affect the decision boundary [ From Marc Polleyfes]

52 Support Vector Machines Set S of points x i R n, each x i belongs to one of two classes y i {-1,1} The goals is to find a hyperplane that divides S in these two classes S is separable if w R n,b R y i ( w.x + b) 1 i Separating hyperplanes [ From Marc Polleyfes] d i d i = w.xi + b w w = Closest point y d i = w.x+ b = 0 w w 1 i

53 Support Vector Machines Optimal separating hyperplane maximizes Problem 1: Minimize Subject to 1 2 y w.w ( w.x + b) 1, i = 1,2 N i i,..., 1 w support vectors 2 w Optimal separating hyperplane (OSH) [ From Marc Polleyfes]

54 Solve using Lagrange multipliers Lagrangian L N = 2 α i i= 1 ( w, b,α) 1 w.w { y ( w.x + b) 1} i i at solution L = b L N i= 1 y i = w w i= 1 α = 0 N i α i yi x i = 0 therefore L N 1 N ( α) = i i j yi y j xi. x j α 0 α α α i= 1 2 i, j= 1 [ From Marc Polleyfes]

55 Decision function Once w and b have been computed the classification decision for input x is given by ( x) = sign( w. x ) f + b Note that the globally optimal solution can always be obtained (convex problem) [ From Marc Polleyfes]

56 Non-linear SVMs Non-linear separation surfaces can be obtained by non-linearly mapping the data to a high dimensional space and then applying the linear SVM technique Note that data only appears through vector product Need for vector product in high-dimension can be avoided by using Mercer kernels: K ( x, x ) = Φ( x ) Φ( x ) i i i i e.g. K ( y) ( x. y) p ( x, y) = ( x.y ) = x ( ) x 1 y 1 + x1x2 y1 y2 x2 y2 x, y exp 2 2 x, = (Polynomial kernel) K + K = Φ( 2 x ) = ( ) σx1, x1x2, x2 K ( x, y) = tanh( κ x. y δ ) (Radial Basis Function) (Sigmoïdal function) [ From Marc Polleyfes]

57 ( x, y) ( x 2, xy, y 2, x, y)= ( u 0,u 1,u 2,u 3,u 4 ) Space in which decision boundary is linear - a conic in the original space has the form au 0 + bu 1 + cu 2 + du 3 + eu 4 + f = 0 [ From Marc Polleyfes]

58 Variability: Camera position Illumination Internal parameters Within-class variations

59 Appearance manifold approach - for every object (Nayar et al. 96) 1. sample the set of viewing conditions 2. Crop & scale images to standard size 3. Use as feature vector - apply a PCA over all the images - keep the dominant PCs - Set of views for one object is represented as a manifold in the projected space - Recognition: What is nearest manifold for a given test image?

60 An example: input images

61 An example: basis images

62 An example: surfaces of first 3 coefficients

63 Parameterized Eigenspace

64 Recognition

65 Appearance-based vision for robot control [ Nayar, Nene, Murase 1994 ]

66 Limitations of these approaches Object must be segmented from background (How would one do this in non-trivial situations? Occlusion? The variability (dimension) in images is large, so is sampling feasible? How can one generalize to classes of objects?

67 Appearance-Based Vision: Lessons Strengths Posing the recognition metric in the image space rather than a derived representation is more powerful than expected. Modeling objects from many images is not unreasonable given hardware developments. The data (images) may provide a better representations than abstractions for many tasks.

68 Appearance-Based Vision: Lessons Weaknesses Segmentation or object detection is still an issue. To train the method, objects have to be observed under a wide range of conditions (e.g. pose, lighting, shape deformation). Limited power to extrapolate or generalize (abstract) to novel conditions.

69 Model-Based Vision Given 3-D models of each object Detect image features (often edges, line segments, conic sections) Establish correspondence between model &image features Estimate pose Consistency of projected model with image.

70 A Rough Recognition Spectrum Appearance-Based Recognition (Eigenface, Fisherface) Shape Contexts Geometric Invariants Image Abstractions/ Volumetric Primitives Local Features + Spatial Relations Aspect Graphs 3-D Model-Based Recognition Function

71 Recognition by Hypothesize and Test General idea Hypothesize object identity and pose Recover camera parameters (widely known as backprojection) Render object using camera parameters Compare to image Issues where do the hypotheses come from? How do we compare to image (verification)? Simplest approach Construct a correspondence for all object features to every correctly sized subset of image points These are the hypotheses Expensive search, which is also redundant.

72 Pose consistency Correspondences between image features and model features are not independent. A small number of correspondences yields a camera matrix --- the others correspondences must be consistent with this. Strategy: Generate hypotheses using small numbers of correspondences (e.g. triples of points for a calibrated perspective camera, etc., etc.) Backproject and verify

73

74 Figure from Object recognition using alignment, D.P. Huttenlocher and S. Ullman, Proc. Int. Conf. Computer Vision, 1986, copyright IEEE, 1986

75 Voting on Pose Each model leads to many correct sets of correspondences, each of which has the same pose Vote on pose, in an accumulator array This is a hough transform, with all it s issues.

76

77 Figure from The evolution and testing of a model-based object recognition system, J.L. Mundy and A. Heller, Proc. Int. Conf. Computer Vision, 1990 copyright 1990 IEEE

78 Figure from The evolution and testing of a model-based object recognition system, J.L. Mundy and A. Heller, Proc. Int. Conf. Computer Vision, 1990 copyright 1990 IEEE

79 Figure from The evolution and testing of a model-based object recognition system, J.L. Mundy and A. Heller, Proc. Int. Conf. Computer Vision, 1990 copyright 1990 IEEE

80 Figure from The evolution and testing of a model-based object recognition system, J.L. Mundy and A. Heller, Proc. Int. Conf. Computer Vision, 1990 copyright 1990 IEEE

81 Invariance Properties or measures that are independent of some group of transformation (e.g., rigid, affine, projective, etc.) For example, under affine transformations: Collinearity Parallelism Intersection Distance ratio along a line Angle ratios of tree intersecting lines Affine coordinates

82 Invariance - 1 There are geometric properties that are invariant to camera transformations Easiest case: view a plane object in scaled orthography. Assume we have three base points P_i (i=1..3) on the object then any other point on the object can be written as P k = P 1 + μ ka ( P 2 P 1 )+ μ kb ( P 3 P 1 ) Now image points are obtained by multiplying by a plane affine transformation, so p k = AP k = AP 1 + μ ka ( P 2 P 1 )+ μ kb P 3 P 1 = p 1 + μ ka ( p 2 p 1 )+ μ kb ( p 3 p 1 ) ( ( ) P 2 P k P 1 P 3 P k+1

83 Geometric hashing Vote on identity and correspondence using invariants Take hypotheses with large enough votes Building a table: Take all triplets of points in on model image to be base points P 1, P 2, P 3. Take ever fourth point and compute μ s Fill up a table, indexed by μ s, with the base points and fourth point that yield those μ s the object identity

84

85 Figure from Efficient model library access by projectively invariant indexing functions, by C.A. Rothwell et al., Proc. Computer Vision and Pattern Recognition, 1992, copyright 1992, IEEE

86 Verification Edge score are there image edges near predicted object edges? very unreliable; in texture, answer is usually yes Oriented edge score are there image edges near predicted object edges with the right orientation? better, but still hard to do well (see next slide) Texture e.g. does the spanner have the same texture as the wood?

87 Application: Surgery To minimize damage by operation planning To reduce number of operations by planning surgery To remove only affected tissue Problem ensure that the model with the operations planned on it and the information about the affected tissue lines up with the patient display model information supervised on view of patient Big Issue: coordinate alignment, as above

88 MRI CTI NMI USI Reprinted from Image and Vision Computing, v. 13, N. Ayache, Medical computer vision, virtual reality and robotics, Page 296, copyright, (1995), with permission from Elsevier Science

89 Figures by kind permission of Eric Grimson; further information can be obtained from his web site

90 Figures by kind permission of Eric Grimson; further information can be obtained from his web site

91 Figures by kind permission of Eric Grimson; further information can be obtained from his web site

92 A Rough Recognition Spectrum Appearance-Based Recognition (Eigenface, Fisherface) Shape Contexts Geometric Invariants Image Abstractions/ Volumetric Primitives Local Features + Spatial Relations Aspect Graphs 3-D Model-Based Recognition Function

93 Matching using Local Image features Simple approach Detect corners in image (e.g. Harris corner detector). Represent neighborhood of corner by a feature vector produced by Gabor Filters, K-jets, affineinvariant features, etc.). Modeling: Given an training image of an object w/o clutter, detect corners, compute feature descriptors, store these. Recognition time: Given test image with possible clutter, detect corners and compute features. Find models with same feature descriptors (hashing) and vote.

94 Figure from Local grayvalue invariants for image retrieval, by C. Schmid and R. Mohr, IEEE Trans. Pattern Analysis and Machine Intelligence, 1997 copyright 1997, IEEE

95 Write Probabilistic interpretation Assume Likelihood of image given pattern

96 Employ spatial relations Figure from Local grayvalue invariants for image retrieval, by C. Schmid and R. Mohr, IEEE Trans. Pattern Analysis and Machine Intelligence, 1997 copyright 1997, IEEE

97 Figure from Local grayvalue invariants for image retrieval, by C. Schmid and R. Mohr, IEEE Trans. Pattern Analysis and Machine Intelligence, 1997 copyright 1997, IEEE

98 Example Training examples Test image

99 Finding faces using relations Strategy: Face is eyes, nose, mouth, etc. with appropriate relations between them build a specialised detector for each of these (template matching) and look for groups with the right internal structure Once we ve found enough of a face, there is little uncertainty about where the other bits could be

100 Finding faces using relations Strategy: compare Notice that once some facial features have been found, the position of the rest is quite strongly constrained. Figure from, Finding faces in cluttered scenes using random labelled graph matching, by Leung, T. ;Burl, M and Perona, P., Proc. Int. Conf. on Computer Vision, 1995 copyright 1995, IEEE

101 Figure from, Finding faces in cluttered scenes using random labelled graph matching, by Leung, T. ;Burl, M and Perona, P., Proc. Int. Conf. on Computer Vision, 1995 copyright 1995, IEEE

102 Even without shading, shape reveals a lot - line drawings

103 Scene Interpretation The Swing Fragonard, 1766

104 Final Exam Closed book One cheat sheet Single piece of paper, handwritten, no photocopying, no physical cut & paste. What to study Basically material presented in class, and supporting material from text If it was in text, but NEVER mentioned in class, it is very unlikely to be on the exam Question style: Short answer Some longer problems to be worked out.