Mathematically, supposing the object of interest to be a Lambertian (or matte) surface, the amount of intensity reected when illuminated by a single l

Transcription

1 Proceedings of the Third International Conference \Communicating by Image and Multimedia (Imagecom'96) 20{22 May 1996, Arcachon, France. Generalized Matching for Recognition and Retrieval in an Image Database Chahab Nastar 1, Baback Moghaddam 2 and Alex Pentland 2 1 INRIA Rocquencourt, B.P. 105, Le Chesnay Cedex, France. 2 MIT Media Laboratory, 20 Ames St., Cambridge, MA 02139, USA. Abstract We describe a novel approach for image matching based on deformable intensity surfaces. In this approach, the intensity surface of the image is modeled as a deformable 3D mesh in the (x; y; I(x; y)) space. Each surface point has 3 degrees of freedom, thus capturing ne surface changes. A set of representative deformations within a class of objects (e.g. faces) are statistically learned through a Principal Components Analysis, thus providing a priori knowledge about object-specic deformations. We demonstrate the power of the approach by recognition and interactive search in a large-scale face database. keywords : Image Matching, Image warping, Deformable Surfaces, Statistical Learning, Content-based Retrieval. 1 Introduction In the next century, image is bound to become the most important component of multimedia and telecommunications. While low-level techniques have now reached their upper limits, the new technologies related to archiving, retrieving, and editing images have to integrate high-level techniques of computer vision. Consider for instance the problem of image retrieval in a large database, a central problem in the eld of computer vision [18, 20]. The user selects an image and asks the computer to nd more images that are in some sense similar to the probe image. This is a very hard problem that has multiple (although partial) solutions, depending on the type and the content of the image. A matching is usually performed to align the probe image to every image in the database, and some metric is minimized for grading the images in the database from the most similar image to the less similar image. In this article, we introduce deformable intensity surfaces for object matching and recognition. Our method allows to deal eectively with object appearance variations. The basic idea is to warp the intensity surface of the test image into the intensity surface of the probe image by using a physics-based model [14, 13], and then measure the low-order strain energy of the deformable surface for recognition. We further show how to tailor the space of allowable XY I deformations to t the actual variation found in individual target classes. This is accomplished by a statistical analysis of observed image-to-image deformations using a Principal Components Analysis. The result is that the image deformation is restricted to the subspace of physically-plausible deformations. In the process, the dimensionality of the matching and the numerical complexity of the governing equation are drastically reduced [15, 11]. By considering only the low-dimensional subspace of plausible deformations, we make the image matching process more robust and more ecient. We in eect \build in" statistical a priori knowledge about how the object can vary in order to obtain the best image-to-image match possible. We choose face recognition as an application for our method [4]. For over 20 years, automatic face recognition has been the center of attention of computer vision researchers. Face recognition is indeed a dicult object recognition problem, because of the wide variation in the appearance of a specic face due to changes in pose, lighting, and facial expression. The applications are numerous, e.g. human-computer interaction, smart surveillance and image retrieval in a database (see [4] for a survey). Our system integrates these abilities and produced a recognition rate of 97% in a database of 7,562 faces[16], a signicant improvement over previously reported results [17]. 2 Unifying shape and texture Image matching is the prerequisite to most object recognition techniques. The matching process denes a metric for image comparison in terms of similarity. Object recognition techniques are classically in one of the following categories : Geometric representations focus on the spatial distribution of image features. Photometric representations model image texture variations. Geometric and photometric approaches are complementary ; while geometric feature matching operates on the 2D shape of the object in the image (e.g. optic ow [8]), photometric approaches model texture variations assuming shape alignment [21]. Current work in the area of image-based object modeling deals with the shape (2D) and texture (grayscale) components of an image in an independent manner [3, 5]. Our novel representation combines both the spatial (XY) and grayscale (I) components of the image into a 3D surface (or manifold) and then eciently solves for a dense correspondance map in the XY I space. This \manifold matching" technique can be viewed as a more general formulation for image correspondance which, unlike optical ow, does not require a constant brightness assumption.

2 Mathematically, supposing the object of interest to be a Lambertian (or matte) surface, the amount of intensity reected when illuminated by a single light source placed at innity, is isotropic : I(x) S I(x; y) = ~ N(x; y): ~ L (1) where ~ N(x; y) is the surface normal vector at point (x; y), ~ L is the light source vector, and is a positive scalar. This equation directly links shape ~ N(x; y) and intensity surface I(x; y) (gure 1). If the shape is relatively smooth, we can represent the image intensity as a continuous surface : (x; y) I(x; y) (2) S Figure 2: 1D representation of S being pulled towards S 0 x Our system focuses on recognition in the 3D space dened by (x; y; I(x; y)), that we will call the XY I space. a b Figure 1: An image and its XY I representation 3 Intensity surface matching Many surface matching techniques are available in the computer vision literature. For instance, we can point out geometric surface matching based on dierential features [6] and the dual approach of physicallybased matching [14]. As we shall see, physically-based matching has a nice compression ability. 3.1 Surface modeling In this section, we model the intensity surface S in order to deform it to the intensity surface S 0. The proposed approach is described in [14]. Intensity surface S is modeled by a deformable mesh of size n n 0 nodes. It is ruled by Lagrangian dynamics : M U + C _U + KU = F(t) (3) c e d f where U = [: : : ; x i; y i; z i; : : :] T is the nodal displacement eld, and M, C, K are respectively the mass, damping, and stiness matrix of the system. F is the external force that has S attracted by S 0. On each node of S, the external force F points to the closest point on S 0 (gure 2). The approach is similar in spirit to \Iterative Closest Point" methods [2, 22]. Figure 3 illustrates four warping experiments. The rst example is the warping of two dierent faces. The three other outline facial expression, changes in lighting, and out-of-image plane rotation. g Figure 3: XY I warps (original image and warped image). Appearance variations are due to : a. b. identity. c. d. facial expression. e. f. lighting. g. h. pose. h

3 3.2 Compression ability It is possible to compress the deformation by a mechanical engineering technique called modal analysis [1, 19, 14, 13]. In this technique, the displacement eld is expressed in the vibration basis of the system. Modal decomposition with high frequency modes neglected is : U(t) PX X P 0 p=1 p 0 =1 ~u(p; p 0 ; t) (p; p 0 ) (4) where P P 0 n n 0 and the scalars ~u are the deformation spectrum coecient also known as modal amplitudes. An analytic expression of the modes is available for surfaces having the topology of the intensity surface [13] : (p; p 0 ) = [: : : ; cos p(2i? 1) 2n cos p0 (2j? 1) 2n 0 ; : : :] T (5) where i and j are surface parameters. Modes have to be normalized to unity. Eigenfrequencies are :! 2 (p; p 0 ) = 4K=M (sin 2 p 2n + sin2 p 0 2n 0 ) (6) where K is the stiness constant and M the nodal mass. 4 Similarity metric The strain energy of the surface has a simple expression in the vibration basis of the system with high frequencies neglected : E strain(p; P 0 ) = 1 2 PX X P 0 p=1 p 0 =1! 2 (p; p 0 ) ~u 2 (p; p 0 ) (7) with P P 0 n n 0. We observe that facial expression is a highfrequency phenomenon while change of lighting and pose are low-frequency phenomenons for the intensity surface (gure 4). Thus, given a probe face, we can retrieve the same (or a similar) face in an image database when the deformation spectrum is either low-frequency, either high-frequency. A spectrum contining all frequencies would denote a change of identity. The low-order strain energy is an ecient similarity metric, since we will measure a low energy during facial expression (high frequencies are neglected in the strain energy expression) and during change of lighting and pose (low frequencies are zeroed out with! 2 ). 5 Statistical modeling In theory, our deformable intensity surface can undergo any possible deformation. Thus, it seems interesting to learn the deformations of a specic class of objects and add them as constraints into our system. This is an important step for guiding the deformations of our mesh when performed within a specic object class. This learning process also increases the compression ability of the system. Our approach to learning the space of allowable manifold deformations particular to a specic object class (eg., frontal faces) is that of unsupervised learning. Particularly, we perform a Principal Components Analysis (PCA) [9] on a selected training set of deformations in order to recover the principal components of the warps. This approach is actually part of a more complete statistical formulation for estimating the probability density function of these warps in the high-dimensional vector space ~U 2 R P (see [12]). The estimated class-conditional density P ( ~Uj) can be ultimately used in a Bayesian framework for a variety of tasks such as regression, interpolation, inference and classication. However, in this paper, we have concentrated mainly on the dimensionality-reduction aspect of PCA in order to obtain a lower-dimensional subspace in which to solve for the manifold correspondance eld. Given a training set of suitable warp vectors f ~U t g for t = 1:::N T, the principal warps are obtained by solving the eigenvalue problem = E T E (8) where is the covariance matrix of the training set, E is the eigenvector matrix of and is the corresponding diagonal matrix of eigenvalues. The unitary matrix E denes a coordinate transform (rotation) which decorrelates the data and makes explicit the invariant subspaces of the matrix operator. In PCA, a partial Karhunen-Loeve Transform (KLT) [10] is performed to identify the largest-eigenvalue eigenvectors and obtain a principal component feature vector ^U = E T M ( ~U? ~U 0 ), where ~U 0 the mean warp vector and E M is a submatrix of E containing the principal eigenvectors. This KL transform can be seen as a linear transformation ^U = T ( ~U) : R P! R L which extracts a lower-dimensional subspace of the KL basis corresponding to the maximal eigenvalues. These principal components preserve the major linear correlations in the data and discard the minor ones. 1 By ranking the eigenvectors of the KL expansion with respect to their eigenvalues and selecting the rst L principal components we form an orthogonal decomposition of the vector space R P into two mutually exclusive and complementary subspaces: the principal subspace (or feature space) fe ig L i=1 containing the principal components and its orthogonal complement F = fe ig P i=l+1. In this paper, we simply discard the orthogonal subspace and work entirely within the principal subspace fe ig L i=1, hereafter referred to simply by the matrix E. Instead of solving the unconstrained governing equation (3), we compute the projection of the unknown U, rst into a modal sub-basis, then into a Karhunen-Loeve subspace : U ~U E ^U (9) where is the low-order modes of vibration matrix and E is the principal modes of variation matrix (see gure 5). 1 In practice the number of training images N T is far less than the dimensionality of the data, P, consequently the covariance matrix is singular. However, the rst L < N T eigenvectors can always be computed (estimated) from N T samples using, for example, a Singular Value Decomposition [7].

4 Figure 4: Facial expressions modify the intensity surface locally (high-frequency eect - top row) while a change of lighting has a global (low-frequency) eect (bottom row). E 1 E 2 E E 10-5 p i -1.5 p i mean 1.5 p i 3 p i Figure 5: Principal modes of variation of the manifold.

5 probe image best match second match third match Figure 6: Recognition examples. In each row, we search for the best match using the far left face as a probe into the database. Only the 3 best matches are displayed in each case. Similarity decreases left to right. The double projection into modal and KL subspaces yields a tremendous compression factor ; the size of the systems to be solved is drastically reduced : 50; E 10 (10) The compression factor is about 5000 : 1. 6 Retrieval in a face database The Media Lab database contains 7,562 \mugshot" face images of approximately 3,000 people. Each person has at least 2 images in the database. The faces are roughly aligned in the same position in all images, so that no normalization against translation, rotation or scale is necessary ; however, changes in lighting and facial expressions, as well as out-of-plane rotations frequently occur. We select a reference face among the 7,562 images in the Media Lab database (the most similar face to the mean face). We then warp all other faces of the database to the reference face, compute the strain energies and store them as a set of KL coecients. The i-th face is stored as the KL spectrum ^u i(l). The relative distance between the faces labeled i and j is : d(i; j) = 1 2 PX X P 0 p=1 p 0 =1! 2 (p; p 0 ) (~u i(p; p 0 )? ~u j(p; p 0 )) 2 (11) To assess the recognition rate, we use a nearest neighbor rule. If the most similar face was of the same person then a correct recognition was scored. In our experiment, the system produced a recognition accuracy of 97%, which corresponds to only six mistakes in matching views of 200 people randomly chosen in the database of 7,562 images. This is signicantly better than had previously been reported for this dataset [17]. Examples of recognition are shown in gure 6. 7 Conclusion We have described a novel approach for image matching based on deformable intensity surfaces. In this approach, the intensity surface of the image is modeled as a deformable surface embedded in XY I space. Our approach is thus a generalization of optical ow and deformable shape matching methods (which consider only changes in XY ), of statistical texture models such as \eigenfaces" (which consider only changes in I an assume an already existing XY correspondance), and of hybrid methods with treat shape and texture separately and sequentially. We have further shown how to tailor the space of allowable XY I deformations to t the actual variation found in individual target classes. This was accomplished by a statistical analysis of observed imageto-image deformations using a Principal Components Analysis. The result is that the image deformation is restricted to the subspace of physically-plausible deformations. In the process, the dimensionality of the matching and the numerical complexity of the governing equation are drastically reduced. By considering only the low-dimensional subspace of plausible deformations, we make the image matching process more robust and more ecient. We in

6 eect \build in" statistical a priori knowledge about how the object can vary in order to obtain the best image-to-image match possible. The method gives excellent results (97%) for image retrieval in a large face image database. References [1] K. J. Bathe. Finite Element Procedures in Engineering Analysis. Prentice-Hall, [2] Paul J. Besl and Neil D. McKay. A method for registration of 3D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):239{256, February [3] David Beymer. Vectorizing face images by interleaving shape and texture computations. A.I. Memo No. 1537, Articial Intelligence Laboratory, Massachusetts Institute of Technology, [4] R. Chellappa, S Sirohey, C.L. Wilson, and C.S. Barnes. Human and machine recognition of faces : A survey. Technical Report CAR-TR- 731, CS-TR-3339, University of Maryland, August [5] T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Active Shape Models - Their Training and Application. Computer Vision and Image Understanding, 61(1):38{59, January [6] J. Feldmar and N. Ayache. Locally ane registration of free-form surfaces. In IEEE Proceedings of Computer Vision and Pattern Recognition 1994 (CVPR'94), Seattle, USA, June [7] G. H. Golub and C. F. Van Loan. Matrix Computations. Johns Hopkins Press, [8] B.K.P. Horn and G. Schunck. Determining optical ow. Articial Intelligence, 17:185{203, [9] I.T. Jollie. Principal Component Analysis. Springer-Verlag, New York, [10] M.M. Loeve. Probability Theory. Van Nostrand, Princeton, [11] B. Moghaddam, C. Nastar, and A. Pentland. Bayesian face recognition using deformable intensity surfaces. In IEEE Proceedings of Computer Vision and Pattern Recognition (CVPR '96), San Francisco, June [12] B. Moghaddam and A. Pentland. Probabilistic visual learning for object detection. In IEEE Proceedings of the Fifth International Conference on Computer Vision, Cambridge, USA, june [13] C. Nastar. Vibration modes for nonrigid motion analysis in 3D images. In Proceedings of the Third European Conference on Computer Vision (ECCV '94), Stockholm, May [14] C. Nastar and N. Ayache. Fast segmentation, tracking, and analysis of deformable objects. In Proceedings of the Fourth International Conference on Computer Vision (ICCV '93), Berlin, May [15] C. Nastar, B. Moghaddam, and A. Pentland. Generalized image matching : Statistical learning of physically-based deformations. In Proceedings of the Fourth European Conference on Computer Vision (ECCV '96), Cambridge, England, April [16] C. Nastar and A. Pentland. Matching and recognition using deformable intensity surfaces. In IEEE International Symposium on Computer Vision, Coral Gables, USA, November [17] A. Pentland, B. Moghaddam, T. Starner, and M. Turk. View based and modular eigenspaces for face recognition. In IEEE Proceedings of Computer Vision and Pattern Recognition, [18] A. Pentland, R. Picard, and S. Sclaro. Photobook: Tools for content-based manipulation of image databases. Storage and Retrieval of Image and Video Databases II, 2185, [19] A. Pentland and S. Sclaro. Closed-form solutions for physically based shape modelling and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-13(7):715{ 729, July [20] R.W. Picard. A society of models for video and image libraries. Technical Report TR-360, MIT Media Lab, [21] M. Turk and A. Pentland. Face recognition using eigenfaces. In IEEE Proceedings of Computer Vision and Pattern Recognition, pages 586{591, [22] Z. Zhang. Iterative point matching for registration of free-form curves and surfaces. International Journal of Computer Vision, 13(2):119{ 152, also INRIA Tech. Report #1658.