Learning the Epitome of an Image

Transcription

1 University of Toronto TR PSI , Nov 10, Learning the Epitome of an Image Brendan J. Frey and Nebojsa Jojic November 10, 2002 Abstract Estimating and visualizing high-order statistics of multivariate data is important for analysis, synthesis and visualization in science and engineering. Often, data consists of measurements on an underlying domain, such as space or time. Examples include images, audio signals and text, where the domains are 2-D space, 1-D time and 1-D symbol index. We introduce a model called the epitome that can simultaneously represent multi-scale high-order statistics as a set of parameters on the same domain as the input data. A cost function measures how well multi-scale patches drawn from the input data match the epitome and this cost function can be optimized efficiently using the EM algorithm. Our technique reduces a large number of high-order statistics to an intuitive, compact representation that is suitable for a variety of data processing applications. We demonstrate our method using problems of object detection, texture segmentation and image retrieval. One approach to the problems of fully or semi-automated visualization, analysis and synthesis of data is to learn a compact model that accurately accounts for interesting properties of the data and can be used as a summary of the data [1, 2]. The hope is that the compact University of Toronto, 10 King s College Rd., Toronto ON., M5S 3G4, Canada Microsoft Research, One Microsoft Way, Redmond WA., , USA 1

2 model will offer a way to visualize complex relationships, discover high-order patterns that are useful for further analysis by human or machine, and provide an efficient parameterization of the data that is useful for synthesizing variations of the data and making predictions. We introduce a parametric generative model called the epitome and an unsupervised learning algorithm for fitting the epitome to input data. When data is measured on some underlying domain such as space or time, a sensible model should account for properties that emerge from invariances in the domain. The epitome accounts for one one of the fundamental elements in Grenander s pattern theory, domain warping [1], and the computations needed to learn the epitome are related to Mumford s proposition for implementation of domain warping in neuronal architectures [3]. An epitome is an image of parameters that specifies a generative model of patches taken from an input image. Although the domain on which the data is measured can have arbitrary dimensionality, for concreteness we focus on 2-dimensional images, such as those shown in Fig. 1 [4]. Given an input image, a training set can be formed by randomly selecting patches of various sizes from the input image. In fact, the patches may be of the same size or of a variety of sizes and they may be taken regularly from the input, or from random locations. The set of patches is then used to obtain a maximum likelihood estimate of the epitome, as shown on the far right in Fig. 1. Although the epitome is over five times smaller in area than the input image, it contains many of the multi-scale high-order statistics present in the input. We now introduce a novel cost function for learning the epitome. The goal is to maximize the marginal probability of the input patches, summing over all possible ways in which the input patches can be generated from the epitome. An input patch X is a function x(k) on 2

3 A B Figure 1: (A) To estimate the epitome of a input image, a training set is formed by extracting patches of random sizes from random positions in the input image. Then, the expectation maximization (EM) algorithm is used to learn the more compact epitome, so that the set of patches is also likely to have been drawn from the epitome. (B) The mean color image of the epitome during learning [9], where each successive picture corresponds to an additional 3 iterations of EM. The final epitome contains many of the large-scale structures in the input image, but also retains many details, such as the sharp boundaries between the flowers and the dark background. an underlying domain K, where k K is a vector whose dimensionality is equal to the dimensionality of the image domain. We assume the domain is discrete, so that K is a subset of the integer tuples. In Fig. 1, the underlying domain is 2-D space, so k is a 2-D integer vector. 3

4 The epitome E is represented by an image of model parameters e(j), a set of mappings M that specifies how patches may be generated from the epitome, and one probability for each mapping. The domain of the model parameters, J, is usually smaller than the domain of the input image. A patch is generated by first selecting a mapping m with probability ρ(m), where m M ρ(m) = 1. Then, each value in the patch, x(k), is generated independently using the corresponding parameter in the epitome, e(m[k]). The density of the patch given the mapping and the epitome is p(x m, E) = k K f(x(k); e(m[k]))), (1) where f( ; ) is the density function for each value in the patch. Out of all parameters, the density of x(k) depends only on the parameter e(m[k]). The most appropriate form of the density function f is application-specific [5]. Multiplying by the prior probability of the mapping and summing over all mappings, we obtain the marginal likelihood of the epitome, p(x E) = ρ(m)p(x m, E). (2) m M The marginal log-likelihood for a set of N i.i.d. training patches X 1,..., X N is L(E) = N n=1 ( log m M )) ρ(m)p(x m, E). (3) This is the log-probability that the training patches were generated from the epitome. Learning entails searching for the epitome E that maximizes the highly nonlinear function L(E). While a variety of nonlinear optimization techniques can be applied, we present an efficient expectation maximization (EM) algorithm [6] that treats the mapping as a hidden variable. Initially, the epitome E is set to a random value and then the algorithm alternates 4

5 between an E step and and M step until convergence of the epitome. In the E step, for each input patch, the posterior distribution over the mapping is computed using Bayes rule: P (m X n, E) = p(x n m, E)ρ(m) m M p(x n m, E)ρ(m). (4) In the M step, ρ(m) is set to 1 N N n=1 P (m X n, E) and e is set to the value of e that maximizes the expected value of log P (X m, E), N P (m X n, E) log f(x n (k); e (m[k])). (5) n=1 k K m M In the M step, the parameter e (j) at location j can be solved for independently of the other parameters, by maximizing N ( n=1 k K m:m(k)=j ) P (m X n, E) log f(x n (k); e (j)). (6) e(j) is adjusted to maximize the log-likelihood of all of the values in every patch, where each value is weighted by the posterior probability that the value was mapped from position j in the epitome. The EM algorithm produces a sequence of parameter estimates with monotonically increasing likelihood [7], We report results on real-valued visual images using the normal distribution for f and the set of mappings corresponding to cutting out axis-aligned rectangular patches from all possible locations in the epitome. See [8] for details on the EM updates and how they can be computed efficiently using convolutions. Here, the parameter e(j) associated with position j in the epitome consists of a mean and a variance. Fig. 1 shows the means, as an image, learned from a color picture of a dog standing on a side-walk in front of a garden [9]. The 5

6 Figure 2: The epitome simultaneously supports a variety of complex data processing tasks, such as detecting individual patterns and segmenting textures with quite different spectral properties. Regions including the dog s nose, the shaded flowers and the sidewalk surface are tagged in the epitome and then the input is reconstructed using the tagged epitome [10]. epitome contains many of the dominant features from the input image, including various features from the dog, light and dark magenta flowers, and the sidewalk. Importantly, these features are integrated together in a consistent way across the epitome, even though they have different scales. The epitome of an image offers a way to easily visualize the high-order statistics in the image, and simultaneously supports a variety of complex data-processing operations. Fig. 2 6

7 shows how the epitome from above can be used to separately tag two textures with quite different scales, and locate an object (the dog s nose) that occurs at only one location in the input image [10]. Complex patterns are often described by a combination of parts, such as specific types of mouth, eyes and hair in images of faces. Fig. 3 shows that learning the epitome of patches drawn from frontal images of faces produces a semi-coherent montage of face parts, which can be used for parts-based image retrieval. In other work, libraries of fixed-size patches have been used successfully for modeling high-order statistics in images [13]. By virtue of the patch index, libraries of patches can model dependencies that span the width of a patch. To account for multiple scales, a huge library of multi-scale patches can be constructed. In contrast, the epitome can model the same range of scales, but uses many fewer parameters. Further, the epitome integrates the multiple sizes of patches together into a single, spatially coherent image. In contrast to most texture models, the epitome can simultaneously model structures at a variety at different scales. Low-order image statistics, low-order Markov random fields, and statistical spectral techniques have been used quite successfully for modeling narrowband textures [14 16]. However, these techniques are not well-suited to modeling inhomogeneous images that have multiple patterns at different scales. Since the epitome is represented as an image, it can model features that occupy a wide range of frequency bands and can model phase dependencies that span the width of the epitome. A quite different approach to modeling long-range dependencies is to introduce hidden variables that store the state of the pattern, in the fashion of hidden Markov models [17] and Markov random fields [2]. While hidden Markov models can potentially model long-range 7

8 A B C D E Figure 3: Parts-based image retrieval using the epitome. (A) The epitome was learned using patches drawn from a library of face images [11]. (B) To find smiling faces, we identified a region in the epitome that contains the corner of a smiling mouth. Then, we retrieved the 25 faces that had the highest total posterior probability of using patches in the identified region. (C) To find faces without smiles, we retrieved the 25 faces that had the lowest total posterior probability of using patches in the identified region. (D) To retrieve images containing a combination of parts, we identified multiple regions in the epitome and retrieved images containing at least one patch that was most likely from each region. Here, we chose regions in the epitome containing a nearly closed eye, the corner of a smiling mouth, and dark hair. (C) Images retrieved when we identified the same regions as above, except that an open eye was identified instead of a closed eye. 8

9 dependencies, the forward-backward learning algorithm is notorious for confusing some of the states in long sequences, thus failing to accurately model long-range dependencies [18]. This problem is worse for Markov random fields, where approximate inference algorithms must be used to infer the state variables during learning, and confusing states for different patterns is even more likely. The epitome avoids this problem by directly fitting segments from the input, so that long-range dependencies within individual patterns are preserved during learning. In comparison with vector-space techniques for clustering [19, 20], dimensionality reduction [21 24] and independent component analysis [25], an advantage of the epitome is that it can account for relationships between patterns that emerge when the data is measured on an underlying domain, such as space or time. By representing the model in this same domain, the epitome can associate input patterns that come from nearby locations in a broader pattern, even when the input patterns are far apart in the corresponding vector space. We view the epitome and other data analysis techniques as complementary. Dependencies between elements in the epitome that are not accounted for by domain warping can be modeled using vector-space machine learning techniques, such as independent component analysis [25]. Since the epitome is a probability model, it can be integrated into other probability models used in machine learning. Our description of the epitome and the EM learning algorithm leaves several interesting avenues of research unexplored. For example, although we reported results on highlyconstrained forms of mappings, another approach is to allow much richer mappings. An arbitrary mapping can be specified using one mapping variable m[k] for each position k in the patch: m = {m[k] : k K}. Without any constraints, the total number of 9

10 such mappings is too large to directly enumerate for all but uninterestingly small models. However, if the distribution over mappings, ρ(m), is described by a tree on the mapping variables {m[k] : k K}, dynamic programming can be used to efficiently compute the summations needed for inference and learning [26]. Our results show that the epitome provides a natural interface to the much larger input image and supports a variety of complex data processing tasks, including texture segmentation, object detection, and parts-based image retrieval. It is also appealing that the epitome can be estimated by optimizing a cost function and that the EM algorithm can be used to efficiently find a solution. By virtue of the fact that the epitome models multi-scale highorder statistics, but is defined on the same domain as the input, we believe the epitome has the potential to be useful in a variety of application areas. References [1] U. Grenander, Lectures in Pattern Theory I, II and III: Pattern Analysis, Pattern Synthesis and Regular Structures (Springer-Verlag, Berlin, ). [2] G. E. Hinton and T. J. Sejnowski, In D. E. Rumelhart and J. L. McClelland (eds), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol I (MIT Press, Cambridge MA, 1986). [3] D. Mumford, in C. Koch, J. Davis (eds) Large-Scale Theories of the Cortex, 125 (MIT Pres, Cambridge, MA., 1994). [4] To account for multi-channel data such as color images, it is useful to allow the input image and the epitome to be fields, i.e., vector images. For example, in color 10

11 visual images such as those shown in Fig. 1, the color at each 2-D spacial index can be represented as a 3-vector representing red, green and blue channels. In spectral representations of time-series data, the spectrum at each 1-D time index can be represented as a vector containing the energies at various frequencies. For notational simplicity, we describe the model and learning algorithm in the case of scalar images. The extension to vector images is straightforward. [5] We have investigated learning epitomes of color images, music, and text. For color images, we took k to be 2-dimensional and x(k) to be a 3-vector of light intensities in the red, green and blue channels. We used the normal density for f, where e(j) consisted of 3 means and a 3 variances. For spectral representations of timeseries data, we took k to be 1-dimensional and x(k) to be an n-vector containing the short-time power spectrum. We used the n-dimensional normal density for f, with diagonal covariance matrix. For text, we took k to be 1-dimensional and x(k) to be a discrete symbol. We used the multinomial distribution for f, where e(j) consisted of one probability for each symbol. [6] A. P. Dempster, N. M. Laird, and D. B. Rubin, Proceedings of the Royal Statistical Society B 39, 1 (1977). [7] R. M. Neal and G. E. Hinton, in M. I. Jordan (ed) Learning in Graphical Models (Kluwer Academic Publishers, Norwell MA, 1998). [8] In this case, the parameter associated with position j in the epitome, e(j), consists of a mean µ(j) and a variance σ 2 (j). The mapping can be represented as an integer vector m that specifies the location in the epitome of the patch, so the parameter used for position k in the patch is e(m + k). It is convenient to define the epitome on a torus, so that m + k is taken modulus the largest integers in J. 11

12 Inserting the normal distribution into Eq. 4, it can be shown that up to an additive constant, the log-posterior distribution over the mapping, log P (m X, E), is log ρ(m) 1 2 k K( log(σ 2 (m + k)) + (x(k) µ(m + k)) 2 /σ 2 (m + k) ). Expanding the square, we obtain the following: log ρ(m) 1 2 k K( log(σ 2 (m + k)) + x(k) 2 /σ 2 (m + k) 2x(k)µ(m + k)/σ 2 (m + k) + µ(m + k) 2 /σ 2 (m + k) ). Each of these terms can be computed efficiently for all m using a convolution. The update for the mean at position i in the epitome is µ(i) ( N n=1 m:i m K P (m X n, E)x n (i m) ) / ( N n=1 m:i m K P (m X n, E) ) and the update for the variance is σ 2 (i) ( N n=1 µ(i)) 2) / ( N n=1 m:i m K P (m X n, E)(x n (i m) m:i m K P (m X n, E) ). These updates can also be computed using convolutions. The above parameter updates can be viewed as the updates for learning a large mixture of J Gaussians [6], where there is one mixture component for each possible patch in the epitome, and the mixture model parameters corresponding to overlapping patches are constrained to be equal. [9] To learn the epitome in Fig. 1, a total of 4000 square training patches with widths 80, 64, 48, 32, 24, 20, 16, 12, 10 and 8 were sampled randomly from the input image. Large patches constrain the learning algorithm more than small patches, so the number of patches of each size was selected to keep the total area of the patches in every size category the same. For each size category, 3 iterations of EM were applied, starting with the patches and finishing with the 8 8 patches. The means for each color channel in the epitome were initialized to the mean of all values in the same channel in the training set, plus Gaussian noise with 1/100th of the standard deviation of the training data. The variances were initialized to the variance of the training data. 12

13 [10] The means of the epitome found in the experiment described in [9] were tagged by hand, to highlight the shaded flowers (green), the sidewalk surface (cyan), and the dog s nose (yellow). Then, 8 8 patches were taken at regular intervals from the input image and the most probable position of each patch in the original epitome was computed using Eq. 4. If the patch in the epitome overlapped with a region in the epitome that had been tagged, the tagged version of the patch was copied to the output image. Otherwise, the original patch from the input image was copied to the output image. To produce a smooth output image, patches were taken at 4-pixel intervals from the input image and overlapping patches in the output image were averaged together. [11] To learn the epitome shown in Fig. 3, a set of 300 frontal face images from the publicly available database described in [12] were cropped and sub-sampled to form a set of 300, images. A total of 40,000 patches of size 24 24, and 7 7 were then drawn randomly from these images, where the number of patches of each size was selected to keep the total area of the patches in every size category the same. For each size category, 10 iterations of EM were applied, starting with the patches and finishing with the 7 7 patches. The means in the epitome were initialized to the mean of all values in the training set, plus Gaussian noise with 1/100th of the standard deviation of the training data. The variances were initialized to the variance of the training data. [12] A. Lanitis, C. J. Taylor, T. F. Cootes, Image and Vis. Comput. 13, 393 (1995). [13] W. Freeman, E. Pasztor, Proc. Internat. Conf. Comp. Vision, 1182 (1999). [14] B. Julesz, Nature 290, 91 (1981). 13

14 [15] S. C. Zhu, Y. N. Wu, D. Mumford, Neural Comput. 9, 1627 (1997). [16] J. Portilla, E. P. Simoncelli, Intern. Journ. Comp. Vision 40, 49 (2000). [17] L. Rabiner, B.-H. Juang, Fundamentals of Speech Recognition (Prentice-Hall, Englewood Cliffs, NJ., 1993). [18] M. Ostendorf, V. Digalakis, D. Kimball, IEEE Trans. Speech & Audio Proc. 4, 360 (1996). [19] S. P. Loyd, IEEE Trans. Info. Theory 28, 129 (1982). [20] J. Shi, J. Malik, IEEE Trans Patt. Anal. Mach. Intell. 222, 888 (2000). [21] T. Kohonen, Self-Organization and Associative Memory (Springer-Verlag, Berlin, 1988). [22] I. T. Jolliffe, Principal Component Analysis (Springer-Verlag, New York, 1989). [23] C. Bishop, M. Svensen, C. Williams, Neural Comput. 10, 215 (1998). [24] S. T. Roweis, L. K. Saul, Science 290, 2323 (2000). [25] A. J. Bell, T. J. Sejnowski, Neural Comput. 7, 1129 (1995). [26] In this case, the distribution ρ(m) over mappings is described by a tree on the mapping variables m[k], k K. The epitome defines a density f(x(k); e(m[k])) for each mapping variable, so p(x m, E)ρ(m) is described by a tree on m[k], k K. Dynamic programming can be used to efficiently compute the marginal likelihood of the epitome in Eq. 2 and the expected value of log p(x m, E) in Eq. 5, which can be used to learn the epitome. Once the epitome is learned, dynamic programming can be used for various data-processing tasks. 14

15 [27] We thank P. Anandan, A. Blake, G. E. Hinton, A. Kannan, S. T Roweis, C. K. I. Williams and S. C. Zhu for helpful discussions. Frey acknowledges support from the Natural Sciences and Engineering Research Council of Canada. 15