Although implementations and applications vary, the idea of the EMD. and to some extent it mimics the human perception of texture similarities.

Transcription

1 Earth Mover's distance èemdè was rst introduced by Rubner et The for color and texture images ë11, 12ë. This distance can be calculated between al. Introduction two collections of points, when there is a measure of similarity measure between any It was shown to work well for image retrieval ë12ë, texture classication and points. ë9ë. segmentation The EMD allows for partial matches,so it is convenient for image Properties: and to some extent it mimics the human perception of texture similarities. retrieval, algorithms exist for computing it, especially in one dimension. However, Ecient is a purely heuristic distance. it Although implementations and applications vary, the idea of the EMD History: been in use for many years. The idea of matching up the closest values èa special has of the EMDè appeared in vision in 1983 ë13ë. In statistics, there is an equivalent case on probability distributions known as Mallows, or Wasserstein, distance, metric a clear probabilistic interpretation, introduced by Mallows in 1972 ë7ë. The use with this metric in physics and probability literatures dates back to the 1940s ë10ë. of we give the probabilistic interpretation of the EMD and discuss Here various statistical issues arising from this interpretation. 1

2 W èp; Q; Fè = m X j=1 f ijdij The Earth Mover's Distance EMD is dened as a distance between two ësignatures" of the form P = The m ;p m èg and Q = fèy1;q1è;:::;èy n ;q n èg. Signatures represent data fèx1;p1è;:::;èx centers of data clusters and the number of points in these clusters. They are not by normalized to have the same mass. necessarily EMD is dened by an optimal ow F =èf ij è, which minimizes the work The to move ëearth" from one signature to another: required d ij = dèx i ;y j è is a measure of dissimilaritybetween points x i and y j, for example, where distance in R d. Euclidean i=1 The ow must satisfy the following constraints: Move non-negative amounts of earth: f ij ç 0 for all i; j è1è 2

3 Take no more than you have, and move no more than ts in: j=1 f ij ç p i ; mx i=1 f ij ç q j for all i; j è2è Make sure all the earth is moved: mx m X = minè ij p i; i=1 f j=1 j=1 q jè è3è Once the optimal ow f ij Pm P n j=1 f ij d ij i=1 P mi=1 P nj=1 f ij i=1 is found, the EMD is dened as EMDèP,Qè = Normalization is done to make sure smaller signatures are not favored over larger ones. 3

4 E F kx, Y k p = m X j=1 f ijkx i, y j k p = m X j=1 f ijd ij : The Mallows Distance let X and Y berandom variables with probability distributions P and Q Now R d. The Mallows distance, M p èp; Qè, between P and Q is the minimum of the in dierence between X and Y,taken over all joint probability distributions expected èx; Y è with marginals P and Q: of p èp; Qè = min M fèe F kx, Y k p è 1=p :èx; Y è ç F; X ç P; Y ç Qg: F p can be any number greater or equal to 1, but the most interesting cases are Here =1and p =2. For the denition to make sense, the distributions P and Q must p nite p-th moments. have apply this denition to discrete distributions P = fèx1;p1è;:::;èx m ;p m èg To Q = fèy1;q1è;:::;èy n ;q n èg, we need to minimize the expectation under F =èf ij è, and joint distribution of X and Y : the The distribution F is subject to the following constraints: i=1 4 i=1

5 Marginals of F must be P and Q: j=1 f ij = p i ; mx i=1 f ij = q j for all i; j è5è Probabilities must add up to 1: mx m X = ij f j=1 n X = i q j =1 è6è j=1 p i=1 Probabilities must be non-negative: i=1 to EMD constraints: Compare è1è and è4è are the same f ij ç 0 for all i; j è4è As long as P and Q have the same total mass, the EMD constraints è2è are to become equalities and are the same as è5è. forced è3è and è6è are the same because both P and Q are proper probability distributions. that normalizing two signatures with the same total mass does not change their Note EMD. pointed out by Rubner et al., for two signatures with equal masses the EMD is a true As on distributions, and it is exactly the same as the Mallows distance metric with p set to 1. 5

6 The case of unequal total masses two sets of data X = f1; 4g and Y = f1; 2; 3; 4g. Example: on distributions: normalize both sets to have weight 1 èeach point in Mallows X has weight 1=2, each point in Y has weight 1=4è. The Mallows distance is then M1èX; Y è=1=2: on signatures: if we give every point weight 1 èso that the total mass EMD X is 2 and the total mass of Y is 4è, then X is completely contained in Y, no earth of needs to be moved, and EMDèX; Y è=0: 6

7 Key dierence: EMD allows for partial matching The EMD allows for matching any part of the distribution, no matter how Even if Y contained a thousand other points with very dierent values, small. distance between X and Y would still be 0. So partial matches may be the especially with textures. spurious, The EMD on signatures is not invariant to weight scaling, unless both signatures are scaled by the same factor. So if, e.g., one of the two texture patches is duplicated to produce a larger image, the distance will change. Partial matching may be appropriate in other non-texture contexts such image retrieval. It is a computationally ecient and convenient way to search a as large image for a small match, but it should be used cautiously. 7

8 Computing the Mallows distance from data In practice, it is important to distinguish between two dierent issues: Choosing the right distance for the problem, e.g., Mallows or ç 2 or L 2. Different 1. problems may require dierent distances. Estimating the distributions well from the available data, e.g., by a xed-bin 2. adaptive-bin histogram, signature, or some other method. The choice histogram, on the amount of available data, the dimensionality of the data, and of depends on the problem. course There are no a priori reasons for these two issues to be connected, other than computational convenience. For any discretized distribution estimate, the Mallows distance can be computed via optimization algorithms for the transportation problem ë5ë. 8

9 i=1 kx i, y j i kp i=1 jx èiè, yèièj p The optimization problem can be stated especially compactly if we have two samples the same size X = fx1;:::;x n g and Y = fy1;:::;y n g and use the empirical of distribution as our estimate, i.e., give every sample point weight 1=n: M p è ^F X ; ^F Y è= 0 B min èj1;:::;jnè n where the minimum is taken over all possible permutations of f1;:::;ng. problem can be solved by the Hungarian algorithm for the optimal This problem ë4ë, a special case of the transportation problem. assignment If the observations are one-dimensional, the optimization problem can be solved in this case the Mallows distance is just the L p vector distance between explicitly: the sorted vectors xè1è ç :::ç xènè and yè1è ç :::ç yènè: M p èx; Y è= 0 B n 9 1 1=p A C 1 1=p A C è7è

10 Applying the Mallows distance to textures now on,we concentrate on applying the Mallows distance èor the EMDè to From features, in this case to vectors of lter responses. One has to decide whether texture to use the joint or the marginal distributions, and how to estimate them from data. Marginals versus joint: Joint distribution of lter responses in theory contains a lot more information than the individual lter marginals. However, a high-dimensional distribution be much harder to estimate accurately. Also, the transportation problem can algorithm becomes slow for large problems, and the estimates must be coarsened to a feasible numberofpoints ènot more than a few hundredè. Marginal distributions, on the other hand, can be estimated very well even a relatively small image. And if one uses the empirical distribution, all one from to do is sort the vectors, which can be done very fast. There is no need needs bin or to use an optimization algorithm. to results below show that good estimates of the marginals outperform The joint. It also agrees with texture classication results in ë9ë, where the joint the distribution of lter responses did not do any better than the marginals. 10

11 Marginal distribution estimates: Fixed-bin histograms are easy to compute and store, but not necessarily accurate and sensitive to the choice of bin width. Adaptive-bin histograms, e.g., signatures, concentrate on the relevant part space only, and result in more compact estimates. They also correspond to the of of textons in the sense of ë6ë, where texton distributions are constructed concept clustering lter responses and computing the frequency of each cluster. However, by choice of clustering algorithm and its parameters make a dierence to the results. the Empirical distribution, where one keeps every point with mass 1=n, does not any additional algorithms or parameters, and is in general a good estimate. involve it with the Mallows distance has an extra advantage, because when all points Using the same weight, computing the distance requires only sorting. have distribution estimates: Joint xed-bin histogram does not work very well in high-dimensional The The empirical distribution cannot be used with Mallows distance, spaces. for large samples the optimization problem becomes computationally infeasible. since An adaptive-bin histogram seems to be the best choice in this case. 11

12 Experimental results experiments showing the usefulness of the EMD for texture analysis Extensive been published ë9, 11, 12ë. Here we test the empirical distribution as an have of the lter marginals as opposed to xed- and adaptive-bin histograms, and estimate marginals to the joint. compare The images were ltered with a lter bank of 40 lters, all rst and second derivatives at dierent scales and orientations, as in ë6ë. Gaussian When marginal distributions were used, the distance between two textures is computed as the sum of Mallows distances M2 between lter marginals. We used the relatively small MeasTex texture database ë8ë, which consists of Brodatz textures. The benchmarking strategy of ë9ë was followed: sets of 16 random non-overlapping square patches were extracted from each texture, with 16 of 16, 32, 64, and 128 pixels. sides For each image size, the classication error is estimated by the ëleave-one-out" i.e., leaving out each image in turn, computing its distance to all the method, images, and assigning it to the class of its nearest neighbor. The error other is the percentage of incorrectly assigned images. rate 12

13 table compares four methods of estimating the marginal: empirical distribution The èno binningè, coarse xed-bin histogram è16 binsè, ne xed-bin histogram è256 and adaptive-bin histogram where responses are clustered into 16 bins by a binsè, type algorithm. For the joint, only adaptive-bin histogram was used. k-means classication results: percent misclassied Texture dist. Image size Marginal estimate Empirical hist Adaptive hist Coarse hist Fine èadapt.hist.è Joint The empirical distribution function contains the most information and consistently does better than other estimates. The adaptive-bin histogram is nearly as good and requires less memory, takes longer to compute. but The xed-bin histograms perform substantially worse. The joint performs worse than the empirical distribution of the marginals. 13

14 References References and Acknowledgments P. J. Bickel and D. A. Freedman. Some asymptotic theory for the bootstrap. Annals of Statistics, 9:1196í1217, ë1ë P. Brodatz. Textures. Dover, New York, ë2ë A. Efros and T. Leung. Texture synthesis by non-parametric sampling. In Proceedings of the IEEE International Conference on ë3ë Vision, pages 1033í1038. Corfu, Greece, Sept Computer H. W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2:83í97, ë4ë G. Luenberger. Linear and Nonlinear Programming. Addison-Wesley, ë5ëd. J. Malik, S. Belongie, J. Shi, and T. Leung. Textons, contours, and regions: cue combination in image segmentation. In Proceedings ë6ë the IEEE International Conference on Computer Vision, pages 918í925. Corfu, Greece, Sept of C. L. Mallows. A note on asymptotic joint normality. Annals of Mathematical Statistics, 43è2è:508í515, ë7ë Meastex image texture database and test suite. Website: ë8ë meastex v1.1èmeastex.html. J. Puzicha, Y. Rubner, C. Tomasi, and J. M. Buhmann. Empirical evaluation of dissimilarity measures for color and texture. In ë9ë of the IEEE International Conference on Computer Vision, pages 1165í1173. Corfu, Greece, Sept Proceedings S. T. Rachev. The Monge-Kantorovich mass transference problem and its stochastic applications. Theory of Probability and its ë10ë 29:647í676, Applications, Y. Rubner, C. Tomasi, and L. Guibas. A metric for distributions with applications to image databases. In Proceedings of the IEEE ë11ë Conference on Computer Vision, pages 59í66. Bombay, India, Jan International Y. Rubner, C. Tomasi, and L. J. Guibas. The Earth Mover's distance as a metric for image retrieval. Technical Report STAN-CS- ë12ë Department of Computer Science, Stanford University, Sept TN-98-86, H. C. Shen and A. K. C. Wong. Generalized texture representation and metric. Computer Vision, Graphics, and Image Processing, ë13ë :187í206, M. Werman, S. Peleg, and A. Rozenfeld. A distance metric for multidimensional histograms. Computer Vision, Graphics, and Image ë14ë 32:328í336, Processing, are grateful to Jitendra Malik, Alex Berg, Alexei Efros, Jan Puzicha, and Jianbo We for helpful discussions and comments. We also thank Serge Belongie for the ltering Shi code and Yossi Rubner for the EMD code. 14

15 i=1 M2 2 èf i;g i è: Z è 1,1 ètè, G,1 ètèj p 1=p dt! : jf 0 Appendix: Some properties of the Mallows distance 1. M p is a metric. Convolution for =2: if R xdf i èxè = R xdg i èxè for i =1:::n, then 2. property n p èf X 1 ::: F n ;G1 ::: G n è ç 2 2 M This is stronger than the triangle inequality. 3. M p èf n ;Fè! 0 if and only if F n! F weakly èiè R kxk p df n èxè! R kxk p df èxè. èiiè If X1;:::;X n are independent observations from a distribution F,andF n is their 4. distribution, i.e., F n ètè =1=n P n i=1 1èX i ç tè, then M p èf n ;Fè! 0. empirical 5. If F and G are distributions on the real line, then M p èf; Gè = case = is especially simple because The p 1 1,1 Z, G,1 Z 1 ètèjdt = ètè jf 0 15,1 jf ètè, Gètèjdt: