Photo Mosaicing. Sam Baird Ben Felsted Zach Gildersleeve David Koop. April 30, 2007

Transcription

1 Photo Mosaicing Sam Baird Ben Felsted Zach Gildersleeve David Koop April 30, Introduction While diverse work has been invested in mosaicing images together, the common goal is usually to combine smaller images to generate a single larger image. In applications such as aerial photography or medical imaging, the smaller images typically originate from the same dataset, and thus each subimage is a small part of a unified whole. Mosaicing can also be thought, more artistically, as a patchwork quilt. Instead of combining similar images to form a larger image, we use a collection of diverse images to compose a single image so that the new image looks from afar to be a single image, but upon closer inspection reveals the individual images. This set of images is typically referred to as a photomosaic. Many different approaches have been taken, but few have closely examined the strengths and weaknesses of each algorithm. We have researched and implemented many different techniques in order to compare. We start out with a simple localized mean squared error technique. We build upon that by introducing antipole clustering to speed up the search. We also analyze boundary matching and discrete cosine transform in choosing a best match. Finally, our work explores global optimization as a technique for creating the mosaics, expanding the local optimization algorithm to include an iterative simulated annealing that converges on an optimal solution given the subimage dataset. 2 Related Work The idea behind photo mosaicing dates back to 1973, when Harmon use block portraits to study human perception and pattern recogntion. He presented a very low resolution image of Abraham Lincoln, where each pixel was a large tile. When the subject stood far away, they could recognize the face. Then in 1976, Dali created a painting similar to Harmon s, called Lincoln in Dalivision, but he included a woman in the center. When looking close-up, you could only recognize the woman. When viewing from far away, all you saw was Lincoln. The first software implementation of photomosaicing was started by Silver while he was at MIT. While earlier attempts used Photoshop to replace a pixel with an image, Silvers replaced a tile (grid of pixels) with an image. He matched the image according to color and shape. In 1998, Hunt took this idea a step farther and implemented photo mosaicing using different size tiles. Klein also furthered Silver s idea by creating a video mosaic. Instead of an image being created from a set of smaller images, a video was created from a set of smaller videos. This adds a level of complexity because time has to now be accounted for in matching. Because 1

2 consistency in the small videos is desired, it is hard to find good matches. To alleviate this problem, they correct the color to match the current frame. Another algorithm was proposed in 1998 that was based off Silver s idea. After choosing the best match for the subimages, each image is altered according to color to best match the target tile. It calculates the desired average color and creates a correction function such that the final average color in the subimage is the same as the average color in the target tile. Most recently, researchers have tackled the problem of panoramic mosaic, where the final image is a set of photographs of the same scene, stitched together from different perspectives. However, the goal is not to notice the stiches and thus make a coherent final image. Typically, you preprocess the image database and extract particular information about each image. For example, you can resize the image for fast comparisons, you can compute their histograms, or you can build a hierarchy to speed up the search process. De Blasi was able to speed up the search process using the Antipole strategy. Using an Antipole Tree Data Structure, you embed a large set of images into a metric space (X, d). Each image is a point in X and distance d is defined as the mean squared error in RGB values. Images can then be grouped into clusters so that far elements lie in different clusters. You define endpoints and then partition the points according to the proximity to the two endpoints. This is repeated again recursively on the two clusters. This gives you a binary tree in which to search. 3 Algorithm Framework While the concept of a photomosaic is fairly straightforward, the exact specification of it is less refined. We are trying to accurately replicate an image I using a set of given images S. In order to make this more precise, we need to define a metric that captures this accuracy. Given the image I, we define a function f : I S such that f(x, y) = s(g(x, y)) S. Thus, f defines 1. the image s S from which the pixel is to be taken 2. the location in s, g(x, y), from which the pixel is to be taken Given this function, we can define a metric based on the difference between I(x, y) and f(x, y). Of course, given this definition, we can simply construct an f that takes the pixel closest to I(x, y) from the set S. This is counterintuitive to the mosaic idea where f(x, y) is taken from the same image as its neighbors unless it lies along a boundary. Thus, we also need to impose a locality constraint on f to enforce this idea. Let B denote a set of intrapixels {(x+1/2, +1/2)}. f(x, y) and f(x ± 1, y ± 1) must map to the same image s unless (x ± 1/2, y ± 1/2) is in B. Often B is a gridded structure, but it may be different shapes as well. Most photomosaics are constructed using a local optimization, meaning that for each contiguous section of our decomposition of I, we find the best s S. However, this idea can be extended to instead consider a global optimization. For this problem, we constrain our selections from S to disallow repetition. Thus, picking the best local match may cause a poor global match. We can still score each candidate image against the region to be painted, but instead of simply taking the candidate with the highest score, we select a candidate with this global view. 2

3 Specifically, let I be decomposed into regions R(I) = {R 1,..., R m }. We wish to determine a mapping m : R(I) S such that m match(r i, m(r i )) i=1 is maximized. As stated, this is a local optimization problem. However, suppose we restrict this mapping so that m(r i ) = m(r j ) = i = j. Then, we have a global optimization problem. 3.1 Greedy Algorithm We first implemented a simple greedy algorithm. We choose a region R(I) at random and select an image from S that best matches the region. The error between R(I) and the image in S is defined by a mean squared error on the RGB values of the pixels. When duplication of images in S is allowed, an optimal arrangement of images is easily found. However, duplication is usually not desired. To remedy this, we take two different approaches. In one method, we penalize images that have already been used, thus enforcing a wider variety of images. In our other method, we don t allow duplication. Finding a global optimal answer becomes very difficult when this uniqueness constraint is enforced. We explore simulated annealing later in this paper in order to itteratively approach the optimal answer. 3.2 Antipole Clustering To choose the best match, we have to search through each image in set S. This needs to be done for every region R(I). This search can be cumbersome considering how many images there are in the database (we have 1500) and how many regions we compute (typically around 5,000-10,000). To speed up this process, we use an antipole clustering approach. By defining a metric space, we can represent an image in S as a point in space. We can define the distance between two points as the mean squared error between the two images. Each image in S is a point, and we can cluster the points based on distance. First, we find the farthest two points from each other. Then we go through each point and cluster it to the closest endpoint. This creates two clusters. We recurse on each cluster until we have only one point for each cluster. We now have a binary tree of clusters. When we process a particular region R(I) of our final image I, we search for the closest match in S. However, we can use our tree structure to find the correct image in logarthimic time instead of linear time. We find which cluster the region is closer to, and recurse down that branch, until we ve found the best match. Because this created a lot of overhead, the benefit of using an antipole structure was not evident with a small image I. However, for large images, the clustering approach was four times faster than the linear approach when a I was a 250x250 image. 4 Boundary Matching In the area of texture synthesis, the major criteria for a good synthesis is that the tesselation boundaries are invisible. For photomosaics, we will get a much better approximation to the original image if we succeed in matching these boundaries. In some cases, this is desirable to 3

4 Figure 1: An Example of Antipole Clustering obtain a better approximation, but often the distinct boundaries are desired to maintain the photomosaic structure. Thus, for adjoining candidate images, we can calculate their boundary matching score by examining the pixels near the corresponding edge. Intuitively, we wish this boundary to be as smooth as possible. This can be obtained by examining the gradient across the edge. With boundaries, our optimization problem is now m (1 α) match(r i, m(r i )) + α i=1 R j B(R i ) boundary(m(r i ), m(r j ), e(r i, R j )) where α represents the weighting of boundary matching. Our work is not increased too much. For each image, we need to compute its match with each region, and for each pair of images, we need to compute its boundary match along each possible edge. We could also match the boundaries by doing a mean squared error on the boundary edges. We add the boundary error to the error between the region and the image from S. However, by adding another error metric, we sacrifice the selection quality. We have a smoother image, but our image quality decreases because it selects suboptimal matches. We have implemented this algorithm in our work. We notice that as we weight the boundary criteria more, the recognition of the image decreases. 5 Discrete Cosine Transformations While the described averaging method is a fairly straightforward means of creating a photomosaic, it is by no means the only method. In theory, any algorithm that provides a means for ranking the similarity between two images can be used to drive the photomosaic. We present an exploration using the Discrete Cosine Transformation (DCT) to rank the subimages and provide a means for comparison to each cell of the master image. The DCT is known best for its use in lossy JPEG compression due to its strong energy compaction property. Compared to the similar Discrete Fourier Transformation, the DCT concentrates most of its energy in lower frequencies. These lower frequencies represent the general outline of objects in an image; JPEG compression is achieved by clamping the DCT transformed image, thus limiting the detail in solid, filled areas. 4

5 There are multiple definitions of the DCT, we will use the the most common DCT-II: D k = N 1 n=0 [ ( π x n cos n + 1 ) ] k N 2 k = 0,..., N 1 To transform a 2D image using the DCT, significant acceleration can be found by using square images. If I is a square image with side n, the DCT of I can be found by I DCT = D I D where the nxn DCT transformation matrix D need only be computed once. 5.1 DCT Photomosaic Algorithm The basic algorithm to use the DCT as matching function within a photomosaic application is as follows. First, we calculate the DCT of each subimage as above. This is done using a precomputed DCT transformation matrix of mxm, where m typically is set to 128 pixels, which we can use throughout the implementation. Due to the high energy compactness of the DCT, it is necessary to only compare the highest n DCT coefficients of each subimage. The value of n is introduced as a parameter, and provides results that quickly converge on the best match for a set of subimages numbering in the hundreds. Beyond the highest n coefficients, no change of R I is possible, due to the effective quantization of a finite number of subimages, but additional coefficients could be introduced as a means to alter R I. The master image I is reduced to its individual regions R(I) = {R 1,..., R m }, as in the initial implementation, but these regions are of arbitrary size (typically smaller than mxm) which does not correspond to the mxm DCT transformation matrix. It is worth noting that D nxn I nxn D nxn D mxm I mxm D mxm therefore we interpolate the cell B to the size of the DCT transformation matrix using a interpolation matrix H m. B m = H m B s B m = D m B m D m B n = B m (1 : n) The result, in DCT space, can be compared against the set of subimages using mean squared error to find the best fit, using either a local or global optimization as discussed. The comparison can be done against the full m 2 values in the DCT images, or clamped to the highest n values as in JPEG compression. If the described photomosaic matching algorithm is conducted in DCT space, matching subimages are located at the low frequency edges of the master image, at the expense of detail in the filled areas. Thus, this algorithm focuses on the edges of an image. However, since we are now matching the lower frequencies of the subimage to the region R i, the result better provides subregion matching, reducing pixelation. 5

6 5.2 DCT Results and Discussion Using the DCT to drive the photomosaic algorithm produces decidedly different results than the initial localized mean square error method. In that method, each subregion within a block is matched according to the average color value. With the DCT method, we are comparing only against frequency. Thus, we get slightly different results. Figure 2: Goal Image (a) (b) 3x3 Averaging DCT Matching Figure 3: Comparison of Photomosaic Results As we can see in Figure 3, in particular the hair region of the goal image Figure 2, the DCT matching method matches the blocks at a subblock level better than the average matching method. Figure 3 is only matching frequency, and not average color, and therefore some of the large solid areas, such as the hat and shirt, are matched to a subimage that shares a similar frequency signature, but is divergent in terms of overall color average. This is to be expected, as this information is essentially lost when we only compare the top n values of our DCT signature. An important observation is that for discrete images, high frequencies at the global scale translate to low frequencies at the local scale; that is, if we interpolate a small local region 6

7 to a much larger size, any high frequencies are expanded out and essentially become low frequencies. As the energy compactness property of the DCT maintains a large proportion of these low frequencies, much of the original image is preserved and, in the application of photomosaic, used to match the sub images. 5.3 Extension of the DCT to Image Searching The Discrete Cosine Tranform, like its Fourier cousin, finds many uses. In addition to lossy compression, visualized using a photomosaic algorithm, the DCT can be used for other shape (low frequency) matching and searching utilities. Bracamonte et al, among others, have noticed that the broad DCT algorithm outlined above can be used for quick image based searching. Such searching can be done in the DCT domain, which for web outlets where JPEG images are the norm, this can account for considerable acceleration. While the details of such an algorithm fall outside the scope of this discussion, a brief examination of image based searching is possible. If we use a block size equal to the goal image we are essentially comparing against our database of subimages the image that best matches the overall low frequency signature of the goal image. This can provide some interesting results, although a complete robust search algorithm would most likely be interested in additional parameters such as color. In Figure 4 we see the fullsized subimage that best matches the highest 256 DCT coefficients of the same goal image, out of 1500 images. This relatively simple concept can be very powerful in its utility. (a) (b) Goal Image DCT Matching Figure 4: DCT Based Image Searching 6 Simulated Annealing The problem of finding a global minimum in error involves a search over a large state space. Each state is a permutation of the ordering of tiles that are to be matched with the goal image. 7

8 In the experiments we performed, the number of permutations was the following: ( ) Number of tiles 1501! Number of blocks! = Number of blocks ( )! Simulated Annealing is a generic probabilistic meta-algorithm for finding a global minimum in a large state space. It iteratively converges to a solution of an NP-problem using a polynomial time algorithm at each iteration. Name of Simulated Annealing (SA) comes from annealing in metallurgy, in which material is heated and cooled in a controlled manner. The goal of this is to increase the crystal size of the material and decrease defects. The addition of heat allows molecules to take a highenergy random walk to become unstuck from local minima of energy configurations. As the temperature cools, the molecules take smaller steps and the total energy eventually converges to a new minimum. SA is closely related to the physical process, except it is applied to the global optimization problem. Each step of SA replaces the current solution with a neighboring solution. The probability of choosing a particular solution depends on the energy (or error) of the current and next solution and on the current temperature, T. The probability distribution is chosen such that when T is large, the probability is almost uniform, and when T is small, the solution converges in the Cauchy sense. Additionally, when T becomes small, the probability is close to one if transitioning to the next solution causes a decrease in energy, and close to zero otherwise. In the context of Photo Mosiacing, we make the following defining assumption: 1. Each state is a permutation of the tiles. We denote the current and next state as e and e. 2. Neighboring states differ by a swap of tiles 3. The energy of state is proportional to the sum of squared errors The probability of a state transition due to Kirpatrick et. al: Pr(e, e, T ) = α exp e e T The formula comes from Metropolis-Hastings algorithm used to generate samples from the Maxwell-Boltzmann distribution. Figure 5 shows the convergence of total energy for a given number of iterations. The final image converged to a solution that had small error on average, but a large error for at least one of the tiles. Using the greedy method resulted in errors that were relatively small on average. 7 Final Discussion At first glance, the photomosaic concept is a purely artistic endeavor. However, powerful concepts fundamental to image processing, signal compression, and global optimization can be visualized via a photomosaic. In the 3x3 algorithm, a simple photomosaic framework was 8

9 3.15 x 10 4 Simulated Annealing: Total energy for a given number of steps 3.1 Total energy Number of iterations Figure 5: Trend of Simulated Annealing Optimization established to provide a testbed for technique comparison. Antipole clustering introduced a binary tree acceleration structure to photomosaicing that substantially increased the speed required to produce an image. For large I, the same image could be determined in roughly a fourth of the time. With this acceleration, it was possible to further refine the photomosaic algorithm, by introducing boundary matching to find subimages R i that matched both the 3x3 block, as well as matched the nearest n blocks to create as smooth a gradient as possible given the set of subimages. To continue this thought, the photomosaic algorithm was further extended to explore global optimization using an iterative simulated annealing optimization. As an open solution problem, using simulated annealing introduces additional questions such as the number of required iterations before an adequate convergence of the final image. Given a discrete dataset of subimages, an ideal solution is theoretically possible given enough processing time, but at what point remains an area for further research. Finally, we explored matching R i with the Discrete Cosine Transform in the frequency domain rather than matching a nearest-neighbor average in the image domain. While this direction also remains an avenue for further work, matching the frequency rather than color allows for more complex subimage matching, as well as low frequency searching and retrieval. References [1] BATTIATO, S., DI BLASI, G., FARINELLA, G. M., AND GALLO, G. Digital mosaic frameworks, [2] JAVIER BRACAMONTE AND ANSORGE AND PELLANDINI AND PIERRE-ANDRÉ FARINE. Efficient Compressed Domain Target Image Search and Retrieval. In CIVR 05: Proceedings of the 4th International Conference on Image and Video Retrieval pp [3] DI BLASI, G., AND PETRALIA, M. Fast photomosaic, [4] FINKELSTEIN, A., AND RANGE, M. Image mosaics. In EP 98/RIDT 98: Proceedings of the 7th International Conference on Electronic Publishing (London, UK, 1998), Springer- Verlag, pp

10 [5] KIM, J., AND PELLACINI, F. Jigsaw image mosaic. In SIGGRAPH 2002 Conference Proceedings (2002), vol. 38, pp [6] KLEIN, A. W., GRANT, T., FINKELSTEIN, A., AND COHEN, M. F. Video mosaics. In NPAR 2002: Second International Symposium on Non Photorealistic Rendering (2002), pp [7] SILVERS, R. Photomosaics. Henry Holt and Co., Inc., New York, NY, USA, [8] TRAN, N. Generating photomosaics: an empirical study. In SAC 99: Proceedings of the 1999 ACM symposium on Applied computing (New York, NY, USA, 1999), ACM Press, pp [9] ZHANG, Y. On the use of cbir in image mosaic generation,