An advanced greedy square jigsaw puzzle solver

Transcription

1 An advanced greedy square jigsaw puzzle solver Dolev Pomeranz Ben-Gurion University, Israel July 29, 2010 Abstract The jigsaw puzzle is a well known problem, which many encounter during childhood. Given a set of parts, one needs to reconstruct the original image. This problem is known to be NP-complete. In this paper I describe a greedy solver with several improvements. Before discussing the general idea and improvements, I will discuss some important known metrics that will help us clarify the solver algorithm and the desired result. I will show several new compatibility metrics, and will focus on one that is better than the best known metric. In addition to compatibility metrics, performance metrics and estimation metrics will be explored. I will explain the generic algorithm and the additional modules that can be plugged-in to the algorithm. I will introduce the shifting problem, and a possible solution using a segmentation module. Since one cannot write a solver without some test results, I will present a benchmark of the solver performance. 1 Introduction Given n non-overlapping jigsaw puzzle parts, one needs to reconstruct the original image. This problem was proved to be NP-complete[1]. My quest for a decent solver started with the brute force approach. This method will have a complexity time of O(n!), which of course is impractical for even relatively small puzzles. Alajlan[2] suggested using dynamic programming and showed that he is able to achieve a 100% precision rate for puzzles up to 8 8. Early dynamic programming algorithms for solving O(n!) problems have managed to run in O(n 2 2 n ) time and use exponential space[3]. By using inclusion-exclusion Karp[4] was able to solve similar problems within a polynomial factor of 2 n of time and polynomial space. This means that the use of a sophisticated exact algorithm for large inputs is also impractical. Leaving exact algorithms, the next stop in my quest was convergence algorithms and heuristic models. Toyama et al.[5] used an interesting genetic algorithm. They were able to solve, like Alajlan[2], puzzles up to 8 8. The 1

2 most recent and advanced solver was developed by Cho et al.[6]. They developed a graphical model and used a loopy belief propagation technique. They reported solving puzzles over 400 parts. Their algorithm is claimed to be the state-of-the-art. In addition to their algorithm they have introduced several metrics to which I will refer. The last type of algorithms I will discuss is greedy algorithms. If we could obtain an accurate parts compatibility function then solving jigsaw puzzle problems could be made in polynomial time by a greedy algorithm[1]. This fact might be a good reason to why greedy algorithms became so popular. Greedy algorithms are known to solve puzzles of up to 320 parts (16x20)[7]. I decided to develop a greedy algorithm. To create a successful one, I was in need for a good parts compatibility function. As mentioned before, that is the holy grail of greedy algorithms. In addition to that, I decided to reinforce the algorithm with some computer vision related algorithms to target specific problems. As opposed to some known solver implementations, no use of evidence or clues is found in the solver. In addition, the solver cannot use the original image to determine convergence. Because of that, a new type of metrics is introduced: estimation metrics. The solver operates on square parts and not on the classical jigsaw parts. This means that the compatibility function can only rely on the image content. The loss of constraint leads to a much more difficult problem to solve, but also to a solver that is easier to code. Summary 1 This paper offers the following contributions A new compatibility metric that is better than the best known metric Introducing a new type of metrics, when no clues are allowed: estimation metrics Introducing the shifting problem and a possible solution using segmentation A greedy algorithm with modules approach 2 Metrics 2.1 Compatibility metrics A compatibility metric is the foundation of almost every jigsaw solver. Its purpose is to convey the likelihood that two given parts are neighbours. Notice that there could be 4 different types of relations between the parts. We will denote the relation by l. Part j could be placed to the left/right/up/down of part i. Thus we seek for a probability function C(i, j, l) that is as close as possible to the actual parts placement. The optimal compatibility function will return 1 iff part j is placed in relation l of part i, and 0 otherwise. 2

3 In order to compare between compatibility functions, we will denote the classification criterion as the ratio between correct placements and the total number of placements. Part j in relation l to part i in the original image is defined as a correct placement by the compatibility metric C if the following hold: k P arts, C(i, j, l) C(i, k, l) Cho et al.[6] have evaluated five compatibility metrics. Among the five, we will discuss only Dissimilarity-based compatibility, since it is by far the best compatibility metric tested. Figure 1: Cho et al.[6] test results. Note that the fifth metric Cho et al. is a combination between dissimilarity-based compatibility and the statistics-based compatibility Dissimilarity-based compatibility Cho et al.[6] defined the metric as summing the squared color difference along the abutting boundaries. Each part is represented by a K K 3 color matrices, where K is the parts square edge length. For example, the left-right dissimilarity for parts x i, x j is defined as: D LR (x i, x j ) = K k=1 l=1 3 (x i (k, K, l) x j (k, 1, l)) 2 Since the dissimilarity returns a non-negative number, I convert it to a probability. I do so in a slightly different manner than Cho et al.[6]: ( C(i, j, l) exp D ) l(x i, x j ) quartile(i, l) 3

4 Where quartile(i,l), is the quartile of the dissimilarity between all other parts in relation l to part i. It is interesting to see that the dissimilarity-based compatibility received the highest accuracy (see Figure 1), even though it is the simplest and most intuitive metric. Because of that, I decided to search for a better compatibility metric, which is similar to the dissimilarity-based compatibility Backward difference-based compatibility I take a 2-pixel band along the abutting boundaries, and use backward difference to estimate the value of the next and unknown pixel. I sum the squared color difference between the actual pixel value and the estimated value; this can be done for both parts. The idea derives from the notion that most real world images have gradient coloring. Therefore, I expect this metric to be more accurate then the dissimilarity metric Central difference-based compatibility Similar to the backward difference metric, only now I use the neighbour pixel from the other part to find the estimated central difference of the boundaries pixels. Using the estimated central difference, I derive the neighbour s pixel value. As I will show shortly, these metrics did not offer better results even though, they use more data and try to estimate the neighbour pixel rather then just calculating the difference between neighbour pixels. I have also tried metrics that are a combination of difference-based and dissimilarity-based metrics. They showed an improvement but were still not able to rise above the dissimilarity metric. An interesting development occurred when I tried the different approach to create an even more simple metric Square absolute dissimilarity-based metric The square absolute dissimilarity (SAD), is the same as the dissimilarity metric, the only difference is that it sums the squared root of the absolute color difference along the abutting boundaries. For example, the left-right SAD for parts x i, x j is defined as: SAD LR (x i, x j ) = K 3 k=1 l=1 x i (k, K, l) x j (k, 1, l) I have assembled a test sample of 10 images. All images have the same size of I tested the different metrics; each time with different part size and number of parts. The results are shown in the following figure. 4

5 Figure 2: Comparing the metrics accuracy. Notice that the dissimilarity metric attained similar results as to those in the Cho et al.[6] test, as seen in Figure1. Surprisingly the SAD metric has achieved a significant improvement over the dissimilarity metric. 2.2 Performance metrics Since finding the original image is a hard problem, measuring the quality of a given solution is an important tool in the ongoing road towards a high quality solver. Cho et al.[6] introduced three metrics that compare a given solution with the original image. I will discuss two of them Direct comparison metric The metric is calculated according to the ratio between the parts in the solution that are placed in their original location and the total number of parts Neighbour comparison metric This metric calculates the ratio between the number of correct neighbours placement for each part and the total number of neighbours. 2.3 Estimation metrics During the execution of the solver, using any performance metrics to imply the convergence of an algorithm is equivalent to the usage of clues (since the original image is unknown). In order to have a sense of the current solution 5

6 quality, I introduce a new term estimation metrics. These metrics are intended to estimate the quality of the current solution Best buddies metric This metric calculates the ratio between the number of best buddies neighbours and the total number of neighbours. Where two parts i, j, their relation l 1 and opposite relation l 2 are said to be best buddies iff the following holds: k P arts, C(i, j, l 1 ) C(i, k, l 1 ) p P arts, C(j, i, l2 ) C(j, p, l 2 ) It means that both parts agree that the other part is their best neighbour in a certain completing relation, and thus are best buddies Compatibility multiplication metric This metric calculates the compatibility multiplication of all relations between neighbour parts in the current solution. 3 The shifting problem Comparing the performance metrics[2.2] on different solver results leads to an interesting observation. It seems that solvers usually build an image that contains regions which are assembled correctly. However those regions are often placed shifted to their original location. This might derive from the fact that a compatibility function will only tell us the likelihood that two given parts are neighbours but not where to place them. These can eventually imply that the same image will receive a different quality score by each performance metric. The direct comparison metric[2.2.1] will not tolerate a part that is not placed in its original location, and thus will probably grant a low score. On the other hand, the neighbour comparison metric[2.2.2] will disregard that regions were shifted, and thus will probably award the solution with a higher score. The following figure3 shows exactly that. 6

7 Figure 3: Cho et al.[6] test results. Comparing the performance metrics accuracy. Notice the high average score for neighbour comparison as opposed to the low average score for direct comparison. The cluster comparison is not discussed in this paper. I decided to define this phenomenon as the shifting problem. A solver that had created a solution could try to improve it, by identifying and moving regions. There could be many different implementations that target this problem. I decided to identify regions via segmentation, and offer a basic method for shifting the regions. Segmentation in computer vision, is a technique to partition an image to meaningful regions. The regions are non-overlapping and their union construct the full image. I used the region growing segmentation algorithm, where the segmentation predicate was based on the best buddies approach[2.3.1]. After creating the segment map, I used the biggest segment as an anchor and moved its frame across the image. For each movement of the frame, I located the other parts in a greedy manner and used estimation metrics[2.3] to single out the best placement. 7

8 (a) Greedy solver basic (b) The basic solution (c) Best shifted solution solution segmentation map Figure 4: The greedy solver generated a solution for a image divided into 100 parts (a). Using the segmentation algorithm, a segments map was created (b). The biggest segment was shifted, and the best placement was at its original location, thus resulting in a new solution that happened to be the original image (c). 4 A Greedy algorithm with modules approach The solver algorithm is not a simple greedy algorithm. I believe that a better approach is to construct a chain of algorithmic generic modules. Each module in the chain will target a specific problem, and will be able to be implemented in various ways. Figure 5: The solver chain module. Each element in the chain is a generic module. In several modules you can see various implementation methods. 8

9 The first module is responsible for calculating the compatibility values. It can be done with one of the several known metrics. The second module is responsible for improving these results. It can use a broader observation than just looking at a single part. I tried to implement such a module using relaxation labeling[8], but unfortunately the module did not converge to a better compatibility indication. The third module is the greedy algorithm. It selects a part and places it in a cell. From that point, an iterative process of selecting the best part placement among unplaced parts and neighbours of placed parts is activated until all parts are placed. The fourth module is responsible for solving the shifting problem. 5 Solver test results The following benchmark shows the solver performance results. The test was executed on a sample of 10 images, and all images have the same size of The performance was calculated with the direct comparison and neighbour comparison metrics. Figure 6: The No improvements bar indicates the usage of dissimilarity-based compatibility without activating the shift module. Notice that when the shift module was activated there was a significant improvement. 9

10 Figure 7: The SAD metric helps the greedy algorithm with making the right choices in comparison to the dissimilarity metric. The shift module offers an additional improvement. It can be seen in figures (6, 7) that the SAD compatibility metric and the usage of a shift module offers a great deal of improvement to the conventional greedy algorithm. Inspecting results from a few test runs will convey the same perception. (a) Greedy with no added improvements (b) With SAD metric (c) With SAD metric + Shift Figure 8: An image divided into 256 parts and three possible solutions with different solver configurations. Figure (a): 0% direct, % neighbour, 18 seconds. Figure (b): 0% direct, % neighbour, 18 seconds. Figure (c): 100% direct, 100% neighbour, 23 seconds. 10

11 (a) Greedy with no added improvements (b) With SAD metric (c) With SAD metric + Shift Figure 9: An image divided into 256 parts. This image contains a relatively simple object to reconstruct, but offers a difficult background. (a) Greedy with no added improvements (b) With SAD metric (c) With SAD metric + Shift Figure 10: An image divided into 256 parts. This image was taken from a computer game named GTA. The image was generated by a computer, and contains repetitive patterns. 6 Conclusion I have introduced a new and tested compatibility metric, that seems to offer a significant improvement over known metrics. The aspiration for a solver that does not use clues led to the foundation of estimation metrics. The shifting problem was introduced, and a solution via segmentation was implemented and tested. Using the previous methods, I presented a greedy algorithm with modules approach, that employs a chain of modules. When assembled together, they provide a strong solver. 11

12 References [1] E. D. Demaine and M. L. Demaine.: Jigsaw puzzles, edge matching, and polyomino packing: Connections and complexity. Graphs and Combinatorics, 23, (2007) [2] Naif Alajlan.: Solving Square Jigsaw Puzzles Using Dynamic Programming and the Hungarian Procedure. American Journal of Applied Sciences 6 (11): , (2009) [3] Bellman, R., Combinatorial Processes and Dynamic Programming, in Bellman, R., Hall, M., Jr. (eds.), Combinatorial Analysis, Proceedings of Symposia in Applied ematics 10, American ematical Society, pp (1960) [4] Karp, R.M., Dynamic programming meets the principle of inclusion and exclusion, Oper. Res. Lett. 1: 4951, doi: / (82)90044-x (1982) [5] Fubito Toyama,Yukihiro Fujiki,Kenji Shoji, and Juichi Miyamichi, Assembly of Puzzles Using a Genetic Algorithm, Faculty of Engineering, Utsunomiya University 7-1-2, Yoto, Utsunomiya-shi, JAPAN (ICPR02) (2002) [6] Taeg Sang Cho, Shai Avidan, William T. Freeman, A probabilistic image jigsaw puzzle solver, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2010) [7] T. R. Nielsen, P. Drewsen, and K. Hansen. Solving jigsaw puzzles using image features. PRL, [8] Robert A. Hummel, Steven W. Zucker. On the Foundations of Relaxation Labeling Processes IEEE Transactions on pattern analysis and machine intelligence, VOL. PAMI-5, NO. 3, May (1983) 12