Professor Louay Bazzi

Transcription

1 Final Year Project Representing & Matching Bi-level Images with Holes Under the supervision of Professor Louay Bazzi Group Members Bilal Fadlallah CCE ID # bhf01@aub.edu.lb Mazen Hajjar CCE ID # mah39@aub.edu.lb Oliver Saleh CCE ID # oss04@aub.edu.lb Submission Date: Tuesday, May

2 Table of Contents Preface Abstract..6 Chapter I. Introduction 7 Chapter II. Literature Review 10 II.1. Concavity Trees II.2. Tree Edit Distance Matching II.3. Complexity Analysis II.4. Further Elaboration...16 II.5. Limitations.. 16 Chapter III. Alternatives & Project Goals 17 III.1. Research Alternatives III.2. Alternatives Assessment. 18 III.3. Project Goals...19 III.3.1. Enhancing the Performance of the TED Algorithm III.3.2. Holes Incorporation for Representation and Matching III.3.3. Implementation and Testing Of the Upgraded Algorithm Page 2 of 76

3 Chapter IV. Analysis and Design 22 IV.1. MEX Files IV.1.1. What are MEX Files...22 IV.1.2. Why MEX Files...22 IV.2. Incorporating Holes in Concavity Trees IV.3. Relative Tree Edit Distance Matching Algorithm IV.4. Random Lines Intersection Matching Algorithm 28 IV.5. Datasets of Images IV.5.1. MPEG7 Dataset IV.5.2. Synthetic Dataset Chapter V. Implementation and Testing 35 V.1. Implementation V.1.1. Converting for-loops to C++ Via MEX Files...35 V.1.2. Implementation of Hole Representation V.1.3. Implementation of Hole Representation V.1.4. Software Implementation...43 V.2. Testing V.2.1. Testing the Algorithms for Image Matching V.2.2. Results Analysis V.2.3. Testing the Running Time Chapter VI. Completed Work 52 Page 3 of 76

4 Chapter VII. Budget, Timeline & Tasks 54 VII.1. Budget VII.2. Distribution of Taks...54 VII.3. Timeline..56 Chapter VIII. Design Constraints 58 Chapter IX. Conclusion & Further Research 59 Appendix A 60 Proximity Software User Manual.60 Bibliography 73 List of Figures & Tables 74 Acknowledgments 76 Page 4 of 76

5 Preface This report comprises the summary of work we achieved during our final year project (FYP). The task we chose to carry out during the year is the design and implementation of a method that represents and matches bi-level images with holes using dynamic programming in an efficient way. The effort is conducted under the supervision of Professor Louay Bazzi. This report provides a detailed exposition of the tasks we completed towards accomplishing our project goals. We begin by introducing the topic we selected and explaining the primary reasons that encouraged us to select this particular subject as our FYP. We provide all the necessary background on the topic through an exhaustive literature survey. We then go over the primary goals we achieved and provide a brief explanation of the path that was followed in order to achieve these goals. We then present the phases for implementing the various tasks of the project, and provide test results reflecting the improvement in performance of the algorithm after having been upgraded. Afterwards we provide a description of the way we divided the various tasks amongst ourselves, and provide tables portraying the timeline of our project as well as all the important milestones. We conclude our report by mentioning the design constraints we came across throughout the life of the FYP and how these constraints were addressed. Page 5 of 76

6 Abstract Concavity trees are data structures used to represent 2-D shape images based on their meta-concavities and convex hulls. The following report expands the scope of concavity trees to cover the compression and retrieval of 2-D bi-level images containing holes, emphasizing on improving the performance and complexity of such an approach. It also shows how to apply the concept of edit distance on these structures to determine a scaled level of similarity between any two given images, a task required in many applications. The running time of the algorithms will be improved by using MEX Files to interface with C++. The concavity tree data structure will be upgraded to represent holes and the matching process will be performed by a multitude of methods including Relative Comparison and Random Lines Intersection. A user friendly GUI is developed to facilitate the compression and matching of bi-level images. The timeline, budget and division of tasks for the two semesters we worked on this project are included in the report, as well as the detailed phases of the project. Also included in the report are the various alternatives we came across, the design goals and decisions we made, the pseudo code of the algorithms we used, the testing results, and the conclusions we came out with regarding the performance, and the weaknesses and strengths of the upgraded algorithms. Page 6 of 76

7 Chapter I. Introduction The purpose of our Final Year Project is to design and implement an efficient method for compressing, retrieving, and matching bi-level images with holes using dynamic programming in addition to conceive, if possible, other enhancements of the overall methodology. The area of Pattern Recognition is one of the fields that have undergone much research in the recent years, and is still the subject of much scrutiny today. The development of a variety of multipurpose algorithms in touch with this field has led to many technological advances in various dependent domains such as computer vision, virtual reality systems, space exploration, criminology, cellular biology, security systems (fingerprints recognition, iris scans ). The field of pattern recognition or analysis holds much prospect in the area of digital image processing, specifically that of shape identification. Its development has proved to be crucial to meet the growing demands for an ever advancing world of technology. Today a simple search over the internet would reveal a large plethora of different techniques and algorithms for specialized image and pattern recognition. Each algorithm and technique is directed towards a specific application of pattern and image recognition, and as a result, the performance of each technique is application specific and provides maximum performance only for a finite set of applications, while providing less than acceptable performance when used in other applications. When we speak of performance of a specific method or algorithm we usually have the following criteria in mind: 1- Speed 2- Accuracy 3- Robustness 4- Ease of implementation 5- Cost of implementation and maintenance 6- Potential to be improved Ideally we would like to come up with a methodology that meets all of the above criteria. Unfortunately this is not possible and as a result there will always exist a certain trade-off. Page 7 of 76

8 We can classify pictorial data input for computer processing into four different categories: 1- Simple RGB colored pictures 2- Full gray scale images 3- Bi-level pictures (i.e. black and white pictures) 4- Continuous curves and lines 5- Sets of discrete points spaced far apart However the span of images we will deal with in this project are two dimensional bi-level (i.e. black and white bit-mapped) images. Dealing with bi-level images is similar to dealing with shape representing images. This is because shape can be considered as solely determined by one of two levels. The case of colored images offers more challenge since shape can only be determined based on color contrast. In other words, color contrast is far more obvious in bi-level images than that of colored images. Also the primary feature on which the recognition process is based on is shape. In order to be able to compare two bi-level images efficiently and rapidly, a certain image compression scheme is necessary because a method conceived to work directly on the matrix of pixels level to match images will be both time consuming and less prone to detect the patterns inside these images. There exist many image compression schemes such as JPEG and MPEG; however a really appropriate compression scheme for the application we have in mind is Concavity Tree Compression, which poses a challenge to the first two in specific cases which we will be outlining in the next chapters. During his summer internship, one of the members of our group, Bilal Fadlallah, worked as a research assistant at the University of Waterloo, located in the town of Waterloo in Ontario Canada. The project he worked on aimed at developing a new way of matching concavity trees using dynamic programming. Concavity Trees are structures that have been elaborated on by Sklansky many decades ago. Ossama El Badawy, the phd student he jointly worked with, applied this concept in such a way as to compress and retrieve images. Matching trees in the phd thesis was however based on conformal mapping. The new algorithm was based on the concept of transforming one tree into another. Although a paper has been written that illustrates the design and implementation and testing of the algorithm, the matching algorithm still exhibits many weaknesses. In addition to the elaboration of a new algorithm that deals with these weaknesses, our final year project aims at extending the whole idea of compressing, retrieving, and matching of bi-level images to include images with holes, along with trying to develop the overall process from both a qualitative and quantitative perspective. Page 8 of 76

9 Before we expose the weaknesses of the algorithm, it should be made clear at this stage that we have used in our project parts of code written by Ossama El Badawy with his agreement and abiding by Copyright Laws. Next we start by showing the advantages for the use of Concavity Trees to represent bi-level images. Two advantages are actually predominant: 1- Compressing the image and reducing the memory requirements for its storage drastically, while retaining exactly enough information for a very close reconstruction of the original image (or at least with an accuracy depending on a user-defined parameter). 2- Relatively fast and accurate comparison of 2 bi-level images, by comparing their concavity trees. Concavity trees provide a considerable amount of potential to be further researched. Due to the very nature of the application, this technique has only been used so far for the compression and comparison of bi-level images without holes. More details about extracting concavity trees from an image will be given in the Literature Review chapter. Page 9 of 76

10 * FYP Report Chapter II. Literature Review II.1. Concavity Trees [1]*: A concavity tree is a rooted tree in which the root represents the whole object whose shape is to be analyzed or represented. The next level of the tree contains nodes that represent concavities along the boundary of that object. Each of the nodes on the following levels represents one of the con-cavities of its parent, i.e., its metaconcavities. If an object or a concavity is itself convex, then the node representing it does not have any children. Figure II.1.1 shows an example of a shape (a), its convex hull, concavities, and meta-concavities (b), and its corresponding concavity tree (c) (Refer to [1]). The shape has five concavities as reflected in level one of the tree. The four leaf nodes in level one correspond to the highlighted triangular concavities shown in (d), whereas the non-leaf node corresponds to the (non-convex) concavity shown in (e). Similarly, the nodes in levels two and three correspond to the metaconcavities highlighted in (f) and (g), respectively. Typically, each node in a concavity tree stores information pertinent to the part of the object the node is describing (a feature vector for example), in addition to tree meta-data (like the level of the node; the height, number of nodes, and number of leaves in the subtree rooted at the node). Fig. II.1.1. An object (a), its convex hull and concavities (b), the corresponding concavity tree (c), and contour sections corresponding to concavities (d-g). Courtesy of [1]. * Taken from Reference. Page 10 of 76

11 The process of extracting the concavity tree from a 2D image can be done through the following steps: 1. Compute the convex hull of the image 2. Start from the leftmost bottom end of the figure 3. Follow the contour points starting from this point 4. Detect concavities: For each concavity apply the algorithm recursively until we arrive to a concave shape. If concavity has meta-concavities or sub-concavities then they will be represented as nodes in the second or n th level of the tree, with respect to their order (meta-concavity of order 2 or n). An illustrative Figure II.1.2 is shown below. This is a metaconcavity of the third order. It is represented at the third level. 6. This is the final node found in the first level of the tree. 5. This is the fourth node found in the first level of the tree. 1. Start Here 2. This is the first node found in the first level of the tree. 3. Applying the algorithm recursively on this part will yield one concavity having 2 metaconcavities each having one concavity 4. This is the third node found in the first level of the tree. Fig. II.1.2. The same object as in I.2.1 with its corresponding concavities with their position in the tree. Page 11 of 76

12 After introducing the way concavity trees are being extracted from images, we now move to explain how images are being matched using their concavity tree representation. II.2. Tree Edit Distance Matching When we talk about matching, we mean finding the best match or matches to an image from a set of given images. This can be directly done by defining a distance between any two images. The least distance image from as set of images to a given one is then the best match we are looking for. Two methods have been suggested to match concavity trees. The first one has been implemented by Ossama El Badawy and Mohamad Kamel [2], and uses mappings from nodes in the first tree to another. The second uses the concept of Tree Edit Distance in the matching process [3]. We will focus on the latter method because it is more efficient and accurate, especially for sets of images that exhibit translations, rotations and flipping [3]. Next we include literature showing the methodology followed by the Tree Edit Distance approach. Note that this part was also included in the report submitted to the University of Waterloo [3]. As we have said, matching concavity trees was implemented in [2] based on the structure of the tree and the information stored in every node: the contour points, the SCX (Solidity, eccentricity, extent) vector, the level, the depth Tree Edit Distance presents a new method for matching two concavity trees based on the shape of the trees and a set of elementary operations used to transform one tree into another. The tree edit distance algorithm can be used to compare any type of trees. Actually, the problem of transforming trees is a crucial issue in ordered tree comparison that is still being researched for improvement. The tree transformation is a set of elementary operations associated with different weights, and the tree edit distance is the cost of the least-cost sequence of operations transforming one tree into another. The idea of the simple edit distance algorithm is the following: Algorithm Idea (Refer to [4] for a detailed analysis of the algorithm): Let T = (V, E 1 ) and S = (W, E 2 ) be two ordered trees. An elementary edit operation on T and S is: 1. Deletion: (v, λ ) or v λ where v V. 2. Substitution: (v,w) or v w where v V and w W. 3. Insertion: ( λ,w) or λ w where w W. Page 12 of 76

13 These elementary operations on T and S have different costs constrained by the cost function γ such that [4]: γ (v,w) 0. γ (v,w) = 0 if and only if v = w. γ (v,w) =γ (w,v). γ (v,w) γ (v,z) + γ (z,w). As a result, the cost of a transformation E for T into S is given by: γ ( E) = v w E γ ( v, w). (, ) So, the edit distance between ordered trees T and S is: { γ } Dist ( T 1, T 2) = min ( E ), where E is a valid transformation of T into S. First, the two concavity trees should be stored in a pre-order traversal. Then we define the tree edit graph of T and S as the graph where the vertices are of the form {vw} for each pair of nodes v { V 0} V and w { W 0} W, where v0 V and w0 W are 2 dummy nodes. The following conditions are the base for building the tree edit graph vertices: if depth[v i+1 ] depth[wj+1], then (v i w j,v i+1 w j ) exists. if depth[v i+1 ] = depth[w j+1 ], then (v i w j,v i+1 w j+1 ) exists. if depth[v i+1 ] depth[wj+1], then (v i w j,v i w j+1 ) exists. for 0 i n1 and 0 j n2, where n1 and n2 are the number of nodes in T1 and T2 respectively, with the nodes numbered according to a pre-order traversal. Example: Let T1 = (V1, E1) and T2 = (V2, E2) be two ordered trees, as shown in Figure II.2.1. T1: T2: (a) (b) Fig II.2.1. T1 (a) and T2 (b) are numbered according to a pre-order traversal. Page 13 of 76

14 The resulting tree edit graph of the trees is shown in Figure II.2.2: Figure II.2.2: The figure shows the tree edit graph of transforming Tree1 into Tree2. Given the Substitution Cost = 0, the Deletion Cost = 1 and the Insertion Cost = 1, the shortest path is highlighted in red and has a cost of 2 whereas one of the possible paths is highlighted in green and has a cost of 6. The implementation of the algorithm can be performed through the following pseudocode: Algorithm: Matching Concavity Trees Using Tree Edit Distance Notation: T1 and T2 are two ordered concavity trees. T1 = (V, E1); T2 = (W, E2); T1 is the tree with the minimum number of root children. V1 and V2 are two vectors. Function: Distance = fted(t1,t2) 1. Fix T1 and rearrange T2 in the second level to have greater similarity. 2. V1 = Pre-order Traversal (T1); 3. V2 = Pre-order Traversal (T2); 4. for i = 1: size of V1 5. for j =1: size of V2 6. if depth[v i+1 ] = depth[w j+1 ] 7. edges (v i w i,v i+ 1w j+1 ), (v i w i,v i+ 1w j ), (v i w i,v i w j+1 ) exist. 8. else if depth[v i+1 ] depth[w j+1 ] 9. edge(v i w j,v i+1 w j ) exists. 10. else 11. edge(v i w j,v i w j+1 ) exists. 12. end 13. end 14. end 15. Transform the edit graph to a directed connected graph. 16. Give weight for each edge according to operation (del, ins, sub). 17. Use Dijkstra to find shortest path between node V 1 W 1 & V size_of_v1 W size_of_v Distance = weight of the shortest path. Page 14 of 76

15 II.3. Complexity Analysis: The most time-consuming step in the algorithm is Dijkstra. Again, when taking T1 = (V1, E1) and T2 = (V2, E2): Let n = number of nodes in T1. Let m = number of nodes in T2. Assume m>n. We know that Dijkstra operates on the n x m matrix in which the tree edit graph information is stored. Hence the number of vertices in the directed graph will be the product of the two dimensions, i.e. mn. Now since the complexity of Dijkstra using binary heaps is of : T (, ) ( )log Dijkstra V E = E + V V, Where E is the number of edges in the graph and V is that number of vertices. Now since each vertex can at most be connected to three other vertices, the worstcase number of edges is of 3mn, and the running time of the algorithm will thus be: T ( n, m) = O{(3 mn + mn)log mn} = O{4mn log mn} fted But because m>n and the limitation of the number of nodes, we will hence have: T n m O m m O m m fted (, ) = {(4 )log } = {8 log } Now since the complexity of Dijkstra s algorithm for graphs with much less than n 2 edges and using a Fibonacci heap improves the running time to O(m + n log n), the algorithm will run in this case in: TfTED ( n, m) = O( E + V log V ) = O(3mn + mn log mn) = O m + m m = O m + m (3 log ) { (3 2log )} Usually, the number of nodes m is very small compared to the number of contour points, which will favor our algorithm in many applications of image matching. Furthermore, our method has many features that can vary the way matching is conducted. Varying the area parameter would result in different option while comparing a set of images. If you are comparing images that belong to the same object, decreasing the area parameter would help see the differences between the images because we would be increasing the weight of the operations in the low levels compared to the root substitution. But, if you are trying to match different objects, it would be better increasing the area parameter to give a higher weight to the general shapes represented by the roots. Page 15 of 76

16 II.4. Further Elaboration After building the tree edit graph, the different operations must be assigned variable costs according to the attributes of the nodes involved. Starting with the substitution cost, it is a function of the area of the concavity relative to the area of the image and the level of the nodes in the concavity tree. It also depends on the different parts of the SCX and the number of children of the two nodes. This cost was tested on different shapes on which it shows a great strength in calculating the distances between different kinds of concavities. Concerning the insertion and deletion cost, they must be equal in order to preserve the symmetric property of the Tree Edit Distance. This cost is directly proportional to the area of the concavity relative to the area of the root (object) and to the number of nodes present in the tree rooted at this node and inversely proportional to the level of the node in the concavity tree. Furthermore, we have multiplied the different costs by an area parameter raised to the power ratio of the area of the concavity to the area of the convex hull of the object which should give greater emphasis for the area factor in our cost analysis. As a result the operations cost can be shown as: Area ( N ) Area ( Object ) Insertion Cost ( N ) = Area Parameter f (# Nodes, Level ) Area ( N ) Area ( N ) Area ( Object1) Area ( Object 2) (, ) = (,, ) Substitution Cost N N Area Parameter f Level SCX Children 1 2 II.5. Limitations Although it shows noticeable improvements in many cases, the algorithm developed still shows many limitations. First the choice of the weights is empirical. This limits the accuracy of the method since empirical choices can not suit all cases even if they can cover a wide spectrum of these cases. Another reason for the non accuracy of the weights is their dependency on features already computed in the phase of concavity tree construction. Chapter III outlines some possible solutions to these limitations, where we introduce alternatives to build upon the concept we have just seen. Page 16 of 76

17 Chapter III. Alternatives & Project Goals III.1. Research Alternatives The alternatives we were able to choose from in the field of representing and matching images with compression and retrieval were the following: 1) Extend the concavity tree process to cover grayscale images. 2) Extend the concavity tree process to cover colored images. 3) Extend the concavity tree process to include images with holes. 4) Extend the concavity tree process to cover multi-object images. 5) Extend the matching process to include images with holes. 6) Extend the matching process to cover multi-object images. 7) Implement the Concavity Tree Algorithm using C++. 8) Design an algorithm for Shape Detection in 3D. 9) Fit theoretical weights computation in the algorithm. 10) Design extreme compression schemes for communication purposes. 11) Design recovery-based schemes for ultimate image retrieval. 12) Design precise algorithms assessment methods over big image sets. 13) Build incremental compression schemes for whole sets. 14) Include caching methods in the storage of image databases. 15) Integrate MEX Files interfacing in the Extraction and Matching Algorithms. 16) Develop a user friendly software and GUI using Matlab. Page 17 of 76

18 III.2. Alternatives Assessment Actually not all the alternatives we mentioned above were prone to being implemented in our FYP. We will briefly explain why some of the above alternatives were not feasible and then select the alternatives we focused on. 1) Extending the concavity tree process to cover grayscale images proves to shift the process far from shape detection, since shape in this case would take on a new definition. 2) Extending the concavity tree process to cover colored images exhibits the same problem as that with grayscale images. 3) Extend the concavity tree process to include images with holes is doable and has many rewards although it does requires much work and brain storming. 4) Extending the concavity tree process to cover multi-object images is also doable although it may require some changes to the matching concept. 5) Extending the matching process to include images with holes can be performed by defining new weights for the holes matching that can be added in a suitable way to the previous definition of the distance between two images. 6) Extending the matching process to cover multi-object images will require elaboration of specific schemes in the matching priority. 7) Implementing the Concavity Tree Algorithm using C++ is doable, and might lead to faster performance, but is tedious and takes away the advantage of using Matlab s environment and powerful built-in functions dealing efficiently with vector and matrix operations as well as powerful display tools. 8) Designing an algorithm for Shape Detection in 3D can be done in either of the following ways: Usage of the 2D shape detection to generalize it for 3D. Work on the 3D problem from scratch and following a similar approach to the 2D problem. 9) Fitting theoretical weights computation in the algorithm proves to be impossible to achieve since the matching algorithm should work for a large variety of images and this can only be accomplished by using empirical methods. Page 18 of 76

19 10) Designing extreme compression schemes for communication purposes can be achieved but with small benefit since this can be obtained by applying the concavity tree algorithm in a relatively lossy manner. 11) Designing recovery-based schemes for ultimate image retrieval can also be done but this can be obtained by a nearly lossless application of the concavity tree algorithm. 12) Designing precise algorithms assessment methods over large image sets is a direct consequence of designing a precise matching algorithm first. 13) Building incremental compression schemes for whole sets is doable but is largely dependant on the nature of the sets. 14) Include caching methods in the storage of image databases has some benefits, and is doable. 15) Integrating MEX Files interfacing in the Extraction and Matching Algorithms proves to be a convenient and useful method, since it combines C++ s speed of execution of lengthy for-loops, with Matlab s complex environment and powerful functions. 16) Designing a user friendly software using Matlab s GUIDE tool that implements the tree extraction and matching processes is doable and almost necessary so as to separate the end user from internal complexity of the fct and fted algorithms, while providing him/her with the benefits of the concavity tree compression and matching methods. III.3. Project Goals After closely analyzing each of the above alternatives we were able to select the alternatives we found most important and at the same time we believed were doable given the time and budget constraints of our FYP. Thus our FYP focused on achieving the following primary goals: 1) Improving the performance of the current Tree Edit Distance algorithm, which will be incorporated in the new matching algorithms. 2) Expanding the Concavity Tree representation of binary images to incorporate holes representation. 3) Designing an algorithm to match concavity trees obtained in the last step based on Relative Comparison of Holes. Page 19 of 76

20 4) Designing a second algorithm to match directly images with holes using Random Lines Intersection on the original images. 5) Implementing the enhanced algorithms using Matlab. 6) Enhancing the running time of all algorithms by using MEX Files that interface C++ with Matlab. 7) Designing a user friendly GUI and software that would place the user at a higher level and separate him/her from the details of the complex algorithms being utilized to compress, retrieve, and match the bi-level images with or without holes. III.3.1. Enhancing the Performance of the TED Algorithm We came up with several ideas for the improvement of the tree edit distance algorithm, namely: a. Checking if one of the two trees is actually any or close to any of the sub trees of the other tree will improve the matching rate because at the present time, the cost of matching a tree with a bigger one containing the first is more than the cost targeted. b. Since the running time of the algorithm is a function of the number of nodes, the performance is made faster by finding a limit to the number of nodes while still preserving the maximum possible useful information about the image. This limit can be achieved by one of two ways, namely: either taking into consideration a limited number of levels, or adjusting the concavity-size determining variable within the concavity tree creation algorithm. c. Optimizing the weights further to improve the accuracy of the results of our matching. d. Substituting the pre-order traversal of the concavity trees by an in-order traversal, this may improve its performance on some 2-D shapes. Experimental results may suggest a combination of traversals along with selective usage according to the shape image. e. Since for loops require very long execution time in Matlab, and since the Matlab implementation of the algorithm is highly dependant on a large number of relatively long (and often embedded) for-loops, the algorithm suffers from a large decrease in performance and increase in running time. Page 20 of 76

21 The algorithm's running time dependency on the programming platform being used can be largely reduced by rewriting all the time consuming for loops in C/C++, and calling them when necessary from Matlab using MEX files. MEX files will be described in sections IV.1 and V.1. III.3.2. Holes Incorporation for Representation and Matching The Concavity Tree representation will be modified to allow holes to be represented in a compressed form. The TED algorithm will then be adjusted so as to take holes, inside the binary images, into account during the matching process. The way we will achieve this is by XORing the original image containing holes with the same image, but with the holes filled out. We then follow a certain procedure to extract each individual hole from the object and represent it as a separate concavity tree. We will conceive a novel method for matching concavity trees for images with holes using the TED algorithm during the matching process. In order to solve the problem of the relative position of holes in the image, we will design another algorithm that dissects the bi-level images by random lines (or lines generated according to a user input), then compare the corresponding dissected parts of each image. The Hole Representation and Comparison Algorithms will be described in greater detail in chapters IV (section IV.2) and V (section V.1). III.3.3. Implementation and Testing of the Upgraded Algorithm We will provide working software written in Matlab (and interfaced with C++), that implements the upgraded TED algorithm. The reasons we chose Matlab and C++ as our programming platforms are that Matlab provides very efficient matrix operation capabilities(a must in the implementation), and C++ (when interfaced with Matlab through MEX files) provides much faster execution time when for-loops are being used (as is the case in the fct and fted algorithms). We will then compare the speed and accuracy of the results of the present algorithm with that of the improved algorithm. We will test our improved algorithm on the MPEG7 Dataset and on a Synthetic Dataset (containing images with holes) and compare the results with those of the present algorithm, and provide a detailed performance analysis indicating the strengths and weaknesses of the upgraded algorithm. Page 21 of 76

22 Chapter IV. Analysis and Design IV.1. MEX Files As mentioned in the previous section, much of the code implementing the tree edit distance algorithm is composed of lengthy for loops. Since the algorithm was implemented in Matlab, we were faced with the issue of drastic decrease in performance of the algorithm due to Matlab's short-coming when it comes to the execution of for loops. Thus in order to reduce the algorithm's running time dependency on the programming platform being used, we searched the net for a solution. The solution came to us in the form of MEX files. IV.1.1. What are MEX Files? MEX stands for Matlab Executable. They are a way to call your custom C/C++ routines directly from Matlab as if they were Matlab built-in functions. MEX-files are dynamically linked subroutines produced from C/C++ source code that, after being compiled, can be run from within Matlab in the same way as Matlab M-files or built-in functions. The external interface functions provide functionality to transfer data between MEX-files and Matlab, and the ability to call Matlab functions from C/C++ code. IV.1.2 Why MEX Files? The main advantage we are hoping to gain from the use of MEX Files is speed. We can rewrite the bottleneck computations, like for-loops, as a MEX files for increased efficiency [6,8]. The most time consuming function in the algorithm is the Dijkstra function which is composed of numerous lengthy for-loops. Rewriting it as a C++ MEX file would largely reduce its running time, and as a result increase the overall performance of the algorithm. In order to demonstrate the speed advantage of using MEX files to write timeconsuming for loops in C++, instead of Matlab, we wrote a function that contains several embedded for loops as both a mat file and as a mex file. We then compared their relative performance. Page 22 of 76

23 The function with its corresponding MEX File is the following: Matlab Code function k=just_for_fun(sentinel) total=0; i=0; j=0; k=0; for i=0:sentinel, for j=0:sentinel, for k=0:sentinel, total = total +1/6; total = total +1/6; total = total +1/6; total = total +1/6; total = total +1/6; total = total +1/6; if total >4000 total= total-98; end end end end k=total; Corresponding MEX File #include "mex.h" void mexfunction(int nlhs, mxarray *plhs[], int nrhs, const mxarray *prhs[]) { double *total; int i=0,j=0,k=0,sentinel; plhs[0] = mxcreatedoublematrix(1,1,mxreal); total = (double *)mxgetpr(plhs[0]); sentinel =(int)mxgetscalar(prhs[0]); *total=0.0; for (i=0;i<=sentinel;i++) for (j=0;j<=sentinel;j++) for (k=0;k<=sentinel;k++) { *total = *total +1.0/6.0; *total = *total +1.0/6.0; *total = *total +1.0/6.0; *total = *total +1.0/6.0; *total = *total +1.0/6.0; *total = *total +1.0/6.0; if (*total > 4000) *total= *total-98.0; } } Upon running both functions for a various number of input values we obtained: Sentinel Matlab sec sec sec sec MEX sec sec sec sec % Improvement 44.38% % % % Avg. % Improvement = % Thus we obtain a % improvement in running time when using the mex file over using the mat file. This clearly shows the advantage of writing time consuming for-loops in C++ instead of Matlab. Page 23 of 76

24 IV.2. Incorporating Holes in Concavity Trees The original comparison algorithm supports bi-level images without holes. We found it necessary to modify the algorithm in order to support images with holes. The addition of holes will expand the range of images that the algorithm can work on, and will improve the quality of the images that are to be compared. The original algorithm could only be used to compare black and white images representing 2-D shapes. However, with the introduction of holes, we can represent 3-D shapes and introduce more clarity in the image. The addition of holes is a major improvement of the Tree Edit Distance algorithm since it allows it to operate on a set of black and white images each with a main black shape and any number of holes inside the shape. After we decided to implement the Tree Edit Distance with holes, we had to search for previous ideas and research that was conducted concerning the comparison of two bi-level images with holes. We found few ideas that stick with this subject and no previous implementation of an algorithm that matches two bi-level images with holes. Therefore, we had to find a new technique for solving the problem. The original algorithm stores any two 2-D shapes with no holes in corresponding concavity trees, and then uses the Tree Edit Distance algorithm to compare the tree with any other tree representing another 2-D shape. Therefore, we had to find first a way of identifying holes in a certain image and storing them as concavity trees, then using the concavity trees of the holes to compare images with holes. The algorithm for finding the holes in a given image works as follows: Suppose that we have a certain image containing an object with a number of holes. Consider for example the image shown in Figure IV.2.1. This figure shows an object with a black background containing 3 holes. For simplicity, we assumed that the object is an ellipse having no concavities since what we care about for the time is finding the holes. 3 holes 3 holes The bitmap image is stored in a matrix of 0's and 1's where background pixels are represented as 0's and foreground pixels are represented as 1's. For the given image, the black pixels representing the contour object are 1's, and the white pixels representing the holes and the background are 0's. Bitmap image Image Contour Object contour object Figure IV.2.1.: A Bi-level Image Containing Holes Page 24 of 76

25 First, we need to find the contour image and place it in a concavity tree. This is done by finding the contour of the image which is the ellipse, and then filling the inside of the ellipse by black pixels. The result is shown in Figure IV.2.2. This is the contour object that represents the image with no holes. This image is then treated as an ordinary 2Dshape with no holes; and by applying the original concavity tree algorithm, we can transform the binary later for comparison. Figure IV.2.2: The Filled Image Image with no holes Image with no holes Next, we need to find the holes and store them each in its corresponding concavity tree. We start by applying an XOR operation between Figure IV.2.1., which shows the image with holes, and Figure IV.2.2., which shows the image with no holes. The result is shown in Figure IV.2.3. We get the 3 holes without the original object, and the three holes are now represented as foreground pixels with a value of 1. Figure IV.2.3: The Hole Extraction Process To extract the holes from this figure, we start by applying the concavity tree algorithm on it. The algorithm usually works on figures containing a single object, and it starts by finding the contour of the object by locating the left uppermost foreground pixels. Therefore, applying the algorithm on the result of the XOR will catch the first hole that it encounters which is the left uppermost and ignore the other 2 holes. The hole that is extracted is shown in Figure IV.2.4, and we have now another concavity tree that stores a hole as an object. Figure IV.2.4: The First Extracted Hole Page 25 of 76

26 To find the other two holes, we XOR the figure containing the 3 holes with the figure containing the first hole. The result is shown in Figure IV.2.5. The new figure contains the two remaining holes. Figure IV.2.5: The Second XORing Process By applying the concavity tree algorithm on the resultant figure, one of the holes will be extracted and will be placed in a concavity tree. The new hole is shown in Figure IV.2.6. Figure IV.2.6: The Extracted First Hole We still need to find the remaining hole and place it in a concavity tree. This is done by XORing the figure containing the two holes with the figure containing the hole that was last extracted. The result of the XOR is shown in Figure IV.2.7. The resultant figure contains the last hole, and applying the concavity tree algorithm will extract the hole to a concavity tree. Figure IV.2.7: Extracting The Second Hole The last hole is shown in Figure IV.2.8. By XORing this figure with the previous one, the result is an empty image consisting of white pixels only. Figure IV.2.8: Extracting The Third Hole Page 26 of 76

27 The final XORing operation is illustrated in Figure IV.2.9. Figure IV.2.9: The Final XORing Operation The empty figure implies that there are no more holes and the algorithm automatically stops. By now, we have concavity trees representing the contour object and the holes inside it separately. This algorithm works for any number of holes. It starts by detecting holes and places each in a concavity tree until no holes are left out. It also works for the case when the object or its holes have concavities since the object and its holes will be stored in concavity trees, and these trees are responsible for representing the concavities and meta-concavities of an object to use it in the comparison with another object. To compare two images with holes, we should start by finding the holes of each object and placing them in concavity trees as well as the contour objects, and then we can apply the Tree Edit Distance algorithm to compare any two trees. Once we have the holes stored in concavity trees, there are several methods for comparing the two images using the Tree Edit Distance based on whether the number of holes of the two images is the same or not. These methods are discussed later, as well as the implementation of this strategy using Matlab. IV.3. Relative Tree Edit Distance Matching Algorithm This algorithm consists in comparing the data structures obtained in the previous section in order to match any two images with holes via the trees generated via the new concavity tree algorithm. In other words, the trees generated by the previous algorithm will the ones considered during the matching process. For the sake of example, let A and B be respectively the concavity trees with holes of Images A and B. The concavity tree of each will contain the concavity tree of the whole shape to which we append sub-concavity trees for each of the nodes. If the two trees have the same number of holes, we can apply a novel version of the Tree Edit Distance Algorithm to handle the holes issue by comparing the holes ones by Page 27 of 76

28 one and giving weights to each of the holes. The weights will have to comply with important criteria such that the area of each hole etc Another alternative would be to compare the holes by the same manner but not at random, i.e. not the first one of A with the first one of B, instead we take the best hole match in B to the first ones of A. This is susceptible of consuming more time but the result will be more accurate. One thing that should be mentioned at this stage is the case when we match image with holes with another one that doesn t contain holes, or at least contains holes but with a smaller number. In this case we propose to compare the hole of the image with a dummy hole. A dummy hole is a special concavity tree made of a unique node. In other words, we are computing the distance between the hole and a standard invariant shape to which we refer by the name dummy hole. Therefore, a large or complex-shaped hole will exhibit a bigger distance from the dummy hole than a small or simple one, and this will influence the final distance between the image with holes and that without holes or with a bigger number of holes. IV.4. Random Lines Intersection Matching Algorithm Once a hole is stored in a concavity tree, it can be regarded as an object with no holes. By matching two concavity trees that represent two holes in two different images, the distance between the concavity trees depends on the Deletion, Substitution, and Insertion costs that are variable and depend on the information stored in the trees. Therefore, when comparing two holes, different attributes of the holes including the area, concavity, and shape are considered in the comparison process and are reflected by the costs of the Tree Edit Distance. However, there is one attribute of the holes that is not accounted for in the comparison which is the location of the hole in the object. This problem wasn't present when we compared two images with no holes because then our primary concern was the similarity between the two shapes, and the location of the shape in the bitmap image wasn't of much interest. With images that contain holes, the location of the hole in the image is very meaningful and ignoring this location in the comparison process can lead to inaccurate results. To see how the location of the hole can affect the comparison of two bi-level images with holes, consider the two images shown in Figure IV.4.1. For simplicity, we assume that the two images don't include any concavities. The two images consist of two objects each with a hole. The objects are two ellipses with exactly the same size and shape, and the holes are two circles with the same area. The only difference between the two images is the location of the holes. This location implies a difference in the two figures and therefore they are not exactly similar. Page 28 of 76

29 We will see whether applying our algorithm to compare the two images will account for this difference. Figure IV.4.1: Two Identical Images With Different Hole Positions First, the algorithm stores the two ellipses and the two circles in four concavity trees. Next, the two concavity trees of the two objects are compared using the Tree Edit Distance and the result is a distance of 0 between the two shapes. Next, since each image has only one hole, the two concavity trees representing the holes are compared using the Tree Edit Distance and will also yield a distance of 0 since the two circles are of equal area. As a result, the algorithm will result in a distance of 0 between the two images implying that the two images are exactly similar. This result is partially wrong and does not account for the difference in location of the holes between the two images. Therefore, we had to add something to our algorithm to solve this problem and get a correct comparison between any two images irrespective of the location and number of holes. The best solution to this problem is the use of dissecting lines to cut the bi-level images, then use the dissected parts of each image in the comparison to get a clear idea about the location of holes. The dissecting line is chosen in a random direction, and it cuts the image with holes transforming it into two images. We start first by selecting a line, then dissecting the two images to obtain four new images. Next, the right portion of the first image is compared with the right portion of the second image, and the same is done for left portions. This comparison is sufficient to account for the placement of holes in the original image. To see the effectiveness of the dissecting lines, consider the previous figure. By applying the dissecting line algorithm to the two figures, we select a random direction and cut the figures along this direction. The result of the process is shown in Figure IV.4.2. The line transforms the two images into four. By comparing the left portion of the first image with the first portion of the second, and doing the same for the right portions, the difference between the two images is revealed and a certain cost is added, therefore the distance between the two images is no more equal to 0, and the two images are not exactly similar. Page 29 of 76

30 Figure IV.4.2. is shown below. Figure IV.4.2: The Dissecting Line Procedure The dissecting lines can also be horizontal, in which case the upper and lower portions are compared, as shown in Figure IV.4.3. The comparison in this case yields the same results as before and uncovers the difference in the location of the holes. We can see that in this case, one dissecting line is sufficient to account for the location of the holes in the two images. Figure IV.4.3: Horizontal Line Dissection Usually, any number of lines can be used to compare two figures, and it might be possible to figure out the similarity between two figures using only one dissecting line. However, since our algorithm deals with any number of holes with as many concavities, we chose to include an arbitrary number of random lines, which means that the process of dissecting the images and comparing them will be done an arbitrary number of times. By choosing the dissecting lines randomly, the lines will be taken in all possible directions, and by selecting a sufficient number of lines the location of all holes will be identified and thus the matching of bi-level images is further improved. The implementation of dissecting lines in Matlab is discussed later. Page 30 of 76

31 IV.5. Datasets of Images The modified TED algorithm has the ability to compare any 2 bi-level black and white images with holes. We decided to use two different datasets of images to test our algorithm with. These datasets are the standard MPEG7 dataset, and a synthetic dataset that we made. IV.5.1 MPEG7 Dataset MPEG7 is a big set of images that is recognized for testing and matching purposes. The images provided by MPEG7 are bi-level images with no holes. These images are realistic and they depict actual shapes, and they are also complicated since they contain a large number of concavities and meta-concavities which results in a big number of nodes in the concavity tree representation. A sample of the original MPEG7 is shown in Figure IV.5.1. Figure IV.5.1: Two classes from the original MPEG7 dataset. Page 31 of 76

32 The MPEG7 dataset is used to test the original TED algorithm that only supports images with no holes. However, our modified algorithm compares images with holes, so we had to modify the MPEG7 images to incorporate holes into them before using these images to test our algorithm. We used a modified subset of the MPEG7 dataset composed of 32 images divided into 8 classes of 4 images each. Figure IV.5.2 shows the 8 classes of images with holes. Figure IV.5.2: Classes of images of the modified MPEG7 dataset. We used these images to test the effectiveness of our algorithm on real shapes since such shapes are at the same time meaningful and suitable to the algorithm. IV.5.2 Synthetic Dataset In addition to the MPEG7 dataset, we made our own dataset to test the modified TED algorithm with. We designed this dataset to include multiple forms that emphasize rotation, cropping and flipping of a certain shape in order to test our algorithm against such variations. Page 32 of 76

33 The synthetic dataset contains random bi-level shapes with a number of holes ranging from 2 to 4. We created 8 classes of images where each class is composed of 8 images. Figure IV.5.3 shows the 8 classes of the synthetic dataset. Figure IV.5.3: Classes of images of the Synthetic dataset. We used the images in the Synthetic dataset to test the effectiveness of our algorithm when comparing two bi-level images with holes. Therefore, we had to use different combinations for the external shapes with the inside holes. Such a variation would be varying the location and (or) shape of holes while keeping the external image unchanged, or changing the external shape while keeping the same holes. Figure IV.5.4 shows two objects that have the same shape and number of holes, but the holes have different shapes. When comparing these images using our algorithm, the difference between the two objects is revealed by the difference between the holes while the two shapes are similar, and so the distance between the two objects is determined according to the location and shape of holes. Page 33 of 76

34 Figure IV.5.4. is shown below. Figure IV.5.4: Two objects with the same shape but with different holes. On the other hand, Figure IV.5.5 shows two objects with the same holes but with different shapes. The distance between these two objects is determined only by their external shapes since the comparison of the holes yields no difference between the two images. Figure IV.5.5: Two objects with the same holes but with different shapes. Another example is to use two objects with different holes and shapes, as shown in Figure IV.5.6. In this case the distance between the two images is reflected by the difference between the holes and the external shapes of the two objects. Figure IV.5.6: Two objects with different shapes and holes. We used images from this dataset to test the robustness of our algorithm against variations in the number, size and location of holes between two objects, as well as changes in the shapes of the objects themselves. Page 34 of 76

35 Chapter V. Implementation and Testing V.1. Implementation V.1.1. Converting for-loops to C++ Via MEX Files We started by locating all the time consuming for-loops in the Matlab code and transforming them into C++ MEX files, then called them from the original Matlab functions. Among the functions that contained the largest number of time consuming for-loops was the Dijkstra function. Thus it was the first function on our list of functions to be converted to MEX files. The effort of rewriting the Dijkstra function as a MEX was worth it, since upon testing the MEX equivalent of the Dijkstra function for large matrices we achieved a very high improvement in speed. As a result we were able to drastically increase the speed of the overall algorithm. We then located several other time consuming for-loops and transformed them to MEX files. During the Spring semester we were able to complete the task, we started during the Fall semester, of transforming all the for loops into equivalent MEX files. After doing so and incorporating the MEX files into the Matlab functions we observed a large increase in performance that resulted from the remarkable decrease in running time of the algorithms. More details on the improvement in performance due to the se of MEX files will be presented in the Testing section. V.1.2. Implementation of Hole Representation We upgraded the Concavity Tree Structure and Tree Edit Distance algorithm to include representation and matching of bi-level images with holes. The first step towards this goal was the modification of the fct function to locate holes and add them to the concavity tree data structure of the image. Upon doing this the modified function is now able to represent any bi-level image with or without holes using a modified version of the concavity tree data structure. Figure V.1.1. shows a bi-level image containing several holes, and Figure V.1.2 shows the image constructed using the modified version of the fct function, fcth, with all the convex hulls superimposed on top of the reconstructed image. Page 35 of 76

36 The original picture versus the reconstructed one are shown below. Figure V.1.1: Original Bi-level Image with Holes Figure V.1.2: Constructed from Original The function fcthplot is used to plot the concavity tree of the resulting image (Figure V.1.3.). Figure V.1.3: Concavity Tree of the Original Image Page 36 of 76

37 The dark blue nodes correspond to the concavities and meta-concavities of the contour of the actual image, while the red, green, and light blue nodes correspond to the concavities and meta-concavities of the contour of the holes inside the image. This is illustrated in Figure V.1.4., where we show the correspondence between the holes and their representation in the tree. From a representation point of view, the holes are stored as disjoint concavity trees in a cell array. In other words the high level data structure of the tree is a cell array of cell arrays where each refers to one of the needed trees. Figure V.1.4: Correspondence between Holes and Tree Page 37 of 76

38 With the introduction of holes, we can represent 3-D shapes and introduce more clarity to the image. Introducing holes requires the preliminary steps illustrated by the pseudocode: Algorithm 0: Extract Concavity Tree with Holes I = Read Image ( Image.bmp ) function CH = CTH { I } 1. H = Locate Holes (I) 2. T = Extract Concavity Tree (I, Parameter) 3. R = Represent Holes (H) 4. F = Field = Null 5. for i = 1: Number of Holes 6. E[i] = Extract Info {H[i]} 7. R[i] = R[i]+E[i] 8. T = T + R [i] 9. F = Update {F} 10. end 11. T[Root] = T[Root]+F 12. Return T Note that the parameter in the previous pseudocode is used to determine the level of compression-retrieval of the concavity tree algorithm. In order to implement the above algorithm in Matlab we wrote the following functions: A. Function fcth This function computes the concavity tree of the entire image (with the holes included). The function heading of the fcth function is shown below, along with a brief explanation of the various input and output arguements: function [cct, mpoly] = fcth( mimg, Param, varargin ) where 1. Before calling the fcth function one must first read a bitmap image into a matrix of 0s and 1s where the 0s represent the background pixels and the 1s represent the foreground pixels. This matrix is stored in the mimg input variable,and can be created using the following statement: mimg = not(imread('image.bmp')); Page 38 of 76

39 2. Param is the parameter that contains information about the holes to be transferred to the concavity tree editing. 3. cct is a concavity tree representing the holes and the contour of the image in a colored tree. 4. mpoly is an array representing the contour points of the initial root. Many fields were added to describe images with holes: One for the existence of the Holes to be detected One for the differentiation in color One for the number of children in holes B. Function fcthplot The fcthplot function plots the Concavity Tree. The syntax of the function is as follows: function fcthplot(x) where x = not(imread('image.bmp')); The nodes of each hole will be assigned their own color, while the nodes of the outer shape will be assigned a deep blue color. V.1.3. Implementation of Matching Images with Holes With the new data structure, the matching algorithm using Tree Edit Distance will change. Two alternatives are present. The first one uses relative comparison to find the minimum distance between the two images. The second one uses lines intersecting both images at random in order to find the metric. Both algorithms will be described below. Page 39 of 76

40 The Relative Comparison Algorithm The Relative Comparison Algorithm can be described by the pseudocode shown below. Algorithm 1: Matching CTs using Relative Comparison Algorithm Notation: T1 and T2 represent concavity trees of images with holes. T1 = CTH { I1 } T2 = CTH { I2 } Dummy = Dummy Hole Node Weight returns a [0 1] scaled weight of the hole depending on Relative Area of the Hole s Concavity Tree function Distance = TEDH1 { T1, T2 } 1. Distance = 0 2. Number Holes Max = Max {Number Holes {T1}, Number Holes{T2}} 3. for i=1 : Number Holes Max 4. if i < Number Holes Max 5. Distance = Distance + Weight(Hole [i]) x TED {T1[i],T2[i]} 6. else 7. Distance = Distance + Weight(Hole [i] ) x TED {T1[i],Dummy} 8. end 9. Normalize Distance function Distance = TED { T1,T2 } T1 and T2 represent concavity trees of images with holes. T1 = (V, E1) T2 = (W, E2) T1 is the tree with the minimum number of root children. V1 and V2 are two vectors. 1. Fix T1 and rearrange T2 in the second level to have greater similarity. 2. V1 = Pre-order Traversal (T1) 3. V2 = Pre-order Traversal (T2) 4. for i = 1: size of V1 5. for j =1: size of V2 6. Construct Edit Graph Edges 7. end 8. end 9. Transform the edit graph to a directed connected graph. 10. Give weight for each edge according to operation (del, ins, sub). 11. Use Dijkstra to find shortest path between node V 1 W 1 & V size_of_v1 W size_of_v Distance = Weight of the Shortest Path. Page 40 of 76

41 The algorithm works as follows: 1. Matching the Root Sub-trees The first root node and its sub-tree are matched to the second root and its sub-tree. 2. Matching the Holes F igures with Same Number of Hole For figures with an equal number of holes there exist 2 Alternatives: Alternative 1: Compare the holes one by one at random. Alternative 2: Rearrange the holes to make the lowest cost match. The complexity of Alternative 2 exceeds that of Alternative 1, while the worst case complexity is given by: O ( Alternative.1 ) x NumberOfHoles Figures With Different Number of Holes When the number of holesof the figures being compared are not equal, we compare the first Alpha holes of the 2 images together by applying the procedure described above, where Alpha = Min(H1, H2), H1 = number of holes in the first image, H2 = number of holes in the second image. We then compare the remaining Beta holes, of the image with the larger number of holes, with dummy holes. As mentioned previously, a dummy hole is defined as a dummy hole is a special concavity tree made of a unique node. Note that Beta is defined as: Beta = H1 H 2 Random Lines Intersection Algorithm In order to implement the Dissecting Line idea programmatically, we wrote a function that cuts the matrix,representing the image, by a number of horizontal, vertical, or positively sloped random lines. The number of random lines is specified by an input parameter. For each slice we compare the right portion of the first image to that of the second image, and the left portion of the first image to that of the second image(in case of a horizontal cut we compare the upper slices of each image together, and do the same for the lower slices). To administer the comparison we use Page 41 of 76

42 the standard fted to compare the corresponding portions of each image, after finding the concavity tree representation for each of the 4 resulting pseudo-images. We add the costs of each trial of the random lines and normalize it to obtain a cost of transforming the first image with holes into the second image. Below is the pseudocode of the Random Line Intersection algorithm: Algorithm 2: Matching CTs using Random Lines Intersection Notation: T1 and T2 represent concavity trees of images with holes. I1 = Read Image ( Image1.bmp ) I2 = Read Image ( Image2.bmp ) T1 = CTH { I1 } T2 = CTH { I2 } function Distance = TEDH2 { I1, I2, Parameter } 1. Distance = 0 2. HI1 = Dissect (I1, Parameter Lines) 3. HI2 = Dissect (I2, Parameter Lines) 4. for i = 1: Parameter 5. Distance = Distance + TED { HI1[i], HI2[i] } 6. end 7. Normalize Distance according to Parameter 8. Return Distance V.1.4. Software Implementation We dedicated a large portion of the Spring semester to creating a software that would provide both a user friendly environment as well as powerful functionality by making use of the fct and fted algorithms we developed and upgraded during this semester and the preceding one. We chose the name Proximity for our software for 2 main reasons, namely: it is relevant to the topic of our FYP (i.e. matching bi-level images by finding the proximity between them), and it is short and easy to remember. We used Matlab as the programming platform for our Software and all the related functions. Page 42 of 76

43 Below (Figure V.1.4.1) is a screen-shot of the Proximity software s GUI (graphical user interface). Figure V.1.4.1: The Proximity Software GUI To run Proximity, it is necessary that you have Matlab 7 installed on your computer. You must first run Matlab, add a path to the directory containing the Proximity software and fcth and fted functions, then type proximity in the command prompt window. Upon following the previous steps correctly the Proximity software is loaded to your Matlab work space and the GUI appears on your screen. The GUI is composed of 4 main sections: The Source image section, the Target image section, the Options section, and the Results section. The Source image section is used to load bi-level images from any directory on your PC to List-A, extract their Concavity Trees and Convex Hulls, and view them in 3 different representations: the original bi-level image representation, the Concavity Tree representation, and the Convex Hull representation. In addition to this the user can also view various details about the loaded image and its Concavity Tree, such as the fcth extraction time, the number of holes in the image, the number of nodes in the Concavity Tree, the SCX attributes of each node, the depth of the Tree, the relative area of each node, etc The Source image list contains the images that will be matched against the images in the Target image list. That is, for each entry in the Source image list there Page 43 of 76

44 exists a separate list of Target images in the Target image list against which the corresponding image in the Source list will be matched. Referring to Figure V we can see that the Source image list contains 2 images (z2 and a10), each of which corresponds to a separate list of Target images. In the above figure we can see that image a10 in the Source image list will be matched against images b8, a6, and sample in z2 s Target image list. The Target image section has the same function as the Source image section, except that it deals with the Target images instead of the Source images. The Options section is used to allow the advanced user to specify certain parameters that will be used by the fcth function to extract the Convex Hull of the image, and which will in turn affect the construction of the Concavity Tree of the corresponding image. Also the user can select the type of matching method he/she would like to use, i.e. either regular hole-to-hole matching or the more advanced (and more accurate) random line matching method. The final section is the Results section. This section displays the final matching parameters that result from matching the selected Source image against the list of corresponding Target images. Upon selecting a Source-Target image pair and clicking on the Details button in the Results panel, a user can view the details of the matching process of the 2 selected images, such as the overall matching time, the matching time of each hole pair, the matching parameter of each hole-pair, etc For a more complete description of the Proximity Software please refer to Appendix A. V.2. Testing V.2.1. Testing the Algorithms for Image Matching In order to test the databases, our testing platform used the function ftestimages. The latter s function is to find the closest n images to the original figure from a set of m images, where m > n, with syntax: function [mtestresults, IndexSequence, ImageCloser]= ftestimages (Group Number, ImageNumber, Param) where 1. Param = n 2. Restraining comparison to a given set of images is possible by defining GroupNumber and ImageNumber. 3. The Distance between the image and each of the images is stored in mtestresults. Page 44 of 76

45 4. The Index Sequence contains the indexes of the n closest images to the given one. 5. ImageCloser contains the names of the.bmp files closest to the image. We tested each of the two matching algorithms on the two datasets. As we have seen, the first one is an 8x4 modified subset of the MPEG7 as to contain holes whereas the second one is a synthetic dataset of 64 images distributed over 8 classes of 8 images each. Testing the Relative Tree Comparison Algorithm on the 8x8 Synthetic Dataset yielded the following results: Class Success Rate (%) A B C D E F G H Table V.2.1. Success Rate for each class of the 64 synthetic dataset using the Relative Tree Comparison Algorithm. Hence, on the whole dataset, the algorithm achieved a success rate of %. As an example, the success rate of class A can be computed in the following way: Class F Success Rate (%) F F F F F F F F Class Average Table V.2.2. Success Rate for class F using each of the images belonging to Class F. Note that a rate of 100% means that the closest 8 images to the first image in the database are the ones belonging to its class. By the same logic, a rate of 87.5% means that one error occurred when searching for the eight closest images among the database. Page 45 of 76

46 As for the Random Comparison Algorithm, the results were the following: Class Success Rate (%) A B C D E F G H Table V.2.3.: Success Rate for each class of the 64 synthetic dataset using the Random Lines Intersection Algorithm. Hence, on the whole dataset, the algorithm achieved a success rate of %. As an example, upon comparing image a1 with the rest of the database, the obtained results are found in the following: Image Compared to a1 a a a a a a a a Class Average Table V.2.4.: Distance of all images in Class A to image a1. Contrasting with random images from other classes, we get the following. Image Compared to a1 a b c d e f g h Random Comparison Average Table V.2.5.: Distance of random images from all classes to image a1. Page 46 of 76

47 Hence we remark how far the figures in the above table are when compared to the previous one. This means that our algorithm is accounting for the difference among the classes by adequate changes in the distance between the images. The following plot shows the range of the distance for the images of class A Distance of Class A Images to a1 Distance Fitting Curve Distance to Image a Image Number in Class A Figure V.2.1.: Plot of the distance of all images in Class A to image a1. The plot corresponding to the second table can be seen below Distance of Random Class Images to a1 Distance to a1 Fitting Curve Figure V.2.2.: Plot of the Distance of random images from all classes to image a1. Page 47 of 76

48 In the following histogram, we can see the closest ten images in the synthetic database to image a1. Figure V.2.3.: For Histogram Class E however, showing the we ten have closest the images following: the database to image a1. Figure V.2.4: Histogram showing the ten closest images in the database to image e1. Page 48 of 76

49 Note that, although the images e2 and e5 are not among the closest eight to image d1, they figure among the closest ten to this image. Repeating exactly the same steps as outlined in the previous section on the MPEG7 subset, we obtain matching rates of 80.14% for the Relative Matching Algorithm and 74.15% for the Random lines Intersection Algorithm. V.2.2. Results Analysis After analyzing the obtained, we can deduce the following: The matching rates for the MPEG7 subset and the synthetic dataset were relatively close. The difference between them did not exceed 3%. The smallest success rate for a class in the database is of % for the Relative Comparison Algorithm and % for the Random Lines Intersectin Algorithm. The Relative Comparison Algorithm performed better on the whole dataset than the Random Intersection Algorithm. The Random Intersection Algorithm peformed better only on Class H. This is expected because the overall outer shape does not vary substantially among the images of this class, which is taken into account by the shape alignment embedded in the Random Intersection Algorithm. Class E exhibits the worst result for the Relative Comparison Algorithm. This is because the images of class E are characterized by a large variation of rotation, changes in outer shape and holes shape, in addition to the complexity of their concavities. The resuts of the Relative Comparison algorithm can be improved by optimizing the Weight function. V.2.3. Testing the Running Time Upon incorporating MEX Files into the algorithms, the running time was reduced by a factor of 46 % (i.e. almost half). However we note that the Random Lines Intersection Algorithm is more time consuming than the Relative Comparison Algorithm. This is because the former algorithm s complexity depends on the number of lines dissecting the images. Page 49 of 76

50 Running Time for fcth: Let n = Number of Contour Pixels () 1 Let α = O Number of Holes Let β = User Input for the Rand. Lines. Int. Al. Let d = Number of Nodes Let h = Height of the Concavity Tree Since the Running Time of the Concavity Tree Algorithm is Onh ( ), and sinceα = O(1), we can write the running time of the fcth function as: Tn ( ) n.. Onh (. ) O(. hn. ) On ( 2 2 = α = α = ) In the worst case we have n h = where k > 4 for the smallest concavity to be considered as node k In this case, the running time is: 2 α 3 Tn ( ) = O( α. hn. ) = O( n) k h n For other cases, h varies according to k.2 n h log2 k In this case, the running time becomes: 2 d 2 2 T1( n) = O( α. hd. ) = O α.log 2. d = O( n log ( n) ) k 2 Running Time for ftedh1: Here the Tree Edit Distance process is executed for α runs, that is why the running time is expressed as: 2 ( { }) { ( )} ( 2 ) 2 ( ) ( ) T ( n) = α. T = α. T E+ V log V = α. O d log d Dijkastra n T 2 2 2( n) = O log 2 d log d = O 3. d.log n k Page 50 of 76

51 Running Time for ftedh2: The running time of the Random Lines Intersection Algorithm depends on the value of the user input β. It can be expressed as: ( β 2 { ( )}) ( β 2 ) T ( n) = O. d log d = O. d 3 Running Time for the Application: As a result, upon running the application in order to represent and match two images, the running time will be expressed as follows: ( ) ( ) 2 = T n T n ( ) ( ) T n for Comparison T n for Random Lines 2 3. d.log n for Comparison T n O n 2 n O n 2 2 n β. d for Random Lines ( ) ( ) = α. log ( ) + = α. log ( ) As can be seen, the overall complexity is polynomial in time with around time. However, the running time of the matching algorithms is much less than 2 O( n ) this since they depend on the number of nodes in a tree which is considerably less than that of the contour pixels. Page 51 of 76

52 Chapter VI. Completed Work During the course of our 2 semester FYP we were able to achieve most of the goals we had set for our selves during the first weeks of the Fall semester. The goals that we did achieve are listed below: During Fall Semester Conducted research on the subject of concavity tree matching, and read a number of papers on the subject Researched the subject of MEX files on the internet, and learnt how to write MEX files in C++, and call them from Matlab. Transformed a large part of the for loops written in Matlab into equivalent C++ mex files, and modified the original Matlab functions accordingly. In particular we were able to improvethe complexity of the fcth, ftednew, and Dijkstra s mat functions by translating large portion of them into C++ MEX functions. Expanded the scope of representing objects with holes and implemented hole detection, extraction, and representation in Matlab. Wrote Matlab code to implement matching of bi-level images containing holes using 2 algorithms: Random line dissection Approach Relative Comparison Approach using: o Dummy Holes o Holes Rearrangements Researched the possibility of comparison of color images and came up with several interesting results that might allow us to also take advantage of color information present in Colored Images, to complement the use of shape information in image comparison. Page 52 of 76

53 During Spring Semester Completed the transformation of all the for-loops into equivalent MEX files, and incorporated them into the fcth and fted mat functions. Completed the Random line matching algorithm for images with holes and tested it on a portion of the MPEG7 dataset, as well as on a Synthetic dataset we created for objects with holes testing. Improved the weights of the fted matching algorithm to obtain more accurate results. Tested the upgraded fcth and ftedhole functions on a portion of the MPEG7 dataset, as well as on a Synthetic dataset we created for objects with holes testing. We obtained exciting results and significant improvements in accuracy, speed, and overall performance of the compression and matching algorithms. Test results were presented in detail in the Testing section of chapter V. Developed software with a well designed GUI that implemented the fcth and ftedhole functions for compression and matching of bi-level images with holes. The FYP Final Report, Poster and Presentation. Page 53 of 76

54 Chapter VII. Budget, Timeline & Tasks VII.1. Budget Actually our FYP doesn t require any financial support or funding. It just requires determination to explore the research potential embedded in this field along with adequate referencing. The only tool we needed to purchase was a new version of Matlab (Version 7.0.1). VII.2. Distribution of Tasks After selecting the project and identifying the tasks that we have to do, we divided the tasks among the 3 of us, and then we agreed on a plan to follow in our work. In order to achieve the largest number of goals and comply with the time limits, we had to carefully divide the various tasks amongst ourselves in such a manner that would allow all of us to work in parallel on the different parts of the project, so as to be able to make the most out of our alliance. We had regular meetings to discuss the progress, and problems encountered by each group member, and to come up with solutions to those problems. We all maintained close contact with our FYP supervisor, and also had regular meetings with him to keep him up to date with our progress and to discuss with him possible solutions to the difficulties facing us or ask his opinion on certain issues. Throughout the semester all the members of our group maintained a high level of academic integrity and discipline. Our work was divided between the fall and spring semester: during the fall semester, we researched various ideas to improve the TED algorithm, then we started implementing those ideas and we performed some tests to verify the performance of the algorithm. During the spring semester, we finished the implementation of the TED algorithm with holes, and we developed a Graphical User Interface that compares images based on the modified TED algorithm. Finally, we tested our algorithm on many datasets and obtained results. At the beginning of the fall semester, we had to do a literature review in order to familiarize ourselves with the Concavity Tree Representation of bi-level images, and the use of Tree Edit Distance to compare two images. After reading a number of published papers related to this topic, we researched many ideas for improving the Page 54 of 76

55 algorithm. We researched the possibility of modifying the present algorithm in order to improve its running time and matching accuracy, and we searched for ways to expand the algorithm in order to support further types of images. After analyzing the various options, we decided on the following: 1. To decrease the running time of the algorithm, the best solution is to convert the time-consuming for-loops from Matlab to C++ using MEX-files. 2. To increase the range of images that the algorithm supports, we decided to expand the algorithm in order to support figures with holes. After we agreed on these improvements, we divided the work and began working on the improvements. During the design phase, we derived ways for adding holes to the bi-level images and then we included the dissecting lines method to improve the matching accuracy of the algorithm. Additionally, we learned how to write MEX-files in C++ and how to interface Matlab with C++. Next, we started to implement the above methods in our algorithm. We implemented the representation and matching of binary images containing holes in Matlab by adding new functions, modifying some of the older ones, and converting most of the for-loops to C++ then calling them in Matlab code (so as to increase speed). At the end of the fall semester, we analyzed the results of our work by comparing the speed of our modified algorithm with the original one to account for the use of MEX-files, and we tested the algorithm using bi-level images that contain holes. At the beginning of the spring term, we converted the rest of the for-loops in the TED functions to MEX files in order to further improve the speed of the algorithm. Next, we finalized the implementation of the TED algorithm to support holes by implementing two different methods: the first uses relative comparison between the holes of two images to compare them, and the second uses the random lines dissection algorithm. We then developed a Graphical User Interface that uses both methods of the modified TED algorithm to compare any number of images. Page 55 of 76

56 Finally, we tested the speed of the finalized algorithm, and we also tested the operation of the Graphical User Interface. We then tested the functionality of the algorithm by running it on images from the MPEG7 dataset and a synthetic dataset that we made, and we ended our work by analyzing the results of the comparison process. VII.3. Timeline The Gantt Chart shown in Figure VII.3.1. shows the timetable that we followed in the fall semester. The diagram shows the list of tasks that we accomplished, the time for each task, the distribution of tasks among group members, and the tasks that we had in mind for the Spring Semester. Figure VII.3.1: Gantt Chart showing tasks allocation and timeline for Fall Page 56 of 76

57 The Gantt Chart shown in Figure VII.3.1shows the timetable that we followed in the spring semester visualizing timing, duration and dependencies between tasks. Figure VII.3.2: Gantt Chart showing tasks allocation and timeline for Fall In the next chapter, we conclude our overview of the FYP by focusing on the overall course of the project, along with what we fulfilled, conducted, or researched. Page 57 of 76

58 Chapter VIII. Design Constraints Our project mainly consists of developing a software application using Matlab, so we had few design constraints to deal with. When designing our project, we didn t face any environmental problems since our project has no effect on the environment as it is a software project, and we didn t consider any health and safety constraints since our project doesn t affect the lives of others. We also didn t have any political constraints since the goal of our work is to develop an algorithm that compares bi-level images. This idea is accepted and researched worldwide and it has a lot of useful applications and can be used by many entities including the government itself. We considered manufacturability issues when we started implementing the TED algorithm. We used Matlab to write the code for the algorithm and develop the Graphical User Interface, and C++ to write MEX files. Therefore, we all had to install the latest version of Matlab and C++ on our computers in order to be able to work on any part of the project and at any time. The only economic constraint we faced was during testing since we used two large datasets of images to test our algorithm with, and we had to compare every image in each dataset with all other images from the same dataset. By using only one computer, this would have taken a lot of time. We solved this problem by running the algorithm simultaneously on many computers including our laptop computers and many others that we operated in the computer labs. This saved a lot of time before obtaining the results of the simulation. We addressed the sustainability of the project when we had to choose between the latest version of Matlab which the version or an older version which is Matlab The latest version offers many additional features that we can use, but the problem is that once we write our code with this version, we will not be able to run it on older versions. We finally decided to use Matlab to benefit from its features, but this made our program only compatible with Matlab This added another limitation for the program: in order for it to run faster, it should be used on a computer with good specifications. A suitable computer is one with the following specifications: PENTIUM M, 2.00 GHz, and 512 MB of RAM. From a social point of view, we communicated our FYP well by attending meetings regularly and respecting the timeline. We also worked as a team by dividing the workload equally among the three of us, and helping each other when necessary. Ethically speaking, it is good to mention that all group members acted responsibly and ethically along the course of the FYP. Page 58 of 76

59 Chapter IX. Conclusion and Further Research During the course of our 2 semester long FYP we were able to achieve the majority of the goals we set for ourselves at the beginning of the Fall term. We were able to speed up the algorithm by translating large portions of the Matlab code into C++ MEX files. We were also able to devise a hole representation and matching algorithm which truly expanded the range of images we could compare. We were able to implement the hole matching algorithm in Matlab and test its accuracy and speed for a large portion of the MPEG7 dataset and a Synthetic dataset (with images containing holes). We also came up with the random line dissection algorithm, which further enhanced the hole matching algorithm by accounting for the relative position of the holes within the image. We researched color image matching, and formulated a method for its integration into our existing algorithm. Finally we developed an advanced GUI for the fcth and ftedhole algorithms. We were able to achieve almost all of our original goals during the course of the FYP, and this was a large achievement indeed given the difficulty and researchorientation of a large part of our goals. Due to careful planning and task division we were able to accomplish much more in less time and with smaller effort. Despite the numerous difficulties we faced throughout the life of our FYP we were able to overcome all the obstacles due to well organized team work and head-strong determination. A large portion of our work during our FYP comprise new concepts that have not yet been researched by others (such as the concepts of representation and matching of images with holes), and the very nature of our FYP was both research-oriented and involved a lot of practical issues that we had to address. Since Concavity tree representation and matching are relatively new concepts, we were faced with many challenges and difficulties mainly due to the inadequate research that has been done so far on the subject, and the scarcity of relevant literature on the topic of concavity trees. Nonetheless we were able to accomplish our goals successfully and provide enhanced versions of the compression and matching algorithms, as well as upgrade the algorithms to enable matching of images with holes. We feel that the FYP was a rewarding experience for all of us, and we benefited from it on both an academic and personal level. Academically we benefited by learning how to conduct research, we sharpened our programming skills, we brain-stormed and devised solutions for difficult problems, and learned how to write professional reports. On the personal level we benefited by improving our communication skills, learning to work better as a team, improving our ability to work well under pressure and within time and budget constraints, and bettering our presentation skills. In conclusion our FYP was a wonderful and rewarding experience. Page 59 of 76

60 Appendix A. Proximity Software User Manual This appendix will provide a detailed description of the Proximity software that we developed during the second part of our FYP. A.1. General Description The Proximity software was written entirely using Matlab and its GUI was designed using Matlab s GUIDE tool. The name Proximity was chosen for its relevance to the topic of our FYP (i.e. bi-level image matching) and for its simplicity and brevity. The Proximity software requires Matlab 7 (or higher) in order for it to run. Its main function is to extract the Concavity Tree representation of a bi-level image (i.e. black and white bit-mapped image), and match it against another bi-level image, so as to provide the user with a matching parameter which indicates how similar the 2 images are. The Proximity software relies heavily on our upgraded fcth and ftedhole functions so as to be able to extract the Concavity trees and perform the TED matching for b-level images containing holes. In what follows we will describe how to use the Proximity software to match bi-level images with/without holes and extract a large variety of useful information from both the image acquisition and matching processes. A.2. GUI To run Proximity on your PC, you must first be running Matlab 7. You must also add a path to the directory containing the Proximity package. Next, type proximity at the Matlab command prompt in order to load Proximity. When Proximity has loaded you should see the GUI shown in Figure A.2.1. The GUI is divided into 4 main sections: - The Source Image Section - The Target Image Section - The Options Section - The Results Section Page 60 of 76

61 We will now describe each of the above 4 sections. A.2.1. The Source Image Section Figure A.2.1: The Proximity Software GUI Referring to Figure A.2.1 we can see that the Source Image section is composed of the Source Image panel, the left-most Preview panel, and the information list situated directly below the Preview panel. This section contains the controls used to load, delete, and view the selected source images from the list box (List-A). A source image is an image located in List-A against which a group of corresponding target images in List-B will be matched. The Source Image panel is composed of a list-box, 4 buttons, and 3 radio-boxes. We will describe the function of each component below. The List-A list-box: This list-box will contain the names (and extension) of the loaded source images. The Load Image button: This button is used to load a bi-level bit-map image from a selected directory on the user s PC, into List-A, and execute the fcth and fcthplot functions to extract the concavity tree and convex hull of the corresponding image. Page 61 of 76

62 Figure A shows how a source image is loaded into List-A. Figure A.2.1.1: Source image loading process The Delete Selection button: This button deletes the selected Source image in List-A from the list, and clears the corresponding target images from List-B. The Clear List button: This button clears the Source image list (List-A) and clears all the Target image lists (List-B) of each source image. This button also clears the Results list in the Results panel (if any exists at all). The Details button: This button displays (inside List-A) image acquisition details about the selected source image. Acquisition details include the image name and location, the fcth options used in the construction of the concavity tree, and the concavity tree construction times of the overall image, and each individual hole. When it is first pressed its name changes to Back and it displays the required image details in List-A, then when it is pressed again its name changes back to Details and the original List-A source image entries are restored. See Figure A Page 62 of 76

63 Figure A is shown below. Figure A.2.1.2: Displaying image acquisition details The image radio-button: When this button is on the original bi-level image of the selected source image in List-A is displayed in the left-most Preview Panel. See Figure A Figure A.2.1.3: Preview of the original image Page 63 of 76

64 The Concavity Tree radio-button: When this button is on the Concavity Tree of the selected source image in List-A is displayed in the left-most Preview Panel (A ). Figure A.2.1.5: Preview of the Concavity Tree of the image The Convex Hull radio-button: When this button is on the Convex Hull of the selected source image in List-A is displayed in the left-most Preview Panel. See Figure A Figure A : Preview of the Convex Hull of the image Page 64 of 76

65 The Preview Panel, as discussed before, is used to display the original image, Concavity Tree, or Convex Hull of the selected source image. The information panel is active only when the Concavity Tree radio-button is on. When a either the image root, a hole root, or a leaf of the Concavity tree is clicked, information of the particular node is displayed in the information panel. The information displayed includes the node s level, color, number of children, hole number, SCX parameters, and number of leaves. See Figure A Figure A.2.1.7: The information panel A.2.2. The Target Image Section Referring to Figure A.2.1 we can see that the Source Image section is composed of the Source Image panel, the left-most Preview panel, and the information list situated directly below the Preview panel. This section is identical to the Source Image Section where the only difference is that the images manipulated using the various functions of this section are the target images of the selected source image. Page 65 of 76

66 A.2.3 The Options Section Referring to Figure A.2.1 notice the Display options check-box. When this check-box is unchecked the Options Panel is invisible. To make the Options Panel Visible select the Display options check-box (see Figure A.2.3.1). The Options Panel contains input text boxes and drop-down buttons that allow the advanced user to configure the input parameters of the fcth functions, and consequently modify how the concavity tree of the image is constructed. The Options Panel is composed of the following input boxes: Figure A.2.3.1: The Options Panel The Minimum Concavity Depth input box: This box allows the user to specify the minimum Concavity Depth of the concavity tree that is to be constructed. It is by default equal to 1. The Minimum Concavity Area input box: This box allows the user to specify the minimum Concavity Area of the concavity tree that is to be constructed. It is by default equal to 0. Page 66 of 76

67 The Minimum Concavity Length input box: This box allows the user to specify the minimum Concavity Length of the concavity tree that is to be constructed. It is by default equal to 0. The Functionality drop-down box: This box allows the user to select function to be performed by the fcth algorithm. The user can select among 4 different functions, namely: representation, compression, decomposition, and envelope. The Matching Method drop-down box: This box allows the user to specify the matching method that should be used to match the selected source image to its corresponding target image list. There are 2 available methods from which a user can choose, namely: relative TED matching, or random line matching. The relative TED matching method was described in Chapter V section V.1.3, while the random line matching method was described in section V.1.4. A.2.4. The Results Section Referring to Figure A.2.1 we can see that the Results section is composed of the Results panel. This section contains the controls used to match the source images to their corresponding target images, and view the matching details for each sourcetarget pair. Figure A.2.4.1: The Results Panel Page 67 of 76

68 The Results panel is composed of a list-box, and 2 buttons.we will describe the function of each component below. The Match button: This button calls the ftedhole or fctrand functions, depending on the user selected matching method, to match a selected source image to its corresponding list of target images. The final matching parameters are then displayed in the Results list. The Details button: This button displays (inside the Result list) the matching details of the selected source-target image pair. The matching details displayed include the target and source image names and locations, the matching function used, and the parameters and matching times of the overall matching process, and of each individual hole-to-hole matching instance. When this button is first pressed its name changes to Back and it displays the required matching details, then when it is pressed again its name changes back to Details and the original final matching parameters are restored. See Figure A Figure A.2.4.2: Displaying matching details The Results list: As previously mentioned this list contains the resulting parameters of the matching process. Page 68 of 76

69 A.2.5. The Menus For each of the previously described buttons we included a menu shortcut. We also added submenus that allow users to save, open, and create new sessions. Also we developed a detailed help menu that explains the function of each component of the Proximity Software. We will provide a description of each of the save/save-as, open, new, and help sub-menus below. A The New sub-menu The new sub-menu is located in the File menu, and can be used to clear a current session and start a new blank one. Before starting a new session, the software asks the user if he/she would like to save the current session. If his/her choise is yes the save function is called, otherwise the workspace is cleared and all options are reset to their default values. See Figure A Figure A : Loading a new session (a) Page 69 of 76

70 A The Save/Save-As sub-menu Figure A : Loading a new session (b) The save/save-as sub-menus are located in the File menu, and can be used to save a current session to a user specified directory, and which can be reloaded in a future session. The session is saved as 2 files, a.fig file containing the axes structures created during the session, and a.mat function containing the internal data structures & cell arrays generated throughout lifetime of current session (A ). Figure A : Saving the current session Page 70 of 76

71 A The Open sub-menu Figure A : Saved session files This sub-menu is also located in the File menu, and can be used to retrieve a previously saved session and loading it to the current work space. Upon opening the saved session a message box appears informing the user that if the images that were used in the session being opened were modified after the last time the session was saved, the results and concavity tree representations of the modified images will not be reflected in the loaded session. See Figure A Figure A : Opening a saved session (a) Page 71 of 76

72 A The Help sub-menu Figure A : Opening a saved session (b) This sub-menu is located in the Help menu. When it is clicked a new interactive help window appears, containing a variety of helpful topics that guide the user through the various stages of image acquisition and matching. See Figure A Figure A : The Help window Page 72 of 76