PATTERN matching, also known as template matching, is

Transcription

1 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. X, XXXXXXX Performance Evaluation of Full Search Equivalent Pattern Matching Algorithms Wanli Ouyang, Member, IEEE, Federico Tombari, Member, IEEE, Stefano Mattoccia, Member, IEEE, Luigi Di Stefano, Member, IEEE, and Wai-Kuen Cham, Senior Member, IEEE Abstract Pattern matching is widely used in signal rocessing, comuter vision, and image and video rocessing. Full search equivalent algorithms accelerate the attern matching rocess and, in the meantime, yield exactly the same result as the full search. This aer rooses an analysis and comarison of state-of-the-art algorithms for full search equivalent attern matching. Our intention is that the data sets and tests used in our evaluation will be a benchmark for testing future attern matching algorithms, and that the analysis concerning state-of-the-art algorithms could insire new fast algorithms. We also roose extensions of the evaluated algorithms and show that they outerform the original formulations. Index Terms Pattern matching, temlate matching, fast algorithms, full search equivalent algorithm, erformance evaluation. Ç 1 INTRODUCTION PATTERN matching, also known as temlate matching, is the task of seeking a given attern in a given image, as illustrated in Fig. 1. Pattern matching is widely used in signal rocessing, comuter vision, and image and video rocessing. It has found alications in manufacturing for quality control [1], image-based rendering [2], image comression [3], object detection [4], suerresolution [5], texture synthesis [6], block matching in motion estimation [7], [8], image denoising [9], [10], [11], road/ath tracking [12], mouth tracking [13], image matching [14], and action recognition [15]. Suose an N 1 N 2 attern has to be sought in a given J 1 J 2 image of J ¼ J 1 J 2 ixels, as shown in Fig. 1. The attern will be comared to candidate windows of the same size in the image. The Full Search (FS) algorithm comutes a similarity or dissimilarity measure between the attern and all its equally sized candidate windows that can be extracted out of the image. We reresent the attern (temlate) as a length-n vector ~X t and the candidate windows as ~X w ðjþ, where subscrits t and w denote temlate and window, resectively, j ¼ 0; 1;...;W 1 and N ¼ N 1 N 2. For examle, if a attern is searched in a image, we have N ¼ 256, J ¼ 65;536, and W ¼ð þ 1Þ 2 ¼ 58;081. There are several ways to comare two vectors in order to comute their similarity. These are reresented by different matching measures that can be used for the comarison. In. W. Ouyang and W.-K. Cham are with the Chinese University of Hong Kong, China.. F. Tombari, S. Mattoccia, and L. Di Stefano are with the University of Bologna, Bologna, Italy. Manuscrit received 13 Set. 2010; revised 31 Jan. 2011; acceted 12 Mar. 2011; ublished online 13 May Recommended for accetance by P. Felzenszwalb. For information on obtaining rerints of this article, lease send to: tami@comuter.org, and reference IEEECS Log Number TPAMI Digital Object Identifier no /TPAMI this aer, we consider a class of matching measures that find vast use in temlate matching alications, i.e., those derived from a distance measure based on the L norm. In articular, we denote as dð ~X t ; ~X w ðjþþ the distance between ~X t and ~X w ðjþ, which measures the dissimilarity between ~X t and ~X w ðjþ. The smaller is dð ~X t ; ~X w ðjþþ, the more similar are ~X t and ~X w ðjþ. There are two different situations in attern matching: 1) Detect all candidate windows having dð ~X t ; ~X w ðjþ Þ <T, for a given threshold T, and 2) find the window that leads to the minimum value of dð ~X t ; ~X w ðjþ Þ among all candidate windows. In this aer, we mainly consider situation 1, where ~X w ðjþ is said to match ~X t when dð ~X t ; ~X w ðjþ Þ <T. As illustrated later, the algorithms for situation 1 can be easily modified to deal with situation 2. The L norm of length-n vector ~Z ¼½z 0 ;z 1 ;...;z N 1 Š T is defined as k~zk ¼ðjz 0 j þjz 1 j þþjz N 1 j Þ 1= : Based on the L norm, the dissimilarity between ~X t and ~X w ðjþ can be measured as k ~X t ~X w ðjþk. If ¼ 1, then the distance is the sum of absolute differences (SAD), while ¼ 2 yields the sum of squared differences (SSD). There is research using other similarity or dissimilarity measures that make attern matching robust to rotation, affine transformations, occlusion, and illumination variations. For examle, the dissimilarity measure between the attern descritors and the candidate window descritors is used in [16], [17], [18], [19], and [20], Hamming distance is used as the dissimilarity measure in [21], normalized cross correlation (NCC) is used as similarity measure in [22], [23], [24], [25], [26], and [27]. To limit the scoe of the aer we have not included these measures and only consider algorithms that use SAD and SSD as a dissimilarity measure. As ointed out in [28], though there are arguments against SSD as a dissimilarity measure for images, it is still widely adoted due to its simlicity. Discussions concerning the use of SSD as a dissimilarity metric can be found in [29], [30], and [31]. ð1þ /12/$31.00 ß 2012 IEEE Published by the IEEE Comuter Society

2 2 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. X, XXXXXXX 2012 TABLE 1 Abbreviations and References of Comared Algorithms Fast Fourier Transform (FFT) aroach is also used as a benchmark of comarison (in articular, we use the OenCV imlementation [51]). The main contributions of this aer are as follows: Fig. 1. Pattern matching in image coule. Since the FS algorithm is unaccetably slow in most alications, many faster aroaches have been roosed in the literature [22], [25], [27], [28], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48]. Among these aroaches, nonexhaustive algorithms yield comutational savings by reducing the search sace [38], [39], [40], [48] or by aroximating atterns and windows using olynomials [41], [42], [43] or linear combination of simle features [44]. Conversely, exhaustive (or FS-equivalent) algorithms accelerate the attern matching rocess and, at the same time, yield exactly the same result as FS. In the case of a dissimilarity-based search, a simle aroach is known as Partial Distortion Elimination (PDE) [45] and its high efficiency consists of terminating the evaluation of the current dissimilarity measure as soon as it rises above the current minimum. Another aroach suitable to dissimilarity-based searches consists of defining a raidly comutable lower bounding function of the adoted dissimilarity measure so as to quickly check one or more sufficient conditions to ski mismatched ositions without carrying out the heavier comutations required by the evaluation of the actual dissimilarity measure. Examles of such an aroach include the algorithms in [28], [32], [33], [34], [35], [49], [46], [47], [50], [22], [26]. This aer is motivated by the intense research activity recently develoed within this toic, but is not motivated by an exhaustive comarison of various roosals. The aim of the work is to comare and analyze state-of-the-art FSequivalent algorithms for attern matching in different conditions. For this aim, the aer selects the following five algorithms, which are recent and have been shown to yield notable seed-us against the FS aroach. 1. Alkhansari s Low Resolution Pruning (LRP) algorithm [32], 2. Tombari et al. s Incremental Dissimilarity Aroximation (IDA) algorithm [33], 3. Hel-Or and Hel-Or s rojection-based algorithm (PWHT) using Walsh-Hadamard Transform (WHT) [28], 4. Ben-Artzi et al. s rojection-based algorithm using GCK (PGCK) [34], and 5. Ouyang and Cham s rojection-based algorithm using fast WHT (FWHT) [35]. Table 1 summarizes the comared algorithms, together with the corresonding abbreviations. In addition to FS, the 1. A unified framework for describing the algorithms evaluated in this aer. Under this framework, we give a theorem that relates LRP to the algorithms that use WHT basis vectors. 2. The comutational analysis of IDA, PWHT, PGCK, and FWHT under the framework of attern matching, which is not rovided in the revious literature. Based on this analysis, we roose a new termination strategy for fast full-search equivalent attern matching algorithms and two additional efficient formulations of LRP. 3. A quantitative erformance evaluation of state-ofthe-art FS-equivalent attern matching algorithms based on a very large data set. This allows us to identify the best erforming methods under a number of nuisances found in real-world alications. Moreover, the data sets, methodology, and results used in our evaluation rovide a reference framework for testing future attern matching algorithms. 1 This aer is organized as follows: Section 2 introduces FS and FFT. Section 3 resents a unified framework that can reresent all evaluated algorithms for further analysis. Based on this unified framework, the comutational comlexity of evaluated algorithms is analyzed in Section 4. Then, the algorithms are comared quantitatively using the data sets described in Section 5, which account for different sizes of images and atterns as well as for distortions caused by different tyes of noise. The testing environment and evaluation criterion will also be described in Section 5. Section 6 illustrates the exerimental results. Finally, Section 7 resents a discussion and Section 8 draws conclusions. 2 FSAND FFT APPROACH 2.1 Full Search Aroach With the FS algorithm, the distance between attern ~X t and each candidate window ~X w ðjþ, i.e., k ~X t ~X w ðjþk for j ¼ 0;...;W 1, is measured. If k ~X t ~X w ðjþk <T, then the window ~X w ðjþ is considered as a matching window; otherwise, it is considered as a mismatched window. When SSD is used, FS requires about 2NW additions and NW multilications. When SAD is used, FS requires about 2NW additions and NW absolute value oerations. 2.2 Fast Fourier Transform The FFT-based aroach can be used only with the SSD. As ointed out in [33], the SSD function can be written as 1. Data sets and code to carry out the exeriments will be made ublicly available through a website.

3 OUYANG ET AL.: PERFORMANCE EVALUATION OF FULL SEARCH EQUIVALENT PATTERN MATCHING ALGORITHMS 3 TABLE 2 A Framework for Pattern Matching Using Lower Bound TABLE 3 Symbols and Terms Used in the Paer k ~X t ~X w ðjþ k2 2 ¼k ~X t k 2 2 þk ~X w ðjþ k2 2 2 < ~X t ; ~X w ðjþ >; ð2þ where < ~X t ; ~X w ðjþ> ¼ð ~X t Þ T ~X w ðjþ is the inner roduct between ~X t and ~X w ðjþ. FFT facilitates efficient comutation of the inner roduct < ~X t ; ~X w ðjþ > in (2). As ointed out in [22], the FFT-based aroach requires about 6Jlog 2 J additions and 6Jlog 2 J multilications for comuting the inner roduct. After that, W additions are required to obtain 2< ~X t ; ~X w ðjþ> from < ~X t ; ~X w ðjþ> for j ¼ 0; 1;...;W 1. To comute the term k ~X w ðjþk2 2 at each ixel location, J multilications are required for squaring ixel values, 4W and 5W additions are required, resectively, by the box-filtering technique [52] and the integral image aroach [53] for summing u the squared values k ~X w ðjþk2 2 ¼ P x w;j;a2 ðx ~X ðjþ w;j;a Þ 2 for j ¼ 0;...;W 1. k ~X t k 2 2 can w be obtained by N multilications and 2N additions. Finally, 2W additions are needed for summing u the three terms in (2). In summary, the FFT-based aroach requires at least 6Jlog 2 J þ 7W additions and 6Jlog 2 J þ J multilications. 3 A UNIFIED FRAMEWORK FOR PATTERN MATCHING ALGORITHMS In this section, we first introduce a unified framework for fast FS-equivalent attern matching. The links and differences among the algorithms comared in this aer are then analyzed within this framework. As highlighted in Table 2, the roosed framework consists of two stes:. Ste a. Mismatching candidate windows are eliminated from set can. This ste is called the rejection ste.. Ste b. The remaining candidate windows in set can undergo FS for finding out the matching windows. This ste is called the FS-ste. In this framework, both FS and FFT directly evaluate the distance k ~X t ~X w ðjþk for all candidate windows to find the matching windows. This corresonds to skiing the rejection ste and starting from the FS-ste in Table 2. The FFT aroach is only different from FS in the comutation of k ~X t ~X ðjþ w k. On the other hand, algorithms IDA, LRP, PWHT, PGCK, and FWHT start from the rejection ste. The rejection ste is a loo of k, where k increases from 1, and comrises three substes. In Ste a.1, a lower bounding function f low ðk; jþ is evaluated such that k ~X t ~X ðjþ w k f lowðk; jþ. InSte a.2, if f low ðk; jþ T, then k ~X t ~X w ðjþk T and we can safely rune ~X w ðjþ from set can, f low ðk; jþ T being the rejection condition. In Ste a.3, the loo of k terminates when a given termination condition, denoted as Cond Ter is satisfied. Throughout the rejection ste, each candidate window undergoes checking of a succession of rejection conditions f low ðk; jþ T in each loo of k until either it is runed or the rejection ste finishes. There are two conditions under which the rejection terminates: 1) k reaches a sufficient number, which is denoted as N Maxk, and 2) the ercentage of remaining candidate windows in iteration k þ 1, denoted as Percan ðkþ1þ, is below a certain threshold, denoted as. The second termination condition, corresonding to Cond Ter in Ste a.3, is a strategy used by Hel-Or and Hel-Or in [28] because it turns out to be faster to directly calculate the actual distance than evaluating the lower bound when the remaining candidates in set can are very few. N k records the actual number of loos of run in the rejection ste. Table 3 shows the meaning for symbols in this aer. The advantage of using f low ðk; jþ is that it is more efficient to comute f low ðk; jþ than to comute k ~X t ~X w ðjþk and a small number of iteration k in the rejection ste can eliminate a large number of mismatching windows. The unified framework can be modified to find the window that has the minimum k ~X t ~X w ðjþk using the aroaches roosed in [50], [28], and [27]. Taking the aroach in [28] as an examle, the threshold T can be adated in each loo of k based on the minimum lower bound found in the kth loo. The differences among the algorithms are summarized in Table 4. IDA, LRP, PWHT, PGCK, and FWHT are alicable for any dissimilarity measure based on the L -norm ( 1), while FFT is based only on L 2 -norm. FFT

4 4 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. X, XXXXXXX 2012 TABLE 4 Differences for the Unified Framework in Table 2 PWHT, PGCK, and FWHT share the column PKs. includes only the FS-ste, while the others include both the rejection ste and the FS-ste. N Maxk is not alicable for FFT; the IDA algorithm has N Maxk ¼ N P ; the LRP algorithm has N Maxk ¼ log h N; PWHT, PGCK, and FWHT have N Maxk ¼ N. The meaning of arameters N P and h is given in Table 3 and will be illustrated later. The FS-equivalent algorithms considered in the aer achieve high efficiency by using a lower bounding function, f low ðk; jþ, that eliminates mismatching candidates early. Hence, the methods for estimating these lower bounds greatly affect the comutational erformance of FS-equivalent algorithms. In the following, we recall the lower bound estimation methods used by IDA, LRP, PWHT, PGCK, and FWHT. 3.1 The Lower Bounding Function for IDA The IDA technique relies on artitioning the attern vector, ~X t, and each candidate vector, ~X w ðjþ, into a certain number of subvectors in order to determine a succession of runing conditions characterized by increasing tightness and comutational weight. Given an N-dimensional vector, IDA establishes a artition P of the vector into N P disjoint subvectors (not necessarily with the same number of comonents). In articular, it defines a artition of set f1; 2;...;Ng into N P disjoint subsets P ¼fP 1 ;P 2 ;...;P NP g, where [ N P m¼1 P m ¼f1; 2;...Ng, 1 N P N, and N P is a arameter. Given artition P, IDA defines the artial L -norm of vector ~X t and ~X w ðjþ limited to the subvector associated with P m 2 P as k ~X t k ;Pm ¼ X n2p m jx t;n j!1 ; ð3þ k ~X ðjþ w k ;P m ¼ X n2p m xw;j;n!1 ; ð4þ where x t;n is the nth element in attern ~X t and x w;j;n is the nth element in window ~X w ðjþ. In addition, IDA defines the artial L -dissimilarity between ~X t and ~X w ðjþ limited to the subvector associated with P m 2 P as k ~X t ~X ðjþ w k ;P m ¼ X n2p m xt;n x w;j;n : ð5þ As roven in [33] using reverse triangle inequality, the lower bound used by IDA in iteration k is f low ðk; jþ ¼ Xk 1 m¼1 þ XNP n¼k k ~X t ~X ðjþ w k ;P m jk~x t k ;Pn k~x ðjþ w k ;P n j ; where the left term is the artial L -dissimilarity between ~X t and ~X w ðjþ and the right term is an estimation of the remaining L -dissimilarity. Let us denote the modulo oeration by %. When artitioning a set of size N into N P subsets, we constrain that N%N P ¼ 0 and the subsets P n for n ¼ 1;...;N P have the same size, i.e., N=N P. Thus, the candidate window is equally artitioned into N P subwindows having the same size. The method resented in [33] does not imly this constraint, but this constraint normally makes IDA comutationally more efficient and facilitates comutational comlexity analysis. Thus, the constraint is used in this aer as it is also the case of the exerimental results reorted in [33]. ð6þ 3.2 The Lower Bounding Function for LRP, PWHT, GCK, and FWHT The transformation that rojects a vector ~X 2 IR N onto a linear subsace sanned by Uð NÞ basis vectors ~V ð0þ ;... ~V ðu 1Þ can be reresented as follows: ~Y ¼ V ðunþ ~X ¼½~V ð0þ... ~V ðu 1Þ Š T ~X; ð7þ where T is matrix transosition, vector ~X of length N is called the inut window, vector ~Y of length U is called the rojection value vector, the U elements in vector ~Y are called rojection values, and V ðunþ is a U N matrix that contains U basis vectors ~V ðiþ of length N for i ¼ 0;...;U 1. When V ðunþ is a square matrix, i.e., U ¼ N, we denote it as V ðnþ. The above transformation is also called rojection, e.g., in [28], [37], and [33], while basis vectors are called rojection kernels in [28] and called filter kernels in [34]. Algorithms using transformation for attern matching are called transform domain attern matching algorithms, e.g., PWHT, PGCK, FWHT, and LRP LRP LRP uses the following lower bounding function: f low ðk; jþ ¼ kv ðuðkþnþ ~X t ~X w ðjþ k kv ðuðkþnþ k ; ð8þ where uðkþ is the number of basis vectors chosen for transform domain attern matching in the kth loo of the rejection ste. The kv ðuðkþnþ k in (8) is the induced matrix -norms is defined as follows: kv ðuðkþnþ kv ðuðkþnþ ~Zk k ¼ max : ð9þ ~Z6¼0 k~zk When not all elements in V ðuðkþnþ are zeros, it follows from (9) that k~zk kv ðuðkþnþ ~Zk kv ðuðkþnþ k : ð10þ

5 OUYANG ET AL.: PERFORMANCE EVALUATION OF FULL SEARCH EQUIVALENT PATTERN MATCHING ALGORITHMS 5 As ointed out in [32], the lower bound in (8) is derived from (10) by relacing the ~Z in (10) with ~X t ~X w ðjþ: k ~X t ~X w ðjþ k kv ðuðkþnþ ~X t ~X w ðjþ k kv ðuðkþnþ k ¼ f low ðk; jþ: ð11þ The LRP algorithm is best exlained using Kronecker roduct, which is denoted as. IfA is a U 1 Q 1 matrix ða n1 ;n 2 Þ and B is a U 2 Q 2 matrix (b m1 ;m 2 ), then A B is the following U 1 U 2 Q 1 Q 2 matrix: 2 3 a 0;0 B a 0;1 B a 0;Q1 1B a 1;0 B a 1;1 B a 1;Q1 1B A B ¼ : ð12þ a U1 1;0B a U1 1;1B a U1 1;Q 1 1B Denote the uðkþuðkþ identity matrix as I ðuðkþþ. The 1D transform matrix for the LRP algorithm roosed in [32] is given by V ðuðkþnþ ¼ I ðuðkþþ 1 ð1ðn=uðkþþþ 2 1 ð1ðn=uðkþþþ 0 ð1ðn=uðkþþþ 0 ð1ðn=uðkþþþ 3 0 ð1ðn=uðkþþþ 1 ð1ðn=uðkþþþ 0 ð1ðn=uðkþþþ ¼ ; 0 ð1ðn=uðkþþþ 0 ð1ðn=uðkþþþ 1 ð1ðn=uðkþþþ ð13þ where 0 ð1ðn=uðkþþþ and 1 ð1ðn=uðkþþþ are 1 ðn=uðkþþ matrices with all elements equal to 0 and 1, resectively, uðkþ ¼h k 1 is the number of basis vectors used for evaluating the lower bound function in (8), and h can be any small integer number. The exeriments in [32] use h ¼ 4. As an examle for the LRP matrix in (13), when uðkþ ¼2, N ¼ 4, we have V ð24þ ¼ I ð2þ 1 ð12þ ¼ 1 0 ½11Š¼ : ð14þ As roven in [32], kv ðuðkþnþ k ¼ðN=uðkÞÞð 1Þ= in (8) for the V ðuðkþnþ defined in (13). For candidate windows in the images, transformation using LRP is comuted by summing ixel values in a rectangle PWHT, PGCK, and FWHT The following inequality about the L 2 norm, roven in [28], is used for PWHT, PGCK, and FWHT: k ~X t ~X w ðjþ k2 2 ¼k ~Dk 2 2 ðv ðuðkþnþ ~DÞ T ðv ðuðkþnþ V ðuðkþnþt Þ 1 ðv ðuðkþnþ ~DÞ ¼ f low ðk; jþ; where ~D ¼ ~X t ~X w ðjþ : ð15þ PWHT and FWHT use WHT as the transform matrix for attern matching. The WHT transform matrix can be recursively defined as M ðnþ ¼ M ð2þ M ðn=2þ ¼ MðN=2Þ M ðn=2þ M ðn=2þ M ðn=2þ ; ð16þ where M ð1þ ¼ 1, M ðnþ ¼½~M ð0þ... ~M ðn 1Þ Š T is an N N matrix. As two examles for (16), we have Fig. 2. WHT basis vectors in sequency order and natural order. White reresents the value þ1 and gray reresents the value M ð2þ ¼ ;M ð4þ ¼ : ð17þ Since the elements in WHT basis vectors contain only 1 or 1, rojection of inut data onto WHT basis vectors requires only additions and subtractions. When N ¼ 8, Fig. 2 shows the order-8 WHT basis vectors in natural order and sequency order. The WHT in (17) and (16) is in natural order. The natural order and the sequency order are different methods for ordering WHT basis vectors [54]. For sequency-ordered WHT, the satial frequency extracted by the basis vector increases as the index i of basis vectors ~M ðiþ increases. According to [28], PWHT is almost two orders of magnitude faster than FS and can deal with illumination effects and multiscale attern matching. The transform matrix for the GCK in [34] can be reresented as follows: V ðnþ ¼ M ðn=rþ S ðrþ ; ð18þ where M ðn=rþ is the N=R N=RWHT matrix and S ðrþ can be any R R orthogonal matrix. The basis vectors in 1ffiffiffi M ðuðkþnþ are orthonormal basis vectors. And we have N kv ðuðkþnþ k 2 2 ¼ 1 if the basis vectors in V ðuðkþnþ are orthonormal. Thus, we have km ðuðkþnþ k 2 2 ¼ N for the WHT matrix in (16). Similarly, we have kv ðuðkþnþ k 2 2 ¼ N=R for the GCK matrix in (18). Considering the general lower bounding function defined in (8), we can select basis vectors of the V uðkþn in (8) from the LRP matrix in (13), from the WHT matrix M ðnþ,or from the GCK matrix in (18). For examle, if N ¼ 4, k ¼ 2, and uðkþ ¼k ¼ 2, then we can select the first two WHT basis vectors ~M ð0þ and ~M ð1þ in (17) for constructing the matrix V ð24þ ¼½~M ð0þ ~M ð1þ Š T and use this matrix as the V ðuðkþnþ in (8). 3.3 Investigation on PWHT, PGCK, FWHT, and LRP The f low ðk; jþ in (11) is used for LRP while the f low ðk; jþ in (15) is used for PWHT, PGCK, and FWHT. Let us consider the L 2 norm; if the basis vectors in V ðuðkþnþ in (15) are orthonormal, then we have kv ðuðkþnþ k 2 ¼ 1 in (11) and ðv ðuðkþnþ V ðuðkþnþt Þ 1 ¼ I ðuðkþþ in (15). Under this condition, both (11) and (15) yield the following relation: k ~X t ~X w ðjþ k2 2 kv ðuðkþnþ ð ~X t ~X w ðjþ Þk2 2 ¼ f lowðk; jþ: ð19þ The inequality in (11) is more general than that in (15) because

6 6 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. X, XXXXXXX the inequality in (15) is only alicable for L 2 norm while the inequality in (11) is alicable for L norm for 1, and. V ðuðkþnþ is required to have rank uðkþ for the inverse matrix in (15) while this is not required in (11). The inequality in (15) roosed in [28] were used by PWHT, PGCK, and FWHT for the L 2 norm only. For other norms, the inequality roosed in [28] is as follows: TABLE 5 Comutation Required by the Algorithms with Symbols Illustrated in Table 3 k ~X t ~X w ðjþ kk ~V ðiþ T ~X t ~V ðiþt ~X w ðjþk : ð20þ k ~V ðiþ k The inequality in (20) is a secial case of that in (7) with V ðunþ being a row vector in (7). The following theorem describes the relationshi between LRP and WHT: Theorem 1. When the u ¼ 2 n basis vectors in V ðunþ WHT are the first sequency-ordered WHT basis vectors, we have the LRP transform matrix V ðunþ LRP ¼ I ðuþ 1 ð1ðn=uþþ ; ð21þ such that: 1) The subsace sanned by the u basis vectors in V ðunþ WHT is equal to the subsace sanned by the u basis vectors in V ðunþ LRP ; 2) for any length-n inut vector ~X, if the basis vectors in V ðunþ LRP and V ðunþ WHT are normalized to have L 2 norm equal to 1, then kv ðunþ ~ WHT Xk 2 ðunþ 2 ¼kVLRP ~Xk 2 2 ; and 3) the transformation V ðunþ ~ WHT X requires at least 3u=2 additions er ixel, while the transformation V ðunþ LRP ~X requires three additions er ixel when comuted on sliding windows. The roof for Theorem 1 is rovided in the Aendix, which can be found in the Comuter Society Digital Library at htt://doi.ieeecomutersociety.org/ /tpami The more the energy is acked, the more candidates can be eliminated by the rejection ste. Theorem 1 states that the u ¼ 2 n sequency-ordered basis vectors in V ðunþ ðunþ WHT and the u basis vectors in VLRP ¼ I ðuþ 1 ð1ðn=uþþ are the same in subsace sanning and energy acking ability. However, the larger u is, the more comutationally efficient the transformation V ðunþ LRP ~X is than V ðunþ ~ WHT X. Recently, WHT have been used for motion estimation in [7], [8] based on L 1 norm, but the algorithms are not FS-equivalent. Actually, we can see from the analysis in (11) that the basis vectors selected from WHT and GCK are also alicable for FS-equivalent attern matching using L norms as dissimilarity measure. 4 COMPUTATIONAL ANALYSIS OF IDA, PWHT, PGCK, FWHT, AND LRP In this section, we analyze the comutational comlexity of IDA, PWHT, PGCK, FWHT, and LRP by counting the number of oerations required by each algorithm. This analysis is summarized in Table 5. The terms related to the attern ~X t are comuted once and for all at initialization time and hence not considered in the comutational comlexity since the associated comlexity is negligible comared with the comutation related to candidate windows on the image. PWHT, PGCK, and FWHT share the same row PKs. In this table, A ðk;hþ LRP ¼ hðk 1Þ Ncan ðkþ. To recall the notation already introduced, the N 1 N 2 attern has N ¼ N 1 N 2 ixels, the J 1 J 2 image has J ¼ J 1 J 2 ixels and W candidate windows. At each iteration of k for k ¼ 1; 2;... in the rejection ste, some candidate windows are eliminated and the remaining ones will be considered in the next loo. We denote the number of candidates examined at iteration k in the rejection ste as Ncan ðkþ and the number of candidates examined in the FS-ste as Ncan ðfsþ. Initially, k ¼ 1, all candidates are examined, and we have Ncan ð1þ ¼ W. Different algorithms have different values of Ncan k. In the comutational analysis, subtractions are given the same weight as additions. The comutation of certain terms deends on the comutation of the L norm in (1), which in turn is related to the comutation of jzj ffiffi and z. The oerations for comuting jzj and ffiffi z are here called ower oeration and root- oeration, resectively. When ¼ 2, ower- oeration comutes the square of z, i.e., z 2, and ffiffi root- oeration comutes the square root of z, i.e., z. When ¼ 1, ower- oeration comutes the absolute value of z, i.e., jzj, and the root- oeration does nothing. As a common ste for algorithms IDA, PWHT, PGCK, FWHT, and LRP, Ste a.2 of Table 2 checks condition f low ðk; jþ T for the Ncan ðkþ candidates at iteration k, which requires P k NðkÞ can comarison oerations. The comutation required by Ste a.3 of Table 2 is negligible. The remaining stes in Table 2 that require analysis are Ste a.1 of the Rejection ste, which comutes the lower bounding

7 OUYANG ET AL.: PERFORMANCE EVALUATION OF FULL SEARCH EQUIVALENT PATTERN MATCHING ALGORITHMS 7 TABLE 6 Comutation of the Lower Bounding Function Using IDA TABLE 7 Comutation of the Lower Bounding Function Using PWHT, GCK, and FWHT 4.2 Comutational Analysis for LRP As ointed out in [32], LRP requires P k ½2Aðk;hÞ LRP þðh 1ÞJŠ additions and P k ðaðk;hþ LRP Þ ower- oerations for obtaining the lower bounding function, where A k;h LRP ¼ hðk 1Þ N ðkþ the FS-ste, LRP requires 2NN ðfsþ can ower- oerations. The sliding transformation V ðuðkþnþ ~X w ðjþ in can.in additions and NN ðfsþ can f low ðk; jþ ¼ kv ðuðkþnþ ~X t V ðuðkþnþ ~X ðjþ w k kv ðuðkþnþ k function f low ðk; jþ, and the FS-ste. In the following, we analyze the comutational comlexity of these two stes for each of the considered algorithms. 4.1 Comutational Analysis for IDA The comutation of the lower bounding function for IDA is illustrated and analyzed in Table 6. When k ¼ 1, which is Case 1 of Table 6, the lower bounding function in (6) is given by f low ð1;jþ¼ XNP m¼1 jk~x t k ;Pm k~x ðjþ w k ;P m j : ð22þ When k>1, i.e., Case 2 of Table 6, the lower bounding function is f low ðk; jþ ¼f low ðk 1;jÞþk~X t ~X ðjþ w k ;P k jk~x t k ;Pk k~x ðjþ w k ;P k j : ð23þ As a summary of Table 6, IDA requires 4W þ 2N P W þ P NP 1 k¼2 ½ð 2N N P þ 2ÞNcan ðkþ Š additions, J þ N P W þ N N P ð P N P 1 k¼2 Ncan ðkþ Þ ower- oerations, and W root- oerations for obtaining the lower bounding function. In the FS-ste, IDA comutes only the L norm dissimilarity for ixels in the last subwindow restricted by artition P NP. This subwindow contains N N P ixels for each candidate window. Thus, the FS-ste requires 2NNðFSÞ can N P additions and ower- oerations. NN ðfsþ can N P is actually the sum of ixel values in rectangles when LRP is used for images. This transformation is comuted by hierarchical summation on the entire image in [32], which requires ðh 1ÞJ additions at each iteration k. This aroach is denoted as LRP hier. 4.3 Analysis for PWHT, PGCK, and FWHT The lower bounding function for PWHT, PGCK, and FWHT is comuted as f low ðk; jþ ¼f low ðk 1;jÞþj~V ðk 1ÞT ~X t ~V ðk 1ÞT ~X w ðjþ j ; ð24þ where k ¼ 1; 2;...;N k, f low ð0;jþ¼0. N k records the actual number of iterations run in the rejection ste. The rocedure and number of oerations required to obtain the lower bounding function is reorted in Table 7. Let BðN;N k ;WÞ be the number of oerations required by PWHT, PGCK, or FWHT to obtain V ðn knþ ~X w ðjþ for W candidates. PWHT [28] has two different aroaches to comute the transformation: to-down and bottom-u. We have BðN;N k ;WÞ¼WN k log N in the worst case and BðN;N k ;WÞ¼2WN k in the best case for to-down PWHT. For bottom-u PWHT, BðN;N k ;WÞ¼ P k NNðkÞ can in the worst case and BðN;N k ;WÞ¼ P k NðkÞ can in the best case. As for PGCK and FWHT, BðN;N k ;WÞ¼2WN k and BðN;N k ;WÞ¼3WN k =2, resectively. As a summary of Table 7, PWHT, PGCK, and FWHT need BðN;N k ;WÞþ additions and P k NðkÞ can ower- oerations in 2 P k NðkÞ can order to calculate the lower bounding function. In the FS-ste, 2NNcan ðfsþ additions and NNcan ðfsþ oerations are needed by PWHT, PGCK, and FWHT. ower-

8 8 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. X, XXXXXXX New Imlementation of LRP The hierarchical summation aroach in [32] is a technique that is deendent on the arameter h. However, we can use the integral image method to comute the sum of ixel values in rectangles. The integral image method initially requires about 2J additions for constructing the integral image and then 3W additions for each iteration k for comuting the sliding transformation over the image. This aroach is denoted as LRP sld. LRP sld is advantageous over the hierarchical summation roosed in [32] because the comutation required by LRP sld is indeendent of h. As another alternative way, we may comute the transformation for the Ncan ðkþ candidates instead of comuting it for the entire image, which would require 3h ðk 1Þ Ncan ðkþ additions for each k using the integral image method. This aroach is denoted as LRP can. 4.5 New Termination Condition Based on Comutational Comlexity Analysis As illustrated before, the termination condition in [28] has been used by PWHT and PGCK for early termination of the rejection ste, which corresonds to Cond Ter at Ste a.3 in Table 2. Let the comutation required for the FS-ste be C FS and the comutation required for iteration k þ 1 in the rejection ste be C ðkþ1þ Rej. The idea behind the strategy in [28] is that the rejection ste should be terminated when it is more efficient to directly calculate the actual distance of the remaining candidates in the FS-ste than to continue the comutation in the rejection ste, i.e., when C FS <C ðkþ1þ Rej. The strategy roosed in [28] terminates the rejection ste when the ercentage of remaining candidate windows checked in iteration k þ 1, i.e., Percan ðkþ1þ, is below a certain threshold. Hence, in the absence of a comutational analysis, setting a threshold is intuitively a simlified version of the comarison C FS <C ðkþ1þ Rej. However, the comutational comlexity given in Table 5 allows us to devise a novel and rinciled aroach to the early termination strategy. The new termination strategy based on comutational analysis consists of terminating the rejection ste if Cond T er;1 is true or Cond Ter;2 is true. Cond T er;1 is true when the comutation required for the iteration k þ 1 of the rejection ste is greater than the comutation required for the FS-ste. Cond Ter;2 is true when the comutation required by iteration k of the rejection ste is greater than the comutation it saves. This strategy is easily alicable to all the algorithms evaluated in this aer. In the following exeriments, this new termination strategy is used for PGCK and FWHT unless secified otherwise. In Section 6.7, we will comare the original strategy in [28] with the roosed strategy. 5 PERFORMANCE EVALUATION 5.1 Data Set In order to evaluate the erformance of the comared algorithms, 14 data sets containing different sizes of atterns and images are used. As shown in Table 8, the data sets are denoted as In 1 Tn 2 for n 1 ¼ 1; 2; 3; 4; 5, n 2 ¼ 1;...; 4, where n 1 corresonds to image size and n 2 corresonds to attern size. Large n 1 or n 2 corresonds to large size. For TABLE 8 Data Sets Used in the Exeriments J and N corresond to the number of ixels in image and attern, resectively. examle, data sets I2 T1 and I2 T2 have the same image size but different attern sizes. The exeriments include a total of 150 grayscale images chosen among three databases: MIT [55], medical [56], and remote sensing [57]. The MIT database is mainly concerned with indoor, urban, and natural environments, lus some object categories such as cars and fruits. The two other databases are comosed, resectively, of medical (radiograhs) and remote sensing (Landsat satellite) images. The 150 images have been subdivided into five grous of 30 images, each grou being characterized by a size of images in I1 T1, I2 T1, I3 T1, I4 T1, I5 T1, (i.e., , , , 1; , 2;560 1;920). For each image, 10 atterns were randomly selected among those showing a standard deviation of ixel intensities higher than a threshold (i.e., 45) for each data set. Data sets having the same image size share the same images but have different atterns in both size and location. For examle, data sets I2 T1 and I2 T2 share the same 30 images having size but have different atterns in size and location. So, each data set contains 300 image-attern airs. Since we have 14 data sets, there are 4,200 image-attern airs in all. This data set originated from that in [33] and is extended in this aer to include more sizes of atterns. 5.2 Evaluation Criterion In the exeriments, both SSD and SAD are used as the dissimilarity measure between attern and candidate window. If the SSD or SAD between any candidate window in the image and the attern is below the threshold, the candidate window is regarded as matching the attern. Given a attern having N ixels, the SSD threshold is set as follows: T SSD ¼ 1:1 SSD min þ N; ð25þ where SSD min is the SSD between the attern and the best matching window. Similarly, we have the following for the SAD threshold: T SAD ¼ 1:1 SAD min þ N; ð26þ

9 OUYANG ET AL.: PERFORMANCE EVALUATION OF FULL SEARCH EQUIVALENT PATTERN MATCHING ALGORITHMS 9 TABLE 9 Parameters Used in the Exeriments TABLE 10 Hardware Environment Used in the Exeriments Fig. 3. Seed-us in execution time for images without noise when SSD is used. The X-axis corresonds to data sets I1 T1;I2 T1;...;I4 T4 in Table 8. where SAD min is the SAD between the attern and the best matching window. We develoed the code for IDA and FWHT since we are authors of these algorithms. The code for Hel-Or and Hel-Or s PWHT [28] is from the authors website [58], with some small modifications introduced by us to deal with large atterns. The code for PGCK [34] is based on the code rovided by Hel- Or and Hel-Or, which was reviously used for motion estimation in [8] and was modified by us for the attern matching task. Finally, we wrote the code for LRP according to the algorithm described in [32]. All of the algorithms are written in C, comiled with VC 6.0, and run on windows XP systems as single-thread tasks. Our imlementation checks that all algorithms find the same matched windows and same distance as FS in all exeriments for assuring correctness results are measured as seed-us over the FS algorithm in terms of execution time. As an examle, the seed-u of IDA over FS in execution time is measured as the execution time required by FS divided by that required by IDA. The arameter N for IDA are the same as in [33]. The arameter h for LRP is 4, which is the same as in [32]. The arameters used in the exeriments are summarized in Table 9. The arameter N Maxk is set as 50 for PWHT in [28], while it is set as 96 in our exeriments because we find that setting N Maxk ¼ 96 makes PWHT faster than setting N Maxk ¼ 50 in most cases, esecially when the attern size is large and the noise level is high. Since all the evaluated algorithms find the same matching windows as the FS, the only concern is comutational efficiency, which is assessed here in terms of execution time. In articular, as listed in Table 10, we have measured the execution time on three Intel CPUs and one AMD CPU with different memory sizes. The memory requirement for all algorithms is smaller than 1 GB in our imlementation. If not otherwise secified, the reorted seed-us in execution time are averages of the seed-us measured in each of the four environments. As an examle, the seed-u of IDA in execution time is the average of the seed-us of IDA over FS measured in the four environments. In Sections 4.2 and 4.4, three different imlementations of the LRP algorithm were introduced. LRP hier is the original imlementation that uses the hierarchical summation. LRP sld comutes the transformation in a sliding manner on the entire image using the integral image, while LRP can comutes the transformation for Ncan ðkþ candidate windows at loo k. These three imlementations were evaluated in the exeriments. Fig. 4. Seed-us in execution time for images without noise when SAD is used. The X-axis corresonds to data sets I1 T1;I2 T1;...;I4 T4 in Table 8. 6 EXPERIMENTAL RESULTS 6.1 Exeriment on Images without Noise In this exeriment, we evaluate algorithms on the data sets described in Table 8, which corresond to different sizes of images and atterns. The seed-us in execution time yielded by the evaluated algorithms using SSD as the dissimilarity measure are shown in Fig. 3. In this exeriment, the algorithms can be ordered from fastest to slowest as follows: 1. PGCK, 2. FWHT, IDA, and LRP, 3. PWHT, 4. FFT. PGCK is faster than FWHT because very few (less than 4) basis vectors are comuted in this exeriment. The exerimental results in [35] also show that FWHT is slower than PGCK when the number of basis vectors used is less than 4. With the excetion of data set I4 T4, where FFT is faster than PWHT, FFT is always slower than the other evaluated algorithms. The seed-us in execution time using SAD are shown in Fig. 4. PGCK and IDA are the fastest in this exeriment, with IDA faster than PGCK in data sets I1 T1, I2 T1, I2 T2, I3 T2, I3 T3, and I4 T4. Ordering of the other algorithms is similar to that attained using SSD (Fig. 3). 6.2 Exeriment on Images with Gaussian Noise In this exeriment, four different levels of iid zero-mean Gaussian noise are added to each image of the data sets 2. Corresonding to 0.005, 0.01, 0.02, and 0.03 on normalized ixel intensities ranging within ½0; 1Š.

10 10 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. X, XXXXXXX 2012 Fig. 5. An image from the data set and its distorted images. First row: The original image and its images with Gaussian noise levels Gð1Þ to Gð4Þ; second row: images with blurring levels Bð1Þ to Bð5Þ; third row: images with JPEG comression quality levels Jð1Þ to Jð5Þ. described in Table 8. The four Gaussian noise levels having variances 325, 650, 1,300, and 2,600 2 for 8-bit ixel value are referred to as Gð1Þ, Gð2Þ, Gð3Þ, and Gð4Þ, resectively. Fig. 5 shows an image from the data set in Table 8 and its distorted images with noise levels Gð1Þ to Gð4Þ, where the distorted images have PSNR 23.23, 20.4, 17.7, and 16.1 when comared with the original image. The seed-us in execution time using SSD for this exeriment are shown in Fig. 6. Each subfigure in Fig. 6 corresonds to a data set in Table 8. In Fig. 6, the to row corresonds to the smallest image size and bottom row corresonds to the largest image size; the leftmost column corresonds to the smallest attern size and the rightmost column corresonds to the largest attern size. This is similar for Figs. 7, 8, 9, 10, 11. As shown in Fig. 6, the seed-u of FFT over FS is indeendent of the noise level, while the seed-us of the other fast algorithms decrease as the noise level increases from Gð1Þ to Gð4Þ, so that FFT turns out to be in most cases the fastest method when the noise level is very high, i.e., Gð4Þ in Fig. 6. At the lower noise levels, LRP sld is, in most cases, the fastest algorithm, and quite close to the fastest in other cases. FWHT turns out to be faster than PGCK. This is because the actual number of comuted bases N k is greater than five in this exeriment, where FWHT is faster than PGCK in comuting the transformation. PGCK is faster than PWHT in most cases because PGCK is more efficient than PWHT in comuting the transformation. FWHT is faster than LRP when noise is low and attern size is small, esecially when image size is large. When the number of required basis vectors is large, LRP outerforms FWHT in comuting the transformation, which is roven in Theorem 1. IDA erforms well when attern-size is and has similar erformance as PWHT in other cases. 6.3 Exeriment for Blurred Images In this exeriment, five different levels of Gaussian low ass filter are used for blurring each image of the data sets described in Table 8. The five blurring levels, which are referred to as Bð1Þ, Bð2Þ, Bð3Þ, Bð4Þ, and Bð5Þ, corresond to Gaussian low ass filter having standard deviation ¼ 0:2; 0:9; 1:6; 2:3; 3. Fig. 5 shows an image from the data set in Table 8 distorted with blurring levels Bð1Þ to Bð5Þ, where the distorted images have PSNR 27.79, 27.18, 25.36, 24.14, and 23.49, resectively, when comared with the original image. In ractice, blur is introduced by changes of camera focus or by alication of simle denoising techniques. The seed-us in execution time using SSD for this exeriment are shown in Fig. 7. FFT, FWHT, and LRP sld are the three fastest algorithms. FFT is the fastest for image size with blurring levels Bð2Þ to Bð5Þ, and, for Fig. 6. Seed-us in execution time for images with Gaussian noise when SSD is used. I: T: corresonds to the attern sought in the image denoted as I1 T 1 in Table 8.

11 OUYANG ET AL.: PERFORMANCE EVALUATION OF FULL SEARCH EQUIVALENT PATTERN MATCHING ALGORITHMS 11 Fig. 7. Seed-us in execution time for blurred images when SSD is used. image sizes and with blurring levels Bð4Þ and Bð5Þ, FWHT is the fastest for data sets I4 T1 and I5 T2 with blurring levels Bð1Þ to Bð2Þ, and, for data set I5 T1 with blurring levels Bð1Þ to Bð3Þ, LRP sld is the fastest in other cases. FWHT is faster than PGCK; PGCK is faster than PWHT. IDA is faster than PGCK for attern size and has similar seed-u as PGCK in other cases. 6.4 Exeriment for JPEG Comressed Images In this exeriment, five different JPEG comression quality levels are used for encoding the original image of the data sets described in Table 8. The JPEG comression quality levels, which are referred to as Jð1Þ, Jð2Þ, Jð3Þ, Jð4Þ, and Jð5Þ, corresond to quality measure Q JPG ¼ 90; 70; 50; 30; 10, resectively The seed-us in execution time using SSD are shown in Fig. 8. The erformance of FFT algorithm is indeendent of Fig. 8. Seed-us in execution time for JPEG comressed images when SSD is used.

12 12 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. X, XXXXXXX 2012 Fig. 9. Seed-us in execution time for images with Gaussian noise when SAD is used. comression quality, while the seed-u of other fast algorithms decreases as the image quality decreases from Jð1Þ to Jð5Þ. We can see that almost all the algorithms outerform the FFT algorithm, with an excetion that PWHT is outerformed by FFT in data sets I4 T4 and I5 T4. PGCK and FWHT are the fastest or close to the fastest in most cases for attern size and LRP can and FWHT are the fastest in most cases for attern size and Exerimental Results When SAD Is Used For the exeriments in this section, seed-us in execution time are evaluated using SAD as the dissimilarity measure. The erturbations corresond to those introduced in Sections Since the FFT aroach cannot be alied when SAD is used, it does not aear in the figures. Fig. 9 shows the results for images with Gaussian noise. IDA is the fastest for attern size 16 16, while LRP can Fig. 10. Seed-us in execution time for blurred images when SAD is used.

13 OUYANG ET AL.: PERFORMANCE EVALUATION OF FULL SEARCH EQUIVALENT PATTERN MATCHING ALGORITHMS 13 Fig. 11. Seed-us in execution time for JPEG comressed images when SAD is used. and LRP sld are the fastest and have similar erformance in other cases. The exerimental results for low ass filter blurred images in Fig. 10 show that the algorithms from the fastest to the slowest for attern size can be ordered as: 1. IDA, 2. LRP sld, 3. LRP can, 4. LRP hier, 5. PGCK and FWHT, 6. PWHT. The order for other attern sizes is 1. LRP sld or LRP can, 2. LRP hier, 3. IDA, 4. PWHT, PGCK, and FWHT, where LRP can is the fastest for data sets I3 T3, I4 T2, I4 T3, and I4 T4 with noise levels Bð1Þ and Bð2Þ, while LRP sld is the fastest for other cases. The exerimental results for JPEG comressed images in Fig. 10 show that IDA is the fastest for attern size IDA is also the fastest for quality levels Jð1Þ and Jð2Þ in all data sets excet that it is outerformed by LRP can for Jð2Þ in I3 T3 and I4 T4. Otherwise, when attern size is larger than and noise level is high, LRP can is the fastest in most cases. In summary, IDA is the fastest in execution time when attern size is 16 16, while LRP sld is the fastest in larger attern sizes for data sets with Gaussian noise and Gaussian low ass filter. LRP can is the fastest for large attern size and high noise level, while IDA is the fastest in other cases for JPEG comressed images. The comarative erformance of LRP sld, LRP can, and IDA using SAD is similar to that using SSD for the three noise tyes. PWHT, PGCK, and FWHT are inefficient in the exeriments when SAD is used and erform better when SSD is used. 6.6 Analysis of the Exerimental Results The exerimental results show some common roerties of the evaluated FS-equivalent algorithms. 1) With attern size fixed, the seed-us over FS achieved by the fast algorithms increase by less than 2 as image size J increases by 4, e.g., from I2 T1 to I3 T1 and from I3 T1 to I4 T1. 2) With image size fixed, the seed-us increase by about 2 to 4 as attern size N increases by 4, e.g., from I2 T1 to I2 T2. 3) The seed-u for IDA, PWHT, PGCK, FWHT, and LRP decreases as the distortion level increases, e.g., from Gð1Þ to Gð4Þ, because the difficulty in efficiently rejecting mismatching windows for these algorithms increases. PGCK is faster than LRP and FWHT for data sets without noise in Figs. 3 and 4 and for data sets having low noise and small attern size in Figs. 8 and 11. LRP and FWHT are faster than PGCK for most cases in Figs. 6, 7, and 8. Analysis of these results is as follows: The comutational efficiency of FS-equivalent attern matching algorithms is deendent on two factors: 1) the rejection ower of lower bounds, which is deendent on the tightness of the lower bound function, and 2) the cost of comuting lower bounds. As a comarison of LRP and WHT: 1) WHT can obtain a tighter lower bound and thus can reject more mismatched candidates comared to LRP when the number of bases u is not a ower of 2; and 2) LRP is more and more efficient than WHT in comuting transformation as u is larger and larger according to Theorem 1. The exeriment in Fig. 12 is used as a examle to comare LRP and other algorithms using WHT. Fig. 12 shows the results on data set I2 T2, i.e., image size and attern size 32 32, with no

14 14 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. X, XXXXXXX 2012 Fig. 12. The ercentage of remaining candidates as a function of the number of bases (left column) and execution time (right column), when no noise is added (uer row) and with Gaussian noise Gð2Þ (bottom row). The exeriment is done for data set I2 T 2 in Table 8 on the PC Env1 in Table 10. noise and Gaussian noise Gð2Þ. The execution time in Fig. 12 is measured on the Env1 in Table 10. When the noise level is low, e.g., the no noise case, uer left subfigure in Fig. 12, small u (less than 4) is sufficient for rejecting a large amount of candidates. When u is small, LRP will not obviously outerform WHT in comuting the lower bound, while the tightness of the lower bound becomes the main factor for comutational efficiency. As a result, WHT algorithms such as PWHT, PGCK, and FWHT outerform LRP when the noise level is low in Figs. 3, 4, 8, and 11, considering that the lower bound for WHT is tighter than LRP when 1 <u<4. PGCK is more efficient than FWHT and PWHT when u<4 in Figs. 3, 4, 8, 11, and 12. On the other hand, as the noise level increases, the rejection ste requires more bases to reject mismatched candidates. Taking the exeriment in Fig. 12 as an examle, to have 2 ercent remaining candidates, only u ¼ 1 basis is required in the no noise case, while about u ¼ 18 bases are required for Gð2Þ noise. As u and the noise level increase, the rejection ability for different algorithms is close, while the time required for the transformation varies and becomes the main factor that influences the comutational efficiency. As illustrated in Theorem 1, the larger u is, the more efficient LRP is comared with PWHT, PGCK, and FWHT in comuting the transformation. Thus, LRP erforms increasingly better comared with PWHT, PGCK, and FWHT as the noise level becomes larger from Gð1Þ to Gð4Þ in Fig. 6, from Bð1Þ to Bð5Þ in Fig. 7, or from Jð1Þ to Jð5Þ in Fig. 8. Similarly, FWHT is faster than PGCK and PWHT in attern matching when the number of comuted bases is large because FWHT comutes the transformation faster. LRP sld erforms better than LRP can as the noise becomes larger from Gð1Þ to Gð4Þ, from Bð1Þ to Bð5Þ, or from Jð1Þ to Jð5Þ. At loo k of the rejection ste, LRP sld and LRP can require 3W and 3h ðk 1Þ Ncan ðkþ additions, resectively, for comuting transformation. If the noise increases, then Ncan ðkþ at loo k increases because the difficulty of rejection ste in rejecting candidate windows increases. As Ncan ðkþ increases, the 3h ðk 1Þ Ncan ðkþ additions required by LRP can increase while the 3W additions required by LRP sld kees unchanged. Hence, LRP sld is increasingly more efficient than LRP can in comuting the transformation and in comuting the attern matching. Assume h ¼ 4, k ¼ 2, and an attern is searched in a image; we have N ¼ 256, J ¼ 65;536, and Ncan ð1þ ¼ W ¼ð þ 1Þ2 ¼ 58;081. Assume Ncan ð2þ ¼ 10 in the first examle, then LRP sld requires 3W ¼ 174;243 additions and LRP can requires 3h ðk 1Þ N ðkþ can ¼ 3 4ð2 1Þ 10 ¼ 120 additions in comuting transformation at loo k ¼ 2. Thus, LRP can is more efficient than LRP sld in comuting transformation for this examle. In this situation, the rejection ste is so efficient that N ð1þ can Nð2Þ can ¼ 58;071 mismatched candidates are eliminated at the first loo of k. Situations similar to this examle usually haen when noise level is small. In the second examle, assume Ncan ð2þ increases from 10 in the first examle to 11W=12 in this examle, then LRP sld requires 3W additions and LRP can requires 3 4 ð2 1Þ 11W=12 ¼ 11W additions for comuting transformation in loo k ¼ 2. Thus, LRP sld is more efficient than LRP can in comuting transformation for this examle. In this situation, the rejection ste is so inefficient that only N ð1þ can Ncan ð2þ ¼ W=12 candidates are eliminated at the first loo of k. Situations similar to this examle usually haen when larger noises are added. LRP sld outerforms LRP hier in all exeriments. LRP hier comutes transformation hierarchically and accesses memory in a juming manner. LRP sld comutes transformation in sliding window manner and accesses memory continuously. Therefore, LRP sld has less cache miss and is faster than LRP hier. 6.7 Termination Strategy Comarison In this exeriment, we comare the roosed termination strategy with the strategy in [28]. PGCK and FWHT are used as the testing algorithms. SSD is used as the dissimilarity measure. The images with JPEG comression quality levels Jð1Þ to Jð5Þ as introduced in Section 6.4 for the data sets in Table 8 are used as the testing data sets. The seed-us are average of the seed-us in the environments introduced in Table 10. The exerimental results in Fig. 13 show that PGCK/FWHT using the roosed termination strategy is faster than PGCK/FWHT using the strategy in [28] when attern size is larger than DISCUSSIONS 7.1 Summary of the Evaluation Results First, we consider execution time as the criterion when SSD is used as the dissimilarity measure.. For data set without noise, PGCK is the fastest.. For data sets with Gaussian noise and blur, FWHT is the fastest when attern is small, image size is large, and distortion level is low, FFT is the fastest when distortion level is high, and LRP sld is the fastest in other cases.. For data sets with JPEG comression, PGCK and FWHT are the fastest when attern size is small or distortion level is small, while LRP can is the fastest in other cases. Second, we consider execution time as the criterion when SAD is used as the dissimilarity measure.. For a data set with Gaussian noise and Gaussian low ass filter, IDA is the fastest in execution time when attern size is 16 16, while LRP sld is the fastest in larger attern sizes.. For JPEG comressed images, LRP can is the fastest for large attern size and high distortion level, while IDA is the fastest in other cases.

15 OUYANG ET AL.: PERFORMANCE EVALUATION OF FULL SEARCH EQUIVALENT PATTERN MATCHING ALGORITHMS 15 Fig. 13. Comarison of termination strategies for JPEG comressed images using SSD. PGCK and FWHT: Results using the roosed strategy; PGCK Hel Or and FWHT Hel Or : results using Hel-Or and Hel-Or s strategy in [28]. Table 11 summarizes the aforementioned results. 7.2 Miscellaneous Proerties of the Evaluated Algorithms Although this aer mainly focuses on the comutational efficiency of the comared algorithms, we list the miscellaneous roerties of these algorithms so that suitable choice of algorithm can be made for secific alications. 1. PWHT, PGCK, and FWHT can be made DC invariant, while FS, IDA, FFT, and LRP currently cannot. DC variation is a secific light change. This roerty of PWHT is used in wide baseline image matching [14]. 2. LRP, PWHT, PGCK, and FWHT can hel deal with multiscale attern matching when the sizes of candidate windows are the integer multiles of the size of the attern or vice versa, while FS, IDA, and FFT currently cannot. PWHT and LRP can comute transformation for windows having different scales at the same time when the attern is scaled by owers of 2. As shown by the exerimental result in [28], when an 8 8 attern is sought within a image at scales 8 8, 16 16, simultaneously, PWHT can comute the transform for window sizes 8 8, 16 16, on the image simultaneously and the remaining window is less than 2 ercent after the first rojection. This roerty is utilized in [7]. 3. Sometimes the attern to be matched may not be rectangular; the method roosed by Ben-Yehuda et al. in [37] hels PWHT, PGCK, and FWHT to deal with this roblem by segmenting the attern into multile dyadic comonents. The irregularity of attern will have no influence on FS, will have small influence on IDA, and will have much influence on algorithms FFT, LRP, PWHT, PGCK, and FWHT that require inut data size to be a 2 b for a ¼ 1; 2;...;b¼ 1; 2;... 8 CONCLUSION In this aer, we have resented execution time evaluation and comutational comlexity analysis of recent FSequivalent algorithms. The algorithms have been comared considering different sizes of images and atterns in the resence of Gaussian noise, image blur, and JPEG comression. Our exerimental results clearly show how the fastest algorithm is different under different conditions. Nonetheless, exerimental evidence also suggests that, overall, LRP may be considered the best erforming algorithm, for it turns out to be the fastest in most cases. This nicely agrees with Theorem 1, which rovides a theory to analyze why LRP is faster than other recent aroaches that use WHT for transform domain attern matching in these cases. Throughout the aer, we have discussed the motivations underinning the relative merits and limits of the considered algorithms under the different working conditions TABLE 11 Best Overall Algorithms for Measured Execution Times for Different Disturbance Factors and Matching Measures

16 16 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. X, XXXXXXX 2012 so as to ossibly insire the develoment of new methods in the active research field of fast full search equivalent attern matching. ACKNOWLEDGMENTS The authors wish to thank Professor Yacov Hel-Or and Professor Hagit Hel-Or for roviding their code imlementing GCK and WHT, Professor Antonio Torralba and CSAIL at the Massachusetts Institute of Technology for the use of the MIT database, Professor Rainer Koster and the Institute for Clinical Radiology and Nuclear Medicine of the Lukas Hosital Neuss for the use of the medical image database, and NASA for the use of the remote sensing image database, and the anonymous reviewers for many constructive suggestions. REFERENCES [1] M.S. Aksoy, O. Torkul, and I.H. Cedimoglu, An Industrial Visual Insection System that Uses Inductive Learning, J. Intelligent Manufacturing, vol. 15, no. 4, , [2] A.W. Fitzgibbon, Y. Wexler, and A. Zisserman, Image-Based Rendering Using Image-Based Priors, Proc. IEEE Ninth Int l Conf. Comuter Vision, vol. 2, , [3] T. Luczak and W. Szankowski, A Subotimal Lossy Data Comression Based on Aroximate Pattern Matching, IEEE Trans. Information Theory, vol. 43, no. 5, , Set [4] R.M. Dufour, E.L. Miller, and N.P. Galatsanos, Temlate Matching Based Object Recognition with Unknown Geometric Parameters, IEEE Trans. Image Processing, vol. 11, no. 12, , Dec [5] W.T. Freeman, T.R. Jones, and E.C. Pasztor, Examle-Based Suer-Resolution, IEEE Comuter Grahics and Alications, vol. 22, no. 2, , Mar./Ar [6] A. Efros and T. Leung, Texture Synthesis by Non-Parametric Samling, Proc. IEEE Seventh Int l Conf. Comuter Vision, , Set [7] C.M. Mak, C.K. Fong, and W.K. Cham, Fast Motion Estimation for H.264/AVC in Walsh Hadamard Domain, IEEE Trans. Circuits Systems for Video Technology, vol. 18, no. 6, , June [8] Y. Moshe and H. Hel-Or, Video Block Motion Estimation Based on Gray-Code Kernels, IEEE Trans. Image Processing, vol. 18, no. 10, , Oct [9] A. Buades, B. Coll, and J.-M. Morel, A Non-Local Algorithm for Image Denoising, Proc. IEEE CS Conf. Comuter Vision and Pattern Recognition, vol. 2, , June [10] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, Image Denoising by Sarse 3D Transform-Domain Collaborative Filtering, IEEE Trans. Image Processing, vol. 16, no. 8, , Aug [11] R. Zhang, W. Ouyang, and W.K. Cham, Image Deblocking Using Dual Adative Fir Wiener Filter in the DCT Transform Domain, Proc. IEEE Int l Conf. Acoustics, Seech, and Signal Processing, , Ar [12] Y. Alon, A. Ferencz, and A. Shashua, Off-Road Path Following Using Region Classification and Geometric Projection Constraints, Proc. IEEE CS Conf. Comuter Vision and Pattern Recognition, vol. 1, , June [13] Y. Shina, J.S. Jua, and E.Y. Kim, Welfare Interface Imlementation Using Multile Facial Features Tracking for the Disabled Peole, Pattern Recognition Letters, vol. 29, no. 13, , Oct [14] Q. Wang and S. You, Real-Time Image Matching Based on Multile View Kernel Projection, Proc. IEEE Conf. Comuter Vision and Pattern Recognition, [15] X. Wu, Temlate-Based Action Recognition: Classifying Hockey Players Movement, MS thesis, Univ. of British Columbia, [16] D.G. Lowe, Distinctive Image Features from Scale-Invarian Keyoints, Int l J. Comuter Vision, vol. 60, no. 2, , [17] K. Mikolajczyk and C. Schmid, A Performance Evaluation of Local Descritors, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 10, , Oct [18] H. Bay, T. Tuytelaars, and L.J.V. Gool, Surf: Seeded u Robust Features, Proc. Euroean Conf. Comuter Vision, vol. 1, , [19] V. Leetit and P. Fua, Keyoint Recognition Using Randomized Trees, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 9, , Set [20] J. Zhang, M. Marszalek, S. Lazebnik, and C. Schmid, Local Features and Kernels for Classification of Texture and Object Categories: A Comrehensive Study, Int l J. Comuter Vision, vol. 73, no. 2, , [21] O. Pele and M. Werman, Robust Real-Time Pattern Matching Using Bayesian Sequential Hyothesis Testing, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 8, , Aug [22] J.P. Lewis, Fast Temlate Matching, Proc. Vision Interface, , May [23] L. Di Stefano and S. Mattoccia, Fast Temlate Matching Using Bounded Partial Correlation, J. Machine Vision and Alications, vol. 13, , [24] L. Di Stefano and S. Mattoccia, A Sufficient Condition Based the Cauchyschwarz Inequality for Efficient Temlate Matching, Proc. Int l Conf. Image Processing, vol. 1, , Set [25] S. Mattoccia, F. Tombari, and L. Di Stefano, Fast Full-Search Equivalent Temlate Matching by Enhanced Bounded Correlation, IEEE Trans. Image Processing, vol. 17, no. 4, , Ar [26] W.H. Pan, S.D. Wei, and S.H. Lai, Efficient NCC-Based Image Matching in Walsh-Hadamard Domain, Proc. 10th Euroean Conf. Comuter Vision: Part III, [27] S.-D. Wei and S.-H. Lai, Fast Temlate Matching Based on Normalized Cross Correlation with Adative Multilevel Winner Udate, IEEE Trans. Image Processing, vol. 17, no. 11, , Nov [28] Y. Hel-Or and H. Hel-Or, Real Time Pattern Matching Using Projection Kernels, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 9, , Set [29] B. Girod, Whats Wrong with Mean-Squared Error?. Chater 15, MIT Press, [30] S. Santini and R. Jain, Similarity Measures, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 9, , Set [31] A.J. Ahumada, Comutational Image Quality Metrics: A Review, Proc. Int l Sym. Soc. Information Dislay, vol. 24, , [32] M.G. Alkhansari, A Fast Globally Otimal Algorithm for Temlate Matching Using Low-Resolution Pruning, IEEE Trans. Image Processing, vol. 10, no. 4, , Ar [33] F. Tombari, S. Mattoccia, and L. Di Stefano, Full Search- Equivalent Pattern Matching with Incremental Dissimilarity Aroximations, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, , Jan [34] G. Ben-Artz, H. Hel-Or, and Y. Hel-Or, The Gray-Code Filter Kernels, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 3, , Mar [35] W. Ouyang and W.K. Cham, Fast Algorithm for Walsh Hadamard Transform on Sliding Windows, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 1, , Jan [36] G.J. VanderBrug and A. Rosenfeld, Two-Stage Temlate Matching, IEEE Trans. Comuters, vol. 26, no. 4, , Ar [37] M. Ben-Yehuda, L. Cadany, and H. Hel-Or, Irregular Pattern Matching Using Projections, Proc. 12th Int l Conf. Image Processing, vol. 2, , [38] A. Goshtasby, 2-D and 3-D Image Registration for Medical, Remote Sensing and Industrial Alications. Wiley, [39] B. Zitova and J. Flusser, Image Registration Methods: A Survey, Image Vision Comuting, vol. 21, no. 11, , [40] W. Krattenthaler, K. Mayer, and M. Zeiler, Point Correlation: A Reduced-Cost Temlate Matching Technique, Proc. IEEE First Int l Conf. Image Processing, vol. 1, , [41] K. Briechle and U.D. Hanebeck, Temlate Matching Using Fast Normalized Cross Correlation, Proc. SPIE Otical Pattern Recognition XII, , 2001.

17 OUYANG ET AL.: PERFORMANCE EVALUATION OF FULL SEARCH EQUIVALENT PATTERN MATCHING ALGORITHMS 17 [42] P.S. Heckbert, Filtering by Reeated Integration, Proc. ACM SIGGRAPH 86, [43] P. Simard, L. Bottou, P. Haffner, and Y. Le Cun, Boxlets: A Fast Convolution Algorithm for Signal Processing and Neural Networks, Proc. Conf. Advances in Neural Information Processing Systems II, vol. 11, , [44] F. Tang, R. Crabb, and H. Tao, Reresenting Images Using Nonorthogonal Haar-Like Bases, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 12, , Dec [45] C.D. Bei and R.M. Gray, An Imrovement of the Minimum Distortion Encoding Algorithm for Vector Quantization, IEEE Trans. Comm., vol. 33, no. 10, , Oct [46] W. Li and E. Salari, Successive Elimination Algorithm for Motion Estimation, IEEE Trans. Image Processing, vol. 4, no. 1, , Jan [47] H.S. Wang and R.M. Mersereau, Fast Algorithms for the Estimation of Motion Vectors, IEEE Trans. Image Processing, vol. 8, no. 3, , Mar [48] O. Pele and M. Werman, Accelerating Pattern Matching or How Much Can You Slide? Proc. Eighth Asian Conf. Comuter Vision, [49] W. Ouyang, R. Zhang, and W.K. Cham, Fast Pattern Matching Using Orthogonal Haar Transform, Proc. IEEE Conf. Comuter Vision and Pattern Recognition, [50] H. Schweitzer, R. Deng, and R.F. Anderson, A Dual Bound Algorithm for Very Fast and Exact Temlate-Matching, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 33, no. 3, , Mar [51] htt://sourceforge.net/rojects/oencvlibrary, [52] M.J. McDonnell, Box-Filtering Techniques, Comuter Grahics Image Processing, vol. 17, , [53] P. Viola and M. Jones, Robust Real-Time Face Detection, Int l J. Comuter Vision, vol. 57, no. 2, , [54] Y.A. Geadah and M.J.G. Corinthios, Natural, Dyadic, and Sequency Order Algorithms and Processors for the Walsh- Hadamard Transform, IEEE Trans. Comuters, vol. 26, no. 5, , May [55] A. Torralba, htt://eole.csail.mit.edu/torralba/images, [56] R Koster, [57] NASA, htt://zulu.ssc.nasa.gov/mrsid, [58] Y. Hel-Or and H. Hel-Or, software.htm, Wanli Ouyang received the BS degree in comuter science from Xiangtan University, Hunan, China, in He received the MS degree in comuter science from the College of Comuter Science and Technology, Beijing University of Technology, Beijing, China. He received the PhD degree in 2011 and is now a ostdoctoral fellow in the Deartment of Electronic Engineering, The Chinese University of Hong Kong. His research interests include image rocessing, comuter vision, and attern recognition. He is a member of the IEEE. Stefano Mattoccia received the Msc degree in electronic engineering and the PhD degree in comuter science from the University of Bologna in 1997 and 2002, resectively, where he is currently an assistant rofessor on the Faculty of Engineering, Deartment of Comuter Science and Systems. His research interests include comuter vision, esecially 3D vision and matching, and embedded comuter architectures for comuter vision. In these fields he is the author of more than 50 refereed articles and two atents. He is a member of the IEEE, the IEEE Comuter Society, IAPR, and the Interest Grou on 3D Rendering, Processing, and Communications of the IEEE MMTC. Luigi Di Stefano received the degree in electronic engineering from the University of Bologna, Italy, in 1989 and the PhD degree in electronic engineering and comuter science from the Deartment of Electronics, Comuter Science, and Systems (DEIS) at the University of Bologna in In 1995, he sent six months at Trinity College Dublin as a ostdoctoral fellow. He is currently an associate rofessor at DEIS. In 2009, he joined the Board of Directors of Datalogic SA as an indeendent director. His research interests include comuter vision, image rocessing, and comuter architecture. He is the author of more than 100 aers and five atents. He is a member of the IEEE, the IEEE Comuter Society, and the IAPR-IC. Wai-Kuen Cham received the graduation degree in electronics from The Chinese University of Hong Kong in He received the MSc and PhD degrees from Loughborough University of Technology, United Kingdom, in 1980 and 1983, resectively. From June 1984 to Aril 1985, he was a senior engineer in Datacraft Hong Kong Limited and a lecturer in the Deartment of Electronic Engineering, Hong Kong Polytechnic (now The Polytechnic University of Hong Kong). Since May 1985, he has been with the Deartment of Electronic Engineering, The Chinese University of Hong Kong. He is a senior member of the IEEE.. For more information on this or any other comuting toic, lease visit our Digital Library at Federico Tombari received the BEng, MEng, and PhD degrees from the University of Bologna in 2003, 2005, and 2009, resectively. Currently, he is a ostdoctoral researcher in Comuter Vision Lab, DEIS (Deartment of Electronics, Comuter Science, and Systems), University of Bologna. His research interests include comuter vision and attern recognition; in articular they include stereo vision and 3D reconstruction, object recognition, algorithms for video-surveillance. He has coauthored more than 40 refereed aers on international conferences and journals and he is a member of the IEEE and IAPR-GIRPR.