A Mult-step Strategy for Shape Smlarty Search In Kamon Image Database Paul W.H. Kwan, Kazuo Torach 2, Kesuke Kameyama 2, Junbn Gao 3, Nobuyuk Otsu 4 School of Mathematcs, Statstcs and Computer Scence, Unversty of New England, Australa 2 Graduate School of Systems and Informaton Engneerng, Unversty of Tsukuba, Japan 3 School of Informaton Technology, Charles Sturt Unversty, Bathurst, Australa 4 Natonal Insttute of Advanced Industral Scence and Technology (AIST), Japan Abstract Emal: kwan@mcs.une.edu.au Smlarty search n mage databases reles on comparng the query wth a set of mages based on features lke shape, colour, texture, and spatal locatons. As the sze of database grows, query processng strateges were proposed to ncrease performance by reducng the number of dstance calculatons. Most strateges are two-step, wth the ntal prune step based on a hgh-dmensonal spatal ndex followed by a refne step performng expensve computaton. They work well wth metrc smlarty models where lower boundng dstance functons exst for prunng. In ths work, smlarty search n a Japanese Kamon Image Database s attempted. The choce of shapes as features s delberate because kamons are n black and whte, and ther meanngs are conveyed by shapes. Further, a three-step prune-flter-refne strategy targetng models wth non-metrc dstance functons s descrbed. Compared to the two-step approach, ths strategy acheves a further reducton n number of dstance calculatons needed but wth close to no change n the precson fgure. Keywords: Smlarty Search, Image Database, Mult-step Strategy, Non-metrc Dstance, Autocorrelaton Introducton As the sze of mage database n practcal applcatons contnues to grow, t has become clear that sequental comparson mght not scale to very large databases. Ths has led researchers to propose query processng strateges that were meant to mprove performance by prunng the search space pror to makng expensve dstance calculatons. In practce, these strateges are largely two-step, wth the ntal prune step based on a hgh-dmensonal spatal ndex followed by a refne step employng expensve computatons []. Most exstng strateges assume a metrc smlarty model for whch a lower boundng dstance functon can be defned for prunng. Recent research on robust mage matchng methods for appearance-based vson has however confrmed that smlarty models behnd human vsual udgment are nherently non-metrc [2]. When applyng these smlarty models, one has to address the problem of non-metrc dstance functons for whch optmal lower bounds mght not exst. In ths work, smlarty search n a Japanese Kamon Image Database that has both cultural and commercal sgnfcances s attempted. The choce of shape as the prmary feature for matchng s delberate because Japanese kamons are n black and whte, and ther meanngs are conveyed by ther shapes. Furthermore, a three-step prune-flter-refne query processng strategy sutable for smlarty models havng nonmetrc dstance functons s descrbed. Compared wth the two-step approach, ths strategy acheves further reducton n number of dstance calculatons but wth close to no change n precson. The rest of ths paper s organzed as follow. Secton 2 descrbes the steps of the proposed mult-step query processng strategy. Secton 3 presents results from experments conducted on a commercal database of 2,000 Japanese kamon mages [3]. Fnally, Secton 4 ends wth concludng remarks. 2 The Mult-step Strategy The proposed strategy conssts of three successve steps, whch are collectvely called the prune-flterrefne (or PFR n short) strategy. In the "prune" step, a mult-dmensonal spatal ndex s used to elmnate mprobable matches by an adustable range threshold. Next, the "flter" step provdes further reducton va a quas lower-boundng dstance that was derved from the non-metrc dstance functon. Fnally, the "refne" stage compares the remanng canddates by a robust matchng method for the fnal smlarty rankng. For ease of explanaton, Fgure llustrates pctorally the proposed mult-step strategy for smlarty search n the Japanese Kamon Image Database.
Keyword Search Name: Sound: Browse Enlarge Shape: Specfcatons End user Smlarty Rankng Smlarty Search Smlarty Search: Name: 土岐桔梗 Sound: き Capton: 五角 Processng Feature Extracton Flter Step Feature Vectors Canddate Lst Browse Result Browse... Database Admnstrator Batch Processng Features Extracton Refne Step Shape Attrbutes Feature Vectors Shape Attrbutes Hgh-dmensonal Spatal Index (Prune Step) Kamon Image DB Fgure : An overvew of the proposed strategy for smlarty search n the Kamon Image Database Because both the feature vectors used n constructng the spatal ndex and the shape attrbutes used by the relaxaton labelng based mage matchng method are output of features extracton, t wll be explaned before descrbng the steps of the proposed strategy. 2. Features Extracton Kwan [4] descrbed a contour-based mage matchng algorthm based on relaxaton labelng. processng s entrely sequental, that s wthout usng any ndexng structure. In ths work, Kwan [4] s appled to the flter-and-refne steps of the proposed strategy, whle the ntal prune step employs a multdmensonal spatal ndex based on the k-d-b tree []. The set of numercal vectors used n constructng the spatal ndex s obtaned by carryng out the Dscrete Fourer Transform (DFT) on both the horzontal and vertcal autocorrelaton plots derved from an mage by treatng these plots as tme sequence data. The algorthm used n constructng these plots s smlar to Nagashma [5]. The dea s llustrated n Fgure 2. Startng from a complete overlap of an mage by a copy of tself, horzontal autocorrelaton s measured by shftng the mage one pxel at a tme from left to rght whle calculatng for each shft the degree of autocorrelatons based on the number of overlappng non-background pxels. The process s repeated untl there s no overlap between the mage and ts copy. A smlar procedure s appled when the vertcal autocorrelaton s measured, albet by shftng from top to bottom nstead. More formally, let x and y be varables that denote the autocorrelaton lengths measured n both the horzontal and the vertcal drectons respectvely. The horzontal and vertcal autocorrelatons can be expressed n terms of x and y as follow: AC AC h v l l Y l x x ( x) = l Y Y = = l l l Y y l y ( y) = = Y Y = f (, Y ) f ( + x, Y ) f (, Y ) f (, Y + y) () (2) Here, f(, ) returns 0 for a background pxel and a for a non-background pxel. Based on the two above equatons, the horzontal and vertcal autocorrelaton plots shown n Fgures 2(c) and (d) can be produced. Takng the horzontal and vertcal autocorrelaton plots as ndvdual tme sequences, the DFT generates for each plot a set of Fourer seres coeffcents n the followng manner. Let a tme sequence x = x ] for t = 0,,, n- be a [ t fnte duraton sgnal. The DFT of x, denoted by = [ f f = n t= ], s gven by: n 0 x e t 2πtf n Here, = s the magnary unt. f = 0,,..., n (3) The nverse DFT of returns the orgnal tme sequence by the followng equaton: n = xt n f = 0 f e 2πtf n t = 0,,..., n (4)
Y Y x l Y y l x a b Fgure 2: (a) and (b) llustrate the horzontal and vertcal autocorrelatons of an mage wth tself c d (c) and (d) are the horzontal and vertcal autocorrelaton plots respectvely For each mage, the two sets of coeffcents are concatenated to form a numercal vector that s ndexed by the spatal ndex. In our experments, the number of coeffcents for each of horzontal and vertcal autocorrelaton s chosen as 5. In ths work, n addton to the numercal vectors of Fourer seres coeffcents, pecewse lne segments that form parts of closed contours of shapes n kamon mages are extracted and approxmated by functons before beng used as matchng attrbutes n the flter-and-refne steps of the proposed multstep strategy [6]. As an llustraton, Fgures 3(a) and (b) present a kamon mage and ts functon approxmated counterpart. Jont ponts detected on the contours are also shown. 2.2 Prune-Flter-Refne Processng 2.2. The Prune Step Smlar to many two-step processng strateges on smlarty search n the lterature, the frst step of the proposed strategy reles on a mult-dmensonal spatal ndex to prune mprobable database obects from further matchng. The feature space conssts of numercal vectors formed from a concatenaton of Fourer seres coeffcents generated by applyng DFT on the horzontal and vertcal autocorrelaton plots of the database mages. In ths work, the k-d-b tree s chosen due to ts search and I/O effcency. When applyng the k-d- B tree n ndexng, the numercal vectors are assumed to be embedded n a d-dmensonal normalzed Eucldean space E d. Let U be the unverse of all such vectors. For any O,O U, ther dssmlarty s defned by a dstance metrc D(O,O ) R + n E d as: D ( ) ( ) 2 d O, O O O +... + ( O O ) 2 = (5) d Fgure 3: A Kamon Image and Its Functon Approxmated Counterpart by Haruk [6] k k where O and O denote the attrbute values of O and O n the k th -dmenson respectvely.
Furthermore, the condton below holds for the dstance functon: ( O O ) d 0 < D,, O, O U and (6) Based on the k-d-b tree based mult-dmensonal ndex, two types of smlarty queres, namely nearest neghbour and range queres are possble. However, n the context of the proposed mult-step strategy, only range queres are relevant because the obectve of the prune step s to reduce the search space to a much smaller canddate set for further processng by the flter-and-refne steps. Gven O q U as the query, by specfyng r R + as the range, such that 0 r d holds, the canddate set (denoted C) that meets the condton below s returned: { O O U and D( O O ) r} C =, (7) p p Lastly, as seen n Fgure, a ponter s appended as the last element of each numercal vector n order to ad retrevng the correspondng shape attrbutes that wll be used n the flter-and-refne steps. 2.2.2 The Flter and Refne Steps The flter-and-refne steps of the proposed strategy are an applcaton of Kwan [7]. Kwan [7] proposed an approxmate query processng approach to address the performance problem of sequental matchng by the method descrbed n Kwan [4]. In smple terms, Kwan [4] matches the shape of obects between the query and each database mage, and ranked them by a heurstc dstance functon. The matchng algorthm was based on probablstc relaxaton labelng, n whch the query mage represents the set of obects whle a database mage the set of labels of the labelng problem. Because the heurstc dstance functon s defned usng probablstc varables whose values are not known untl the teratve probablty updatng of relaxaton labelng has converged, the space of all obects s not formed untl a query s entered. Further, ths space s not metrc n the sense of obeyng the trangle nequalty on dstances, renderng t dffcult to desgnate one database mage as the vantage pont for a possble ndex []. In Kwan [7], the core concept s a lower boundng dstance functon used for flterng out rrelevant database obects from expensve steps of computng actual query dstances [8]. Whereas related work use provable lower bounds due to the metrc nature of ther smlarty models, n research ncludng ours where non-metrc dstance functons are used, a provable lower bound mght not exst. To address ths ssue, a quas lower boundng dstance functon s ntroduced. Ths functon (whch could be many) s defned based on the non-metrc query dstance p q functon and a confdence factor used for scalng. In practce, t s computed at the pont where the ntal state of the relaxaton labelng system s set, but pror to the teratve probablty updatng for actual query dstance calculaton has commenced. Through flterng, t s expected that a sgnfcant reducton n number of expensve query dstance calculatons for the refne step can be obtaned. Further detals on the flter-and-refne steps can be found n Kwan [7], and wll not be repeated here. 3 Expermental Evaluaton 3. Evaluaton Crtera A number of experments were performed on the database of 2,000 kamon mages that came wth Come on Kamon Ver. 2.0 by System Product Corp. of Japan [3]. Three prmary crtera for evaluatng the set of experments are:. The effectveness of Fourer seres coeffcents derved from both the horzontal and vertcal autocorrelaton plots as values of numercal vectors for the mult-dmensonal ndex. 2. The effectveness of the combnaton of quas lower boundng dstance and confdence factor n reducng the number of expensve dstance calculatons n the flter-and-refne steps whle not compromsng the precson. 3. The advantage of the proposed strategy over the 2-step prune-refne approach n terms of reducton n the number of mages matched. 3.2 Expermental Results Frst, results shown n Tables and 2 collectvely addresses crteron () above. Table are dstances measured n the normalzed Eucldean space E d among the group of sx kamon mages shown n Table 2. In ths experment, two vsually smlar mages from 3 of the 9 shape categores are chosen. It s obvous from Table that the dstance s closer between mages that are vsually smlar n Table 2. Though not explctly gven here, ths s supported by the shape of the autocorrelaton plots for these mages, underlyng the effectveness of employng Fourer seres coeffcents n the ndex for prunng. Table : Dstance n E d space between the set of 6 mages shown n Table 2 K K2 K3 K4 K5 K6 K 0 0.64 2.37 2.40 3.09 3.3 K2 0.64 0 2.43 2.38 3.00 3.07 K3 2.37 2.43 0.99 3.29 3.40 K4 2.40 2.38.99 0 3.08 3.09 K5 3.09 3.00 3.29 3.08 0.08 K6 3.3 3.07 3.40 3.09.08 0
Table 2: Pars of smlar mages from 3 of the 9 shape categores n the Japanese Kamon Image Database Round Quadrangle Hexagon K K2 K3 K4 K5 K6 Second, to address evaluaton crteron (2), retreval result of a k-nearest neghbour query (k = 20) by the method of Kwan [4] s shown n Table 3. It wll be used as the benchmark for precson n the rest of our evaluaton. Note that n Table 3, D query refers to the query dstance computed by usng the nonmetrc dstance functon of Kwan [4] whle D ndex s the dstance n the ndexed E d space. Whle they do not exhbt full correlaton, D ndex alone could stll be qute effectve n approxmate smlarty rankng. The second group of experments s meant to verfy how effectve flterng by the quas lower boundng dstance can be acheved based on the number of database mages that have to go through actual dstance calculatons. Two metrcs are defned. The frst of these s a reducton rato defned as follow: Re = (# mages actually match / # total database mages) 00% (8) The second metrc s the precson defned as follow: Pr = (# correct responses n the fltered result / # responses returned) 00% (9) Here, the denomnator of Pr s k, whch s the number of nearest neghbours to retreve. The precson s defned as the percentage of correct responses that are ncluded n the fltered result. For each settng of the confdence factor [7], both reducton rato and precson are computed. Results are summarzed n Fgures 4 and 5. For k = 20, one could observe that the flter-andrefne steps acheved a sgnfcant reducton (more than 70%) n number of actual dstance calculatons whle the precson s mantaned (that s, Pr =.0) at the pont where the confdence factor s deduced automatcally. Compared to ths, both for k = 0 and k = 5, although the reductons are greater, the precsons suffered. Nevertheless, n applcatons where ether the number of smlar mages to return s not overly small or that approxmate results can be accepted, the savngs n computaton by the flter-and-refne steps are hghly sgnfcant. Fnally, to address crteron (3), results of the prevous set of experments and the followngs are combned. Although no automatc way exsts for deducng an optmal value for range r as a functon of the query that could provde maxmum prunng whle mnmzng the number of false drops, t s stll possble to heurstcally deduce an approxmate value for the database of our experment through smulaton. Ths result s summarzed n Table 4. Two addtonal metrcs are defned, denoted by Pr 2 and Re 2. Ther defntons are gven as follows: Re 2 = n / n (0) Here, n and n are the total szes of the database and the number of mages remaned after prunng. Pr 2 = k / k () Here, k s the number of answers n Table 3 that reman after prunng. Here, 0 Pr holds. Table 3: Result of a k-nearest neghbour query (k = 20) by usng Kwan [4], wth the query tself ranked st Rank: 2 3 4 5 6 7 8 9 0 D query:0.0 0.6 0.66 0.7 0.74 0.75 0.77 0.78 0.79 0.80 D ndex:0.0 0.754 0.558.526.272 2.89 2.2.35.492.623 2 3 4 5 6 7 8 9 20 0.8 0.82 0.84 0.86 0.866 0.869 0.872 0.875 0.883 0.886.630.592.937.548.429.347.49.374.339.573
k = 20 c_factor : 0.9536 R.Rato (Matched): 0.2947 k = 0 c_factor : 0.938 R.Rato (Matched): 0.403 k = 5 c_factor : 0.937 R.Rato (Matched): 0.0772 k = 20 c_factor : 0.9536 Precson:.0 k = 0 c_factor : 0.938 Precson: 0.7 k = 5 c_factor : 0.937 R.Rato (Matched): 0.6 Fgure 4: Reducton acheved by flter-and-refne From Table 4, t s reasonable to conclude that the optmal value r should satsfy ths condton, 2.8 < r 2.9 for the query and the database that are used n ths experment. At r = 2.9, prunng s about 8% whle the precson s mantaned at.0. When we assumed that prunng usng a value of r = 2.9 has been done before the flter and refne steps are performed n the proposed mult-step strategy, the reducton n actual dstance calculatons s close to 80% for k = 20. Table 4: Precson and reducton rato by varyng r Range (r) Precson (Pr 2 ) 5.0.0.0 4.0.0 0.99 3.0.0 0.94 2.9.0 0.92 2.8 0.95 0.90 2.6 0.95 0.84 2.4 0.95 0.74 2.2 0.90 0.65 2.0 0.90 0.55.8 0.85 0.42.6 0.75 0.28.4 0.4 0.3 4 Concluson Reducton Rato (Re 2 ) In ths paper, a novel three-step "prune-flterrefne" strategy for shape smlarty search n a Japanese Kamon Image Database s descrbed. Frst, the "prune" step adopts a spatal ndex to elmnate mprobable matches va an adustable dstance threshold. Second, the "flter" step uses a novel quas lower-boundng dstance derved from a non-metrc dstance functon. Thrd, the "refne" stage evaluates the remanng canddates by a robust matchng method for fnal smlarty rankng. Expermental results confrmed that the proposed strategy acheves larger reducton n actual dstance calculatons than two-step approaches wth close to no false drops n the fnal retreval result. Fgure 5: Relaton between Precson and Reducton 5 References [] G.R. Haltason and H. Samet, Propertes of embeddng methods for smlarty search n metrc spaces, IEEE Trans. Patt. Anal. Mach. Intell. Vol. 25, no. 5, pp. 530-549, 2003. [2] D. Jacobs, D. Wenshall and Y. Gdalyahu, Classfcaton wth Nonmetrc Dstances: Image Retreval and Class Representaton, IEEE Trans. Patt. Anal. Mach. Intell. Vol. 22, no. 6, pp. 583-600, 2000. [3] System Product Co. Ltd., Home Page, http://www.e-spc.co.p/kamon2/. [4] P. Kwan, K. Kameyama and K. Torach, On a relaxaton-labelng algorthm for real-tme contour-based mage smlarty retreval, Image and Vson Computng, Vol. 2, no. 3, pp. 285-294, 2003. [5] H. Nagashma, S. Tsubak and J. Nakama, A Classfcaton for Trademark Images Usng the Auto-correlaton Functon Graph Fgure, IEEJ Trans. EIS, Vol. 23, no. 9, pp. 547-554, 2003. [6] R. Haruk, K. Torach and Y. Ohtak, A Mult-Stage Algorthm of Extractng Jont Ponts for Generatng Functon-Fonts, n Proc. 2nd Int l Conf. on Document Analyss and Recognton, Tsukuba, Japan, pp. 3-34, 993. [7] P. Kwan, K. Torach, H. Ktagawa and K. Kameyama, Approxmate Processng for a Content-Based Image Retreval Method, In: V. Malk et al.(eds.): DEA 2003, LNCS 2736, Sprnger-Verlag, pp. 57-526, 2003. [8] P. Cacca, and M. Patella, Searchng n Metrc Spaces wth User-Defned and Approxmate Dstances, ACM Trans. Database Systems, Vol. 27, no. 4, pp. 398-437, 2002.