On computing global similarity in images

Transcription

1 On coputing global siilarity in iages S Ravela and R Manatha Coputer Science Departent University of Massachusetts, Aherst, MA Eail: Abstract The retrieval of iages based on their visual siilarity to an exaple iage is an iportant and fascinating area of research Here, a ethod to characterize visual appearance for deterining global siilarity in iages is described Iages are filtered with Gaussian derivatives and geoetric features are coputed fro the filtered iages The geoetric features used here are curvature and phase Two iages ay be said to be siilar if they have siilar distributions of such features Global siilarity ay, therefore, be deduced by coparing histogras of these features This allows for rapid retrieval and exaples fro collection of gray-level and tradeark iages are shown 1 Introduction The advent of large ulti-edia collections and digital libraries has led to a need for good search tools to index and retrieve inforation fro the For text available in achine readable for (ASCII) a nuber of good search engines are available However, there are as yet no good tools to retrieve iages The traditional approach to searching and indexing iages using anual annotations is slow, labor intensive and expensive In addition, textual annotations cannot encode all the inforation available in an iage There is thus a need for retrieving iages using their content The indexing and retrieval of iages using their content is a difficult proble A person using an iage retrieval syste usually seeks to find seantically relevant inforation This entails solutions to such hard probles as autoatic segentation, robust feature detection and recognition, all of which are as yet unsolved However, any iage attributes like color, texture, shape and appearance are often directly correlated with the seantics of the proble For exaple, logos or product This aterial is based on work supported in part by the National Science Foundation, Library of Congress and Departent of Coerce under cooperative agreeent nuber EEC , in part by the United States Patent and Tradearks Office and the Defense Advanced Research Projects Agency/ITO under ARPA order nuber D468, issued by ESC/AXS contract nuber F C-0235, in part by the National Science Foundation under grant IRI and in part by NSF Multiedia CDA Any opinions, findings and conclusions or recoendations expressed in this aterial are the author(s) and do not necessarily reflect those of the sponsors packages (eg, a box of Tide) have the sae color wherever they are found The coat of a leopard has a unique texture while Abraha Lincoln s appearance is uniquely defined These iage attributes can often be used to index and retrieve iages A coon odel for retrieval, and one that is adopted here, is that iages in the database are processed and described by a set of feature vectors A priori these vectors are indexed During run-tie, a query is provided in the for of an exaple iage and its features are copared with those stored Iages are then retrieved in the order indicated by the coparison operator In this paper, objects siilar in visual appearance to a given query object are retrieved by coparing with a set of database iages using a characterization of their iage intensity surfaces Arguably an object s visual appearance in an iage is closely related to several factors including, aong others, its three diensional shape, albedo, surface texture and the iaged viewpoint It is non-trivial to separate the different factors constituting an object s appearance For exaple, the face of a person has a unique appearance that cannot just be characterized by the geoetric shape of the coponent parts In this paper a characterization of the shape of the intensity surface of iaged objects is used for retrieval The experients conducted show that retrieved objects have siilar visual appearance, and henceforth an association is ade between appearance and the shape of the intensity surface Specifically, this paper focuses on a representation for coputing global siilarity That is, the task is to find iages that, as a whole, appear visually siilar The utility of global siilarity retrieval is evident, for exaple, in finding siilar scenes or siilar faces in a face database In addition, practical applications such as finding siilar tradearks in a tradeark database significantly benefit fro global siilarity retrieval The iage intensity surface is robustly characterized using features obtained fro responses to ulti-scale Gaussian derivative filters Koenderink [8] and others [3] have argued that the local structure of an iage can be represented by the outputs of a set of Gaussian derivative filters applied to an iage That is, iages are filtered

2 with Gaussian derivatives at several scales and the resulting response vector locally describes the structure of the intensity surface By coputing features derived fro the local response vector and accuulating the over the iage, robust representations appropriate to querying iages as a whole (global siilarity) can be generated One such representation uses histogras of features derived fro the ulti-scale Gaussian derivatives Histogras for a global representation because they capture the distribution of local features (A histogra is one of the siplest ways of estiating a non paraetric distribution) This global representation can be efficiently used for global siilarity retrieval by appearance and retrieval is very fast The choice of features often deterines how well the iage retrieval syste perfors Here the task is to robustly characterize the 3 diensional intensity surface A 3-diensional surface is uniquely deterined if the local curvatures everywhere are known Thus, it is appropriate that one of the features be local curvature The principal curvatures of the intensity surface are invariant to iage plane rotations, onotonic intensity variations and further, their ratios are in principle insensitive to scale variations of the entire iage However, spatial orientation inforation is lost when constructing histogras of curvature (or ratios thereof) alone Therefore we augent the local curvature with local phase, and the representation uses histogras of local curvature and phase Local principal curvatures and phase are coputed at several scales fro responses to ulti-scale Gaussian derivative filters Then histogras of the curvature ratios [7, 1] and phase are generated Thus, the iage is represented by a single vector (ulti-scale histogras) During run-tie the user presents an exaple iage as a query and the query histogras are copared with the ones stored, and the iages are then ranked and displayed in order to the user The rest of the paper is organized as follows Section 2 surveys related work in the literature In section 3, the notion of appearance is developed further and characterized using Gaussian derivative filters and the derived global representation is discussed Coparisons are ade in the context of tradeark retrieval with the traditional oent invariants A discussion and conclusion follows in Section 4 2 RELATED WORK Several authors have tried to characterize the appearance of an object via a description of the intensity surface In the context of object recognition [14] represent the appearance of an object using a paraetric eigen space description This space is constructed by treating the iage as a fixed length vector, and then coputing the principal coponents across the entire database The iages therefore have to be size and intensity noralized, segented and trained Siilarly, using principal coponent representations described in [5] face recognition is perfored in [19] In [17] the traditional eigen representation is augented by using ost discriinant features and is applied to iage retrieval The authors apply eigen representation to retrieval of several classes of objects The issue, however, is that these classes are anually deterined and training ust be perfored on each The approach presented in this paper is different fro all the above because eigen decopositions are not used at all to characterize appearance Further, the ethod presented uses no learning and, does not require constant sized iages It should be noted that although learning significantly helps in such applications as face recognition, however, it ay not be feasible in any instances where sufficient exaples are not available This syste is designed to be applied to a wide class of iages and there is no restriction per se In earlier work we showed that local features coputed using Gaussian derivative filters can be used for local siilarity, ie to retrieve parts of iages [12] Here we argue that global siilarity can be deterined by coputing local features and coparing distributions of these features This technique gives good results, and is reasonably tolerant to view variations Schiele and Crowley [16] used such a technique for recognizing objects using grey-level iages Their technique used the outputs of Gaussian derivatives as local features A ulti-diensional histogra of these local features is then coputed Two iages are considered to be of the sae object if they had siilar histogras The difference between this approach and the one presented by Schiele and Crowley is that here we use 1D histogras (as opposed to ulti-diensional) and further use the principal curvatures as the priary feature The use of Gaussian derivative filters to represent appearance is otivated by their use in describing the spatial structure [8] and its uniqueness in representing the scale space of a function [9, 6, 21, 18] The invariance properties of the principal curvatures are well docuented in [3] In the context of global siilarity retrieval it should be noted that representations using oent invariants have been well studied [13] In these ethods global representation of appearance ay involve coputing a few nubers over the entire iage Two iages are then considered siilar if these nubers are close to each other (say using an L2 nor) We argue that such representations are not able to really capture the appearance of an iage, particularly in the context of tradeark retrieval where oent invariants are widely used In other work [12] we copared oent invariants with the technique presented here and found that oent invariants work best for a single binary shape without holes in it, and, in general, fare worse than the ethod presented here Texture based iage retrieval is also related to the appearance based work presented in this paper Using Wold

3 W h W $ i i 9 9 odeling, in [10] the authors try to classify the entire Brodatz texture and in [4] attept to classify scenes, such as city and country Of particular interest is work by [11] who use Gabor filters to retrieve texture siilar iages The earliest general iage retrieval systes were designed by [2, 15] In [2] the shape queries require prior anual segentation of the database which is undesirable and not practical for ost applications 3 Global representation of appearance Three steps are involved in order to coputing global siilarity First, local derivatives are coputed at several scales Second, derivative responses are cobined to generate local features, naely, the principal curvatures and phase and, their histogras are generated Third, the 1D curvature and phase histogras generated at several scales are atched These steps are described next A Coputing local derivatives: Coputing derivatives using finite differences does not guarantee stability of derivatives In order to copute derivatives stably, the iage ust be regularized, or soothed or band-liited A Gaussian filtered iage obtained by convolving the iage I with a noralized Gaussian is a band-liited function Its high frequency coponents are eliinated and derivatives will be stable In fact, it has been argued by Koenderink and van Doorn [8] and others [3] that the local structure of an iage I at a given scale can be represented by filtering it with Gaussian derivative filters (in the sense of a Taylor expansion), and they ter it the N-jet However, the shape of the soothed intensity surface depends on the scale at which it is observed For exaple, at a sall scale the texture of an ape s coat will be visible At a large enough scale, the ape s coat will appear hoogeneous A description at just one scale is likely to give rise to any accidental is-atches Thus it is desirable to provide a description of the iage over a nuber of scales, that is, a scale space description of the iage It has been shown by several authors [9, 6, 21, 18, 3], that under certain general constraints, the Gaussian filter fors a unique choice for generating scale-space Thus local spatial derivatives are coputed at several scales B Feature Histogras: The noral and tangential curvatures of a 3-D surface (X,Y,Intensity) are defined as [3]: "!#!%$& '! (*),+ #!-! $/!0213 ' ) :7! 0! $;6 ( )!"!! $&!< : Where : and! 78 are the local derivatives of Iage I around point using Gaussian derivative at scale Siilarly ( >=?#=@,!%A=B =, and #!"!%>=?#=@ are the corresponding second derivatives 9 The noral curvature and tangential curvature are then cobined [7] to generate a shape index as follows: C :7 EDF>DGIH The index value C is L when either ) 9KJ678 when 9 and is undefined and are both zero, and is, therefore, not coputed This is interesting because very flat portions of an iage (or ones with constant rap) are eliinated For exaple in Figure 2(iddle-row), the background in ost of these face iages does not contribute to the curvature histogra The curvature index or shape index is rescaled and shifted to the range M N< O"P as is done in [1] A histogra is then coputed of the valid index values over an entire iage The second feature used is phase The phase is siply defined as QR678S TDF>DG + : #!%:U8 78 Note that Q is defined only at those locations where C is and ignored elsewhere As with the curvature index Q is rescaled and shifted to lie between the interval M N< O"P Although the curvature and phase histogras are in principle insensitive to variations in scale, in early experients we found that coputing histogras at ultiple scales draatically iproved the results An explanation for this is that at different scales different local structures are observed and, therefore, ulti-scale histogras are a ore robust representation Consequently, a feature vector is defined for an iage as the vector V2W YXZ[\^]#^_ _#_8ZK[`62a<^Zcbd6^]"U_ _#_AZcbe2a<f where Zcb and Z[ are the curvature and phase histogras respectively We found that using 5 scales gives good results and the scales are O= =#=>g in steps of half an octave C Matching feature histogras: Two feature vectors are copared using noralized cross-covariance defined as W@i VjBk%l W = VjBk%l VjBk%l VKjBk%l VjBk%l V W where )onqp DG V W Retrieval is carried out as follows A query iage is selected and the query histogra V VUr vector W is correlated with the database histogra vectors using the above forula Then the iages are ranked by their correlation score and displayed to the user In this ipleentation, and for evaluation purposes, the ranks are coputed in advance, since every query iage is also a database iage 31 Experients The curvature-phase ethod is tested using two databases The first is a tradeark database of 2048 i-

4 ages obtained fro the US Patent and Tradeark Office (PTO) The iages obtained fro the PTO are large, binary and are converted to gray-level and reduced for the experients The second database is a collection of 1561 assorted gray-level iages This database has digitized iages of cars, stea locootives, diesel locootives, apes, faces, people ebedded in different background(s) and a sall nuber of other iscellaneous objects such as houses These iages were obtained fro the Internet and the Corel photo-cd collection and were taken with several different caeras of unknown paraeters, and under varying uncontrolled lighting and viewing geoetry In the following experients an iage is selected and subitted as a query The objective of this query is stated and the relevant iages are decided in advance Then the retrieval instances are gauged against the stated objective In general, objectives of the for extract iages siilar in appearance to the query will be posed to the retrieval algorith A easure of the perforance of the retrieval engine can be obtained by exaining the recall/precision table for several queries Briefly, recall is the proportion of the relevant aterial actually retrieved and precision is the proportion of retrieved aterial that is relevant [20] It is a standard widely used in the inforation retrieval counity and is one that is adopted here Queries were subitted each to the tradeark and assorted iage collection for the purpose of coputing recall/precision The judgent of relevance is qualitative For each query in both databases the relevant iages were decided in advance These were restricted to 48 The top 48 ranks were then exained to check the proportion of retrieved iages that were relevant All iages not retrieved within 48 were assigned a rank equal to the size of the database That is, they are not considered retrieved These ranks were used to interpolate and extrapolate precision at all recall pointsin the case of assorted iages relevance is easier to deterine and ore siilar for different people However in the tradeark case it can be quite difficult and therefore the recall-precision can be subject to soe error The recall/precision results are suarized in Table 1 and both databases are individually discussed below Figure 1 shows the perforance of the algorith on the tradeark iages Each strip depicts the top 8 retrievals, given the leftost as the query Most of the shapes have roughly the sae structure as the query Note that, outline and solid figures are treated siilarly (see rows one and two in Figure 1) Six queries were subitted for the purpose of coputing recall-precision in Table 1 Experients are also carried out with assorted gray level iages Six queries subitted for recall-precision are shown in Figure 2 The left ost iage in each row is the query and is also the first retrieved The rest fro-left to right are seven retrievals depicted in rank order Note that, flat portions of the background are never considered because the principal curvatures are very close to zero and therefore do not contribute to the final score Thus, for exaple, the flat background in Figure 2(second row) is not used Notice that visually siilar iages are retrieved even when there is soe change in the background (row 1) This is because the doinant object contributes ost to the histogras In using a single scale poorer results are achieved and background affects the results ore significantly The results of these exaples are discussed below, with the precision over all recall points depicted in parentheses For coparison the best text retrieval engines have an average precision of 50%: 1 Find siilar cars(65%) Pictures of cars viewed fro siilar orientations appear in the top ranks because of the contribution of the phase histogra This result also shows that soe background variation can be tolerated The eighth retrieval although a car is a isatch and is not considered 2 Find sae face(874%) and find siilar faces: In the face query the objective is to find the sae face In experients with a University of Bern face database of 300 faces with a 10 relevant faces each, the average precision over all recall points for all 300 queries was 78% It should be noted that the syste presented here works well for faces with the sae representation and paraeters used for all the other databases There is no specific tuning or learning involved to retrieve faces The query find siilar faces resulted in a 100% precision at 48 ranks because there are far ore faces than 48 Therefore, it was not used in the final precision coputation 3 Find dark textured apes (642%) The ape query results in several other light textured apes and country scenes with siilar texture Although these are not is-atches they are not consistent with the intent of the query which is to find dark textured apes 4 Find other patas onkeys (471%) Here there are 16 patas onkeys in all and 9 within a sall view variation However, here the whole iage is being atched so the nuber of relevant patas onkeys is 16 The precision is low because the ethod cannot distinguish between light and dark textures, leading to irrelevant iages Note, that it finds other apes, dark textured ones, but those are deeed irrelevant with respect to the query 5 Given a wall with a Coca Cola logo find other Coca Cola iages (638%) This query clearly depicts the liitation of global atching Although all three database iages that had a certain texture of the wall

5 Figure 1: Tradeark retrieval using Curvature and Phase Figure 2: Iage retrieval using Curvature and Phase (also had Coca Cola logos) were retrieved (100% precision), two other very dissiilar iages with cocacola logos were not 6 Scenes with Bill Clinton (728%) The retrieval in this case results in several isatches However, three of the four are retrieved in succession at the top and the scenes appear visually siilar While the queries presented here are not optial with respect to the design constraints of global siilarity retrieval, they are however, realistic queries that can be posed to the syste Misatches can and do occur The first

6 Table 1: Precision at standard recall points for six Queries Recall Precision(tradeark) % Precision(assorted) % average(tradeark) 611% average(assorted) 663% is the case where the global appearance is very different The Coca Cola retrieval is a good exaple of this Second, isatches can occur at the algorithic level Histogras coarsely represent spatial inforation and therefore will adit iages with non-trivial deforations The recall/precision presented here copares well with text retrieval The tie per retrieval is of the order of illiseconds In on going work we are experienting with a database of iages and the aount of tie taken to retrieve is still less than a second The space required is also a sall fraction of the database These are the priary advantages of global siilarity retrieval That is, to provide a low storage, high speed retrieval with good recall/precision 4 Conclusions and Liitations This paper deonstrates retrieval of siilar objects on the basis of their visual appearance Visual appearance is characterized using filter responses to Gaussian derivatives over scale space In addition, we clai that global representations are better constructed by representing the distribution of robustly coputed local features Currently we are investigating two issues First is to scale the database up to about iages and second is to provide a echanis for cobining global and local siilarity atching in a single fraework 5 Acknowledgeents We would like to thank Yasina Chitti for providing results on the faces We would also like to thank Bruce Croft and CIIR for supporting this work References [1] Chitra Dorai and Anil Jain Cosos - a representation schee for free for surfaces In Proc 5th Intl Conf on Coputer Vision, pages , 1995 [2] Myron Flickner et al Query by iage and video content: The qbic syste IEEE Coputer Magazine, pages 23 30, Sept 1995 [3] L Florack The Syntactical Structure of Scalar Iages PhD thesis, University of Utrecht, Utrecht, Holland, 1993 [4] M M Gorkani and R W Picard Texture orientation for sorting photos at a glance In Proc 12th Int Conf on Pattern Recognition, pages A459 A464, October 1994 [5] M Kirby and L Sirovich Application of the kruhnen-loeve procedure for the characterization of huan faces IEEE Trans Patt Anal and Mach Intel, 12(1): , January 1990 [6] J J Koenderink The structure of iages Biological Cybernetics, 50: , 1984 [7] J J Koenderink and A J Van Doorn Surface shape and curvature scales Iage and Vision Coputing, 10(8), 1992 [8] J J Koenderink and A J van Doorn Representation of local geoetry in the visual syste Biological Cybernetics, 55: , 1987 [9] Tony Lindeberg Scale-Space Theory in Coputer Vision Kluwer Acadeic Publishers, 1994 [10] Fang Liu and Rosalind W Picard Periodicity, directionality, and randoness: Wold features for iage odeling and retrieval IEEE Trans PAMI, 18(7): , July 1996 [11] W Y Ma and B S Manjunath Texture-based pattern retrieval fro iage databases Multiedia Tools and Applications, 2(1):35 51, January 1996 [12] R Manatha, S Ravela, and Y Chitti On coputing local and global siilarity in iages In Proc SPIE conf on Huan and Electronic Iaging III,, 1998 [13] BM Methre, MS Kankanhalli, and WF Lee Shape Measures for Content Based Iage Retrieval: A Coparison Inforation Processing and Manageent, 33, 1997 [14] S K Nayar, H Murase, and S A Nene Paraetric appearance representation In Early Visual Learning Oxford University Press, February 1996 [15] A Pentland, R W Picard, and S Sclaroff Photobook: Tools for content-based anipulation of databases In Proc Storage and Retrieval for Iage and Video Databases II,SPIE, volue 185, pages 34 47, 1994 [16] Bernt Schiele and Jaes L Crowley Object recognition using ultidiensional receptive field histogras In Proc 4th European Conf Coputer Vision, Cabridge, UK, April 1996 [17] D L Swets and J Weng Using discriinant eigen features for retrieval IEEE Trans Patt Anal and Mach Intel, 18: , August 1996 [18] Bart M ter Har Roeny Geoetry Driven Diffusion in Coputer Vision Kluwer Acadeic Publishers, 1994 [19] M Turk and A Pentland Eigenfaces for recognition J of Cognitive NeuroScience, 3:71 86, 1991 [20] C J van Rijsbergen Inforation Retrieval Butterworths, 1979 [21] A P Witkin Scale-space filtering In Proc Intl Joint Conf Art Intell, pages , 1983