1. Introduction. Abstract

Transcription

1 Image Retreval Usng a Herarchy of Clusters Danela Stan & Ishwar K. Seth Intellgent Informaton Engneerng Laboratory, Department of Computer Scence & Engneerng, Oaland Unversty, Rochester, Mchgan dstan@oaland.edu & seth@oaland.edu Abstract The goal of ths paper s to descrbe an effcent procedure for color-based mage retreval. The proposed procedure conssts of two stages. Frst, the mage data set s herarchcally decomposed nto dsjont subsets by applyng an adaptaton of the -means clusterng algorthm. Snce Eucldean measure may not effectvely reproduce human percepton of a vsual content, the adaptve algorthm uses a non- Eucldean smlarty metrc and clustrods as cluster prototypes. Second, the derved herarchy s searched by a branch and bound method to facltate rapd calculaton of the -nearest neghbors for retreval n a raned order. The proposed procedure has the advantage of handlng hgh dmensonal data, and dealng wth non-eucldean smlarty metrcs n order to explore the nature of the mage feature vectors. The herarchy also provdes users wth a tool for quc browsng. 1. Introducton The ncreasng rate at whch mages are generated n many applcaton areas, gves rse to the need of mage retreval systems to provde an effectve and effcent access to mage databases, based on ther vsual content. Whle t s perfectly feasble to dentfy a desred mage from a small collecton smply by browsng, technques that are more effectve are needed wth collectons contanng thousands, or mllons of tems. The current mage retreval technques can be classfed accordng to the type and nature of the features used for ndexng and retreval. Keyword ndexng technques manually assgn eywords or classfcaton codes to each mage when t s frst added to the collecton and use these descrptors as retreval eys at search tme. Ther advantages consst of hgh expressve power, possblty to descrbe mage content from the level of prmtve features to the level of abstract features, nvolvng a sgnfcant amount of reasonng about the meanng and purpose of the objects or scenes depcted. On the other hand, manual ndexng presents few drawbacs regardng the usefulness of the assgned eywords and the ndexng tme. Snce the same pcture can have dfferent meanngs for dfferent people, dfferent eywords could be assocated wth the same pcture [1]. When the ndexng tme for every mage taes few mnutes, to ndex a collecton of mllon mages s an ntensve and tme consumng wor. Methods that permt mage searchng based on features automatcally extracted from the mages themselves are referred as content-based mage retreval (CBIR)

2 technques [2]. Color retreval yelds the best results, n that the computer results of color smlarty are smlar to those derved by a human vsual system [3]. The retreval becomes more effcent when the spatal arrangement and couplng of colors over the mage are taen nto account or when one more low-level feature, such as texture or shape, s added to the system. To be along wth the user s percepton of mage chromatc contents, the mages are parttoned nto blocs and a color hstogram s calculated for each bloc. In ths case, smlarty matchng also consders adjacent condtons among blocs wth smlar hstograms. The most notable example of queryng by color s IBM s QBIC system [4] that has been appled successfully n color matchng of tems n electronc mal order catalogues. One drawbac of the current content-based mage retreval systems s ther lmtaton to the low level features even f some researchers have attempted to fll the gap between low-level features and semantc features, by dervng hghlevel semantc concepts (harmony, dsharmony, calmness, exctement) from color arrangements [5]. Another problem wth the exstng mage retreval systems s that these systems do not provde a summary vew of the mages n ther database to ther users. The necessty of a summary vew appears when the user has no specfc query mage at the begnnng of the search process and wants to explore the mage collecton to locate mages of nterest [6]. The ndexng structure s also a bg ssue for CBIR systems. Image features are often very hgh dmensonal or the smlarty metrcs are too complex to have effcent ndexng structures. The exstng mult-dmensonal ndexng technques concentrate only on how to dentfy and mprove ndexng technques that are scalable to hgh dmensonal feature vectors n mage retreval [7]. The other nature of feature vectors n Image Retreval,.e. non-eucldean smlarty measures, cannot be explored usng structures that have been developed based on Eucldean dstance metrcs such as the -d trees, the R-d trees and ts varants. The goal of ths paper s to provde a CBIR system that s scalable to large sze mage collecton and s based on an effectve ndexng module that solves both hgh dmensonalty and non-eucldean nature of some color feature spaces. The module s bult usng an adaptaton of -means clusterng n whch the metrc s a non-eucldean smlarty metrc and the cluster prototype s desgned to summarze the cluster n a manner that s suted for quc human comprehenson of ts components. These prototypes gve the system the capablty of quc browsng through the entre mage collecton. The proposed system also uses a branch and bound tree-search module that appled to the herarchy of the resultant clusters wll facltate rapd calculaton of the nearest neghbors for retreval. The paper s organzed as follows. Secton 2 descrbes the color feature representaton of the mages from the database used n the proposed procedure. Secton 3 explans how the herarchy of smlar groups s bult by the adaptve - means algorthm and Secton 4 descrbes how the search s carred out by the branch and bound algorthm. Secton 5 consders the effectveness of the approach and how the user can browse elegantly through the mage database; these consderatons are expounded wth experments on a database of 2100 mages. The paper concludes wth some fnal comments and a note on future wor.

3 2. Color feature representaton Color s one of the most wdely used features for mage smlarty retreval. Ths s not surprsng gven the facts that color s an easly recognzable element of an mage and the human vsual system s capable of dfferentatng between nfntely large numbers of colors. In ths paper, we use the Color-WISE representaton for mage retreval descrbed n detal n [8]. The representaton s guded prmarly on three factors. Frst, the representaton must be closely related to human vsual percepton snce a user determnes whether a retreval operaton n response to an example query s successful or not. Color-WISE uses the HSV (hue, saturaton, value) color coordnate system that correlates well wth human color percepton and s commonly used by artsts to represent color nformaton present n mages. Second, the representaton must encode the spatal dstrbuton of color n an mage. Because of ths consderaton, Color-WISE system reles on a fxed parttonng scheme. Ths s n contrast wth several proposals n the lterature [9] suggestng color-based segmentaton to characterze the spatal dstrbuton of color nformaton. Although the color-based segmentaton approach provdes a more flexble representaton and hence more powerful queres, we beleve that these advantages are outweghed by the smplcty of the fxed parttonng approach. In the fxed parttonng scheme, each mage s dvded nto M N overlappng blocs as shown. The overlappng blocs allow a certan amount of fuzzy-ness to be ncorporated n the spatal dstrbuton of color nformaton, whch helps n obtanng a better performance. Three separate local hstograms (hue, saturaton and value) for each bloc are computed. The thrd factor consdered by the Color- WISE system s that fact that the representaton should be as compact as possble to mnmze storage and computaton efforts. To obtan a compact representaton, Color-Wse system extracts from each local hstogram the locaton of ts area-pea. Placng a fxed-szed wndow on the hstogram at every possble locaton, the hstogram area fallng wthn the wndow s calculated. The locaton of the wndow yeldng the hghest area determnes the hstogram area-pea. Ths value represents the correspondng hstogram. Thus, a more compact representaton s obtaned and each mage s reduced to 3 M N numbers (3 represents the number of hstograms for HSV). 3. Herarchy of clusters Clusterng s a dscovery process n data mnng. It groups a set of data n a way that maxmzes the smlarty wthn clusters and mnmzes the smlarty between two dfferent clusters. The dscovered clusters can explan the characterstcs of the underlyng data dstrbuton and serve as foundaton for other analyss technques [10]. Clusterng s also useful n mplementng the dvde and conquer strategy to reduce the computatonal complexty of varous decsonmang algorthms n pattern recognton.

4 We use a varaton of -means clusterng to buld a herarchy of clusters. At every level of the herarchy, the varaton of -means clusterng uses a non-eucldean smlarty metrc and the cluster prototype s desgned to summarze the cluster n a manner that s suted for quc human comprehenson of ts components. The resultant clusters are further dvded nto other dsjont sub-clusters performng organzaton of nformaton at several levels, gong for fner and fner dstnctons. The results of ths herarchy decomposton are represented by a tree structure n whch each node of the tree represents a cluster prototype and at the last level, each leaf represents an mage. The herarchy of the cluster prototypes gves the system the capablty of quc browsng through the entre mage collecton. Ths adaptaton of -means algorthm s requred snce the color trplets (hue, saturaton, and value) derved from RGB space by non-lnear transformaton, are not evenly dstrbuted n the HSV space; the representatve of a cluster calculated as a centrod also does not mae much sense n such a space. Instead of usng the Eucldean dstance, we need to defne a measure that s closer to the human percepton n the sense that the dstance between two color trplets s a better approxmaton to the dfference perceved by human. We present below the used smlarty metrc that taes nto account both the perceptual smlarty between the dfferent hstograms bns and the fact that human percepton s more senstve to changes n hue values; we also present how the cluster representatves are calculated and what s the splttng crteron. 3.1 Color smlarty metrc Clusterng methods requre that an ndex of proxmty or assocatons be establshed between pars of patterns [10]. A proxmty ndex s ether a smlarty or dssmlarty. The more two mages resemble each other, the larger a smlarty ndex and the smaller a dssmlarty ndex wll be. Snce our retreval system s desgned to retreve the most smlar mages wth a query mage, the proxmty ndex wll be defned wth respect to smlarty. Dfferent smlarty measures have been suggested n the lterature to compare mages [3, 11]. We are usng n our clusterng algorthm the smlarty measure that, besdes the perceptual smlarty between dfferent bns of a color hstogram, recognzes the fact that human percepton s more senstve to changes n hue values [8]. It also recognzes that human percepton s not proportonally senstve to changes n hue value. Let q and t represent the bloc number n a query Q and an mage T, respectvely. Let ( h q, s ) q, v q and ( h ) t, s t, v t represent the domnant huesaturaton par of the selected bloc n the query mage and n the mage T, respectvely. The bloc smlarty s defned by the followng relatonshp:

5 S ( q, t ) 1+ a D h ( h, h ) + b* D ( s, s ) + c D ( v, v ) q t 1 s q t v q t (1) Here D, D and h s D v represent the functons that measure smlarty n hue, saturaton and value. The constants a, b and c defne the relatve mportance of hue, saturaton and value n smlarty components. Snce human percepton s more senstve to hue, a hgher value s assgned to a than to b. The followng functon was used to calculate D : h D h ( h, h ) q t 1 cos h q h 2 t 2π 256 (2) The functon Dh explctly taes nto account the fact that hue s measured as an angle. Through emprcal evaluatons, a value of equal to two provdes a good non-lnearty n the smlarty measure to approxmate the subjectve judgment of the hue smlarty. The saturaton smlarty s calculated by: D ( s, s ) s q t s q s 256 t (3) The value smlarty s calculated by usng the same formula as for saturaton smlarty. Usng the smlartes between the correspondng blocs from the query Q and mage T, the smlarty between a query and an mage s calculated by the followng expresson: S( Q, T ) M N m S 1 M N 1 ( q, t ) The quantty m n the above expresson represents the masng bt for bloc and M N stands for the number of blocs Cluster prototypes The cluster prototypes are desgned to summarze the clusters n a manner that s suted for quc human comprehenson of ts components. They wll nform the user about the approxmate regon n whch clusters and ther descendants are m (4)

6 found. By buldng the herarchcal tree havng the cluster prototypes as nteror nodes, the system wll allow users to browse the mage collecton at dfferent levels of detals. We defne the cluster prototype to be the most smlar mage to the other mages from the correspondng cluster; n another words, the cluster representatve s the clustrod pont n the feature space,.e., the pont n the cluster that maxmzes the sum of the squares of the smlarty values to the other ponts of the cluster. If C s a cluster, ts clustrod M s expressed as: 2 M arg max S ( I, J ) I C J C S, s ther smlarty value. We use arg to denote that the clustrod s the argument (mage) for whch the maxmum of the sums s obtaned. Here I and J stand for any two mages from the cluster C and ( I J ) 3.3. Splttng crteron To buld a partton for a specfed number of clusters K, a splttng crteron s necessary to be defned. Snce the herarchy ams to support smlarty searches, we would le nearby feature vectors to be collected n the same or nearby nodes. Thus, the splttng crteron n our algorthm wll try to fnd an optmal partton (5) J e K 2 ( K ) w S ( I M ), 1 I C where, w 1 n that s defned as one that maxmzes the crteron sum-of-squared-error functon: M and I stand for the clustrod and any mage from cluster C, respectvely; 2 S ( I, M ) represents the squared of the smlarty value between I and M, and n represents the number of elements of cluster C. The reason of maxmzng the crteron functon comes from the fact that the proxmty ndex measures the smlarty; that s, the larger a smlarty ndex value s, the more two mages resemble one another. Once the partton s obtaned, n order to valdate the clusters,.e. whether or not the samples form one more cluster, several steps are nvolved. Frst, we defne the null hypothess and the alternatve hypothess as follows: H 0 : there are exactly K clusters for the n samples, and H A : the samples form one more cluster. Accordng to the Neyman-Pearson paradgm [12], a decson as to whether or not to reject H 0 n favor of H A s made based on a statstcs T ( n). The statstc s nothng else than the cluster valdty ndex that s senstve to the structure n the data: (6)

7 T ( n) J J e e ( K ) ( K +1) (7) To obtan an approxmate crtcal value for the statstc, that s the ndex s large enough to be unusual, we use a threshold that taes nto account that, for large n, J e ( K) and J e ( K +1) follow a normal dstrbuton. Followng these consderatons, we consder the threshold τ defned n [13] as: 2 τ 1 π d α π d n d (8) The rejecton regon for the null hypothess at the p-percent sgnfcance level s: T ( n) < τ (9) The parameter α n (8) s determned from the probablty p that the null hypothess H 0 s rejected when t s true and d s the sample sze. The last nequalty provdes us wth a test for decdng whether the splttng of a cluster s justfed. 4. The browsng and search strategy The sgnfcant feature of our scheme s the possblty of quc browsng of the mage set when no query mage s specfed. The user can browse frst the hghest level of the tree representng the herarchy and get summary vews of the entre mage collecton n the form of the prototypes of the clusters at that level. By traversng down the tree, the user gets fner and fner detals from one level to another. Usng an analogy wth the vew layers defned usng a herarchy of selforganzaton maps [6], we can consder the frst level of the tree as a global vew level of the entre mage collecton, the ntermedate levels as regonal levels and the last layer of the tree as a local layer gvng the most detaled summary vews for the mages. Each node from the last layer ponts to a group of smlar mages named mage layer. When a query mage s present, the second phase of our algorthm s nvolved. The search strategy mples a branch and bound algorthm n order to facltate rapd calculaton of the -nearest neghbors for retreval. We use the method defned n [14] whch tests the nodes of the tree by two smple stoppng rules that elmnates the necessty of calculatng many dstances. The frst rule s meant to elmnate from consderaton the node and ts correspondng group of samples f the dstance between the query and the node (clustrod) s greater than the sum between the current dstance to the nearest neghbor and the farthest dstance from the centrod to any sample from the cluster. The second rule reduces the number of calculatons

8 of dstances between the query and the samples of the node that survved to rule 1. If the dstance from the query to the clustrod s greater than the sum between the current dstance to the nearest neghbor and the dstance from the clustrod to a sample, do not calculate the dstance between the sample and the query anymore. To perform smlarty search, the color representaton of the query mage s frst matched at the frst layer to determne the most smlar cluster prototypes (nodes) that should be searched further. We elmnate from consderatons each node from frst layer for whch rule 1 s satsfed. The matchng s then repeated for the chldren of one of the nodes from the prevous layer that survved to rule 1, and so on untl the last layer s reached, whch brngs out a group of mages that can be the most smlar to the query mage. We do not need to compare each one of these mages wth the query mage snce rule 2 flters out the mages that not satsfy t. For the mages that fnally survve, the dstances to the query mage are calculated and ordered to fnd the current nearest neghbors. Then the algorthm s appled for the next node that was carred on after applyng rule 1 and the table of the current nearest neghbors s updated as needed. 5. Expermental Results We evaluate our algorthm for browsng and retreval on an mage database of 2100 mages. The color vector representaton of each mage has 3*8*8 elements snce each mage s parttoned nto 8*8 overlappng blocs and the mage color content s characterzed by three components: hue, saturaton and ntensty. To perform color-based smlarty retreval, the values of the constants ( a, b and c ) n formula (1) are expermentally chosen as beng 2.5, 0.5 and 0, respectvely. We rescale hue and saturaton to values between 0 and 255. In order to obtan the frst level (global layer) of the herarchy, we apply -means algorthm for 2, 3 and at each consequent, the cluster valdty s checed, to ensure that the number of elements n every cluster s a moderate one and the sum-of-squared-error crteron to be satsfed. Comparng the values of the test statstc (7) and the values of the threshold (8) wth respect to nequalty (9), the possble number of clusters for dfferent small values of the sgnfcance level s obtaned. Snce the value of the statstc for K 31 s greater than the threshold for consecutve small values of p, we choose the value of K to be 30. Further, we splt the nodes havng at least 30 mages (at least 2% out of the data set) by applyng -means algorthm agan and so, a lower level (regonal layer) of the herarchy s obtaned. The mnmum number of elements n every cluster to go further wth splttng s decded as a compromse between the sze of the termnal nodes and the number of nodes n the tree. Fewer elements n the fnal groups produce fewer dstance computatons n the retreval stage, but larger number of dstance computaton n the search stage. We end up wth a search tree havng 81 nodes, 4 levels and an average of 40 mages per termnal node. Fg. 2 shows a retreval result for browsng mode. The user browses the frst level (global layer) of the tree and hypothetcally speang, the user decdes to loo for mages smlar wth the prototype of cluster 9. The mage wll be updated wth the mages (found n herarchcal clusterng process) that are close to the centers of

9 clusters at the next layer. Assumng that the user decdes to see mages smlar wth the frst prototype of the second layer, the thrd layer (mage layer) wll dsplay the group of mages smlar wth the prevous chosen prototype. Global layer Local layer Image layer Fgure 2. Fg. 3 shows a retreval result for search mode. The mage query s n the top left of the mage. The user wants to retreve the most three smlar mages wth the mage query. Applyng the proposed scheme, the followng nodes are reached n order to fnd the 3-nearest neghbors: node 5 at frst level, node 28 at second layer, 76 at the fnal level. The nearest neghbors are pced up from the group of mages ponted by node 76. Query mage Fgure 3. For more results on color smlarty retreval vst our home page at 6. Conclusons and future wor Ths paper presented an effcent method for mage retreval. Snce the proposed procedure organzes the color nformaton as a herarchy of dfferent clusters, the user s provded wth summary vews of the entre mage collecton at dfferent level of detals. Fast calculaton of the -nearest neghbors s possble by usng a branch and bound algorthm as a search strategy. As future wor, we want to experment our system wth semantc features n addton to the low level ones. Snce browsng computerzed nformaton has a socal dmenson, we wll also

10 develop an nterface for better vsualzaton of the nformaton patterns beng browsed and more effectve means of communcatng the browsng process. References 1. Enser PGB. Pctoral nformaton retreval. Journal of Documentaton, 51(2), pp , Lew MS, Hujsmans DP, Denteneer D. Content based mage retreval: Optmal eys, Texture, Projectons or Templates. Image Databases and Mult-Meda Search, 1997; 8; Faloutsos C, Equtz W, Flcner M. Effcent and Effectve Queryng by Image Content. Journal of Intellgent Informaton Systems 3, 1994; Nblac W, Barber R, Equtz W. et al. The QBIC project: queryng mages by content usng color, texture, shape. Proc. SPIE: Storage and Retreval for Images and Vdeo Databases 1993; 1908; Corrdon J, Delbmbo A, Pala P. Image retreval by color semantcs. ACM Multmeda System Journal, Seth IK, Coman I. Image retreval usng herarchcal self-organzng feature maps. Pattern Recognton Letters 20, 1999; Ru Y, Huang TS, Chang SF. Image Retreval: Past, Present, and Future. Journal of Vsual Communcaton and Image Representaton, 1999; 10; Seth IK, Coman I, Day B et al. Color-WISE: A system for mage smlarty retreval usng color. Proc. SPIE: Storage and Retreval for Image and Vdeo Databases, 1998; 3312; Smth JR and Chang SF. Tools and Technques for Color Image Retreval. Proceedngs of the SPIE: Storage and Retreval for Image and Vdeo Databases IV, 1996; 2670; Jan AK, Dubes RC. Algorthms for Clusterng Data. Prentces Hall Advanced Reference Seres, Swan MJ, Ballard DH. Color Indexng. Internatonal Journal of Computer Vson, 1991; 7(1); Rce JA. Mathematcal Statstcs and Data Analyss. Duxbury Press, Duda RO, Hart PE. Pattern classfcaton and scene analyss. John Wley & Sons, Inc., Fuanaga K, Narendra PM. A branch and bound algorthm for computng -nearest neghbors. IEEE Transactons on Computers, 1975; C -24; Danela Stan s a graduate student n Computer Scence and Engneerng at Oaland Unversty. She receved her M.A. n Computer Scence n 1999 from Wayne State Unversty and B.S. n Mathematcs n 1993 from Unversty of Bucharest, Romana. She s currently nvolved n the area of content -based mage retreval and data mnng. Ishwar K. Seth s currently Professor and Char of Computer Scence and Engneerng at Oaland Unversty. Hs research nterests nclude pattern recognton, data mnng, and multmeda nformaton processng and ndexng. Professor Seth serves on the edtoral

11 boards of several nternatonal journals ncludng IEEE Trans. Pattern Analyss and Machne Intellgence and IEEE Multmeda.