Vsual Thesaurus for Color Image Retreval usng Self-Organzng Maps Chrstopher C. Yang and Mlo K. Yp Department of System Engneerng and Engneerng Management The Chnese Unversty of Hong Kong, Hong Kong ABSTRACT The technque of searchng n content-based mage retreval has been actvely studed n recent years. However, ths technque cannot gve user an overvew of the database. In ths paper, we propose a browsng technque usng Kohonen s Self-Organzng Map to retreve general color mage database effectvely. Both chromatc and textural feature of mages are analyzed to represent the content of mages. Keywords: Vsual database, Content-based Retreval, Image Browsng, Self-Organzng Map, Color Hstogram, Gabor Flter 1. INTRODUCTION Recently, many approaches for content-based mage retreval (CBIR) have been developed; for example, IBM s Query by Image Content (QBIC), and UC Santa Barbara s Alexandra Dgtal Lbrary (ADL) proect. Most of them use the technque of query-by-example, a searchng approach to fnd mages that are smlar to the gven example mage. Smlarty between mages s quantfed n terms of some global or local features, such as colour, texture, shape, pattern, or specfc spatal area of the above. The feature selecton and matchng technques are vtal to searchng thus much research worked on these technques. However, there are some serous lmtatons on usng searchng solely. For example, query-by-example technques often generate results of a relatvely small number of mages, whch may not be nterested by the user; as a result, the user may not be able to make further queryng. In contrast, browsng envronment offer an alternate approach to the problem, but t has receved relatvely lttle attenton. Accordng to Craverl et al. [1], browsng s a technque, or a process, that the users vew nformaton rapdly and decde whether the content s relevant to ther needs. The user can get an overvew or summary of content quckly and then focus on partcular sectons of nterest. Craver et al. modeled a general process of mage browsng. The frst step s to extract the relatonshp among the mages. The second step s to fnd representatve nstances of mages durng browsng. The fnal step s to vsualze the mages wth ntutve presentaton of the relatonshp among mages. Recent research n mages browsng uses dfferent data structure to organze mages n a meanngful way to present the relatonshp among mages. Craver et al. descrbed a technque based on multple space-fllng curves. Chen et al. [] ntated a new approach called actve browsng. The research made use of ther prevous work on smlarty pyramds. Those technques provde an attractve feature that the process of nserton of new mages s computatonally fast; complete re-ndexng s not necessary. We nvestgate the unsupervsed artfcal network, Kohonen s self- organzng map (SOM) [3, 4], for mage browsng. SOM reduces hgh-dmensonal nput sgnal space nto low-dmensonal space whle the map preserves the relatonshp among the nput sgnals. Zhang et al. [5] addressed the mportance of developng effectve ndexng scheme for mage database searchng. They studed an ndexng scheme that apples SOM approach. Three texture features MRSAR, coarseness and gray hstogram were used for ndexng monochrome textural mages and color hstogram was used for ndexng general color mages. They developed a set of herarchcal self-organzng maps (HSOMs) to construct an ndex tree, whch provdes a space for searchng. The evaluaton of performance s calculated by retreval rate. Han et al. [6] uses SOM for mage retreval. Ther system used SOM for generatng maps for human browsng but not for machne ndexng. However, ther system s restrcted to handle mages of obects, from whch shape features such as roundness, rectangularty, elptcty, eccentrcty and bendng energy are extracted, where only the luance nformaton s consdered. Besdes, they used a set of randomly selected mages to label the regons n the map, whch s not necessary representng the most sgnfcant features of the node. In ths paper, we present an approach to provde a mage-browsng nterface for human to retreve general colour mage database usng SOM. We have tested advanced chromatc and texture features to generate maps and constructed three methods to label the resultant maps.. ALGORITHM Our proposed algorthm can be dvded nto two man sectons: mage representaton and self-organzng map generaton. The former secton transforms the mage database nto representaton, whch can be used for evaluatng smlarty among the mages. The latter secton generates SOM usng the above nformaton. Image Representaton There are a number of colour coordnate systems for mages. Tradtonally colour mages are represented n RGB format when processed n computer systems. However, ths colour coordnate system does not match wth human percepton of colour. Colour percepton research generally beleve that human recognze colour by hue and saturaton. Snce ths research s amed at provdng an nterface for human, we have adopted LHS color coordnate system that corresponds to human color percepton. The process of the proposed algorthm for mage representaton s shown as follows.
Raw Image Fgure 1. The process for mage representaton Feature Extracton: In order to extract the content of mages n the database, we need to convert raw data of mages nto some presentable and comparable features. There are varous knds of features of mages, for example, colour, texture, shape and pattern. In ths thess, we have chosen colour and texture features because the type of database s general colour mages, shape and pattern features are not sutable. For the colour feature, we used the hue and saturaton component n LHS color coordnate system. Although all three components are necessary to represent a colour, luance of LHS space s omtted n our colour feature. It s because mages captured n the same scene at dfferent angles wll affect ts hue and saturaton dstrbuton only slghtly but luance dstrbuton wll be easly affected due to lghtng of the envronment. Besdes, the texture feature s analyzed by the luance component only. Therefore, the process s denoted as: For each mage, Feature Extracton RGB Extracton LHS Transformaton Feature Analyss Chromatc Analyss Textural Analyss Image Representaton 1. Extract RGB values of each pxel n the mage. Convert the RGB values nto LHS color coordnate system 3. Perform textural analyss wth the L value 4. Perform chromatc analyss wth the H and S values 5. Combne both analyss result to represent the mage feature RGB L HS Textural Analyss Chromatc Analyss Image Feature The hue component vares from 0 to 360 degrees and the saturaton component vares from 0 to 1. In order to buld the hstogram, the hue and saturaton values are quantzed nto several levels. We have chosen 10 levels for each component, so that a 10 by 10 two-dmensonal hstorgram h[10,10] for each mage s bult n our system. A Pxel wth 0 h < 36 and 0 s < 1 s sorted n the frst bn h[1,1] and so on. Textural Analyss: There s a long hstory for texture analyss research. Accordng to Manunath et al. [8], texture analyss algorthms uses varous technques, ncludng random feld model and multresoluton flterng technques such as wavelet transform. In the research of Manunath et al., the experment results of a number of texture features are compared. The texture features ncludes the conventonal pyramd-structured wavelet transform (PWT) features, tree-structured wavelet transform (TWT) features, and the multresoluton smultaneous autoregressve model (MR-SAR) features and the Gabor wavelet features. The expermental results show that Gabor features gve the best performance. In another recent research carred out by Chen et al. [9], they compared four flterng technques ncludng Fourer transform, spatal flter, Gabor flter and wavelet transform for texture dscraton. The expermental results also show that Gabor flter generally gves the best performance. However, Chen et al. stated that the executon tme of Gabor flter s the longest out of the four features. Scne texture analyss are generated once only n ths knd of mage retreval applcaton, the dsadvantage of longer executon tme of generatng Gabor features s not crtcal. Therefore, we adopt Gabor flter for textural analyss n ths research due to ts best performance. In the followng, we wll revew the Manunath et al. s method of usng Gabor flter for texture feature extracton. A two-dmensonal Gabor functon can be wrtten as: Fgure. The nformaton flow of the process for mage representaton x y g ( 1 1 x, y) = exp + + πwx πσ xσ y σ x σ y () Chromatc Analyss: We adopt the concept of colour ndexng method proposed by Swan et al. [7]. Ths method uses hstogram of ntensty values to represent the colour dstrbuton. Ths knd of hstogram captures the global chromatc nformaton of an mage. The advantage of ths method s that the hstogram s nvarant under translaton and rotaton about the vew axs. Despte changes n vew, change n scale, and occluson, the hstogram only changes slghtly. As mentoned before, we use H and S values for chromatc analyss. The algorthm s to buld a two-dmensonal hstogram wth one axs for hue and another for saturaton. The defnton of HS hstogram s: Hstogram h, s [ h, s] = N Pr( H = h, S = s) where H and S s the hue and saturaton channels, N s the number of pxels n the mage. (1) Then a self-smlar flter dctonary can be obtaned as a mother Gabor Wavelet G(x, y) by approprate dlatons and rotatons of Eq. () as: G = a where S m G θ, θ ) ( x y h = heght of hsde = (h -1) mage, w /, wsde θ = ( x hsde)cos( + ( y wsde)sn( x θ = ( x hsde)sn( + ( y wsde)cos( y a > 1, m,n are ntegers = wdth of mage, = (w - Gven an mage wth luance ( x y) 1) / I,, a Gabor decomposton can be obtaned by multplyng the luance by the magntude of the Gabor Wavelet: (3)
W (4) The mean and standard devaton of the magntude of the transform coeffcent are used to represent the texture feature for classfcaton and retreval purpose: The Gabor feature vector s constructed by usng σ as feature components: f ( x, y) = I( x, y) G G r + ( x y) W, dxdy µ = h w σ [ µ σ µ σ µ σ ] = 00 00 01 01... ( S 1)( K 1) ( S 1)( K 1) ( W ( x, y) ( x, y ) dxdy = µ (5) (6) µ and where S s number of scales and K s number of orentaton. In the followng experment, we use S=3 and K=4. Self-Organzng Map After generatng mage representaton, we need to vsualze the representaton of all mages n database by categorzaton. Categorzaton s a process whch nvolves groupng tems of smlar nature. There are several clusterng algorthms avalable, such as K-means algorthm, sngle-lnk clusterng and complete-lnk clusterng (based on mum-spannng tree). A newer connectonst approach called Self-Organzng Map (SOM) s developed by Kohonen n 1981. SOM s supposed to reduce the hgh dmensonal nput space nto lower dmensonal space, usually two-dmensonal. The resultant feature mappng s a non-lnear proecton from the nput space. The topologcally close nodes n the map are senstve to the nputs that are smlar. Structure: Accordng to Kohonen, the structure of SOM s typcally an array of nodes (neurons) arranged n two-dmensonal space. The nodes can be arranged n varous ways, for example, rectangular, hexagonal, or even rregular. Rectangular arrangement s used n ths research, as t s most sutable for vsualzaton of a set of mages. The structure of SOM used n ths research s llustrated as follows. Input Pattern Input Layer Kohonen Layer Fgure 3. The structure of D SOM (7) The nput patterns are the feature vectors of mages n the database. Each node n nput layer (component of feature vector) s fully connected to the Kohonen Layer. Tranng: The notatons and tranng algorthm of SOM used n ths research s as follows. I Set of all mages n database x Feature vector of mage I M Set of all nodes n Kohonen Layer w (t) Weght vector of node M at tme t R Set of mages that are mapped to node M v Label mage of node M 1. Randomze w ( 0), M. For each teraton,.1 Shuffle the presentaton order of mages. For each mage n the orderng lst,..1 Fnd a wnnng node c.. Update the wnnng node c and ts neghbors.3 Contnue tranng wth decreased learnng rate and reduced area of neghborhood functon 3. For each I, put nto R Frstly, the weghts of all nodes are assgned wth a vector of random values. If there are dentcal weghts of node, there may be a case that there are two wnnng nodes. To prevent ths stuaton, the random process s used. In each teraton, all mages wll be presented to the system. However, f the sequence of presentaton of nodes s fxed, the map wll be nfluence heavly by the frst few mages. Therefore, the order of presentaton s shuffled n the begnnng of each teraton to mze ths effect. When an mage s presented to the system, the feature vector x s compared wth all weght vector w (t) of all nodes n Kohonen Layer. The most smlar node, called the wnnng node, can be found. Smlarty s defned by Eucldean dstance. Denotng the wnnng node as c (t), fndng a wnnng node s formulated as: c ( t) = arg x w M () t Then the wnnng node c (t) and ts neghbors are updated. The obectve of updatng s to make the weghts of the wnnng node and ts neghbors become more smlar to that nput vector. The formula and the llustraton of the vectors are shown as follow. w ( t + 1) = w () t + ηh () t [ x w () t ] c where 0 < η < 1 s the learnng factor and h c () t s the neghborhood functon. For convergence t s necessary η h c () t 0 when t. In ths research, the learnng factor η s a lnearly decreasng functon, of whch the ntal and fnal values are specfed explctly. In ths research, we use a constant weghtng for the neghborhood functon. The area of t dshed wth respect to tme. (8) (9)
h c () t 1 = 0 f N c ( t) f N ( t) c (10) where N c (t) refers to a neghborhood set of array ponts around node c at tme t. It has been depcted as: y = k k R R x arg x y f R > 0 v = R undefned otherwse (14) 3. Label by result-smlarty: For each node n the map, fnd an mage R such that the Eucldean dstance between w and x s mal. c N c (t 1 ) N c (t ) N c (t 3 ) v arg x w f R > 0 R undefned otherwse = 3. Illustraton (15) Fgure 4. Neghborhood set (t 1 < t < t 3 ) We defned the length of the sde of square shape neghborhood set as: length (,3) () t = max max( SOM, SOM )( 1 t) wdth heght (11) At the begnnng (t = 0), the neghborhood set covers at most the whole SOM. Its sze dshed wth respect to t untl the length equals to 3. After the network s traned through a number of teratons, a converged SOM s generated. The resultant SOM contans weghts whch represent the dstrbuton of nput space. Fnally, feature vectors of mages are mapped to the nodes by the mum Eucldean dstance, formulated as: k = arg x w R = M { k = } () t where k denotes the node whch mage s mapped to, and R denotes the set of mages mapped to node. (1) Labelng: After mappng the feature vectors, nodes n the SOM may contan zero or many mages. To vsualze the SOM, we need to label each node by a representatve mage. Three algorthms of labelng are tested n ths research as follows. 1. Label by smlarty: For each node n the map, fnd an mage I as label such that the Eucldean dstance between x and w s mal. v = arg x w I (13). Label by result-mean: For each node n the map, calculate the mean y of weght vectors of mages n R. Next, fnd the mage wth mal Eucldean dstance to the mean as label. We mplemented the system n Java (JDK 1.1.8) and run t on a Pentum II-33 PC. There are two modules n the system. The frst one s feature module, whch extract chromatc and textural feature of mages n the database. It also ncludes user-nterface for retrevng feature nformaton of each mage. A smple query-by-example s mplemented n ths module for testng the performance of the features. The second one s SOM module whch trans SOMs usng feature databases. It contans a user-nterface to vsualze the labeled result for browsng the mages. In the followng, we wll evaluate SOMs usng dfferent mage features and labelng method. Feature dfferences We have used a database wth 500 colour mages (640x480x4bts) n the followng experments. Fgure 5 (a-c) shows a 10x10 SOM traned wth dfferent features. Three SOMs are traned for 50 teratons 1. Learnng rate begns at 0.5 and ends at 0.01. Resultng maps are labeled by result-smlarty. The number n left-lower corner n each node ndcates the number of mages n the node ( R ). After selectng a node n the map by user, the rght colu dsplays the mages (R ) mapped to the selected node. All the resultng maps can provde an overvew of the database. And more mportantly, they contan rch nformaton. They vsualze the underlyng structure of the feature space by dsplayng topologcally regons of maor concepts n the mage space and the dstrbuton over the concept regons. We wll evaluate the three SOMs n the followng. SOM n fgure 5(a) only uses HS-hstogram, a chromatc feature. It vsualzes the dstrbuton of color of mages. For example, mages wth blues are put on the rght-bottom of map. However, t cannot dfferentate mages wth smlar colors but dfferent textures. For example, some wnter scenery photos and some obect photos both are almost black and whte, therefore the concepts of these two categorzes of mages are not clearly dstngushed n the map. Also, obects wth dfferent colors lke tenns-ball and baseball are separate apart. 1 The term teraton n our algorthm s defned as number of presentaton of all mages (see secton.). However, most SOM algorthms defne teraton as number of presentaton of nput patterns. Usng the latter defnton, there are 500 x 50 = 5000 teratons n ths experment
Gabor flter, a textural feature, s used to tran the SOM shown n fgure 5(b). Contrary to the former SOM, t can separate obects from scenery photos perfectly but photos wth smlar colors are not grouped together. For nstance, photos wth blues and greens are n dfferent regons n the map. The fnal one combnes HS-hstogram and Gabor flter. Advantages of chromatc and textural features are found n ths map. Obects and scenery photos are separate apart and mages wth smlar colors are close together. Of course the performance s not as good as fgure 5(a) n terms of chromatc dfferentate performance, and fgure 5(b) n terms of textural dfferentate performance. Nevertheless, t provdes a balance of usng both features. Labelng dfferences We have also compared dfferent labelng methods. Fgures 5(c) and 6(a-b) dsplays the same SOM traned by same parameters n secton 3.1 but labeled wth dfferent methods. 5(a) HS-Hstogram The map n fgure 6(a) s labeled by smlarty, that s, usng the most smlar mage n the database to represent the node. It vsualzes best the nformaton of the weghts traned n the map thus present an overvew of dstrbuton of mages n the database and the map s generally very contnuous. However, the label mage of a node may not be an mage mapped to that node. And there are cases of duplcate representatve mages. If t s used as a user-nterface for browsng, users may confuse f ther nterested mage exsts repeatedly throughout the map; they do not know whch node they should further explore. Fgure 6(b) shows a map labeled by result-mean. The label mage s chosen from the most smlar mage to the mean of features vector of mages mapped to that node. It prevents the problem ust mentoned of labelng by smlar and t fnds the best mages to represent mages mapped to that node. However, the label mages are not the most smlar one to the weghts of the nodes thus the map s vsually more fragmented. 5(b) Gabor Flter The last map s shown n fgure 5(c), whch s labeled by result-smlarty. It has some advantages of labelng by smlarty and labelng by result-mean. It provdes a balance between representng weghts of nodes and representng the features of mages mapped to the nodes. Two mportant propertes of result-mean and result- smlarty methods are that the label mage must be ncluded n mages mapped to the node and duplcate label mage wll not exst. 4. CONCLUSION We have proposed an approach for effcent colour mage retreval. Image feature and SOM are used to vsualze a summary of all mages n the database. Experments show that mage features can be combned to balance the advantages of dfferent features. And also, SOM labeled by result-nearest s most sutable for representng both the map and the mages mapped to the nodes. 5(c) Combnaton of HS-hstogram and Gabor Flter Fgure 5(a-c): 10x10 SOMs traned by dfferent features (all three maps are labeled by result-smlarty)
exstng SOM to adapt the new mages. Nevertheless t cannot apply for HSOM/M-SOM approach; modfcatons of top-level SOM make lower-level SOM to be re-traned. We wll contnue to study on ths feld n order to solve ths problem. A problem we have encountered s quanttatve evaluaton of the results. We clam that the usefulness of maps can be evaluated by practcal tests only. Therefore, we wll setup human experment for testng the performance of human n dfferent varables. As SOM provde a vsual overvew of the database, once subects have used a SOM for a perod, we clam that they wll have a better performance due to mplct learnng. 6(a) Label by Smlarty Eventually, browsng and searchng are not compettve technques. We wll work on a proposal whch combnes the two maor approaches. More features and new combnatons of features wll be tested. For nstance, text-annotaton s a useful feature for manual categorzaton of mage database. 5. ACKNOWLEDGEMENT Ths proect s supported by the Earmarked Grant for Research of the Research Grants Councl of Hong Kong, CUHK 7034-98E. 6. REFERENCES 6(b) Label by Result-Mean Fgure 6(a-b): 10x10 SOMs labeled by dfferent methods (both are traned by combned features) The results of experments are encouragng. We thnk that SOM s an deal technque for generatng maps for mage browsng snce t can reflect the relatonshp among all mages n the database. However, there are some restrctons of current approach usng SOM. Frst, f the number of mages s large, the number of mages mapped n one node wll be large. It s neffcent for user to browse. An mmedate derved approach to ths problem s to generate SOMs for the mages mapped to each node. We can apply ths approach recursvely untl the number of mages n the node s less than a specfc amount. Actually, ths approach s smlar to the Herarchcal Self-Organzng Map (HSOM) from Zhang et al. and the Multlayered Self-Organzng Feature Map (M-SOM) from Chen et al. [10]. The purpose of these two research are vsual ndexng and nternet categorzaton respectvely. Ths knd of SOM varatons can be appled to our research also. We wll have to desgn approprate algorthms for labelng. Another restrcton s lack of dynamc mantenance. Inserton or deleton mages needs to re-tran the SOM. For a normal sngle layer SOM, we can apply addtonal tranng teratons to the [1] S.Craver, B.-L. Yeo, and M.M. Yeung, Image browsng usng data structure based on multple space-fllng curves. To appear n the Procceedngs of the Thrty-Sxth Aslomar Conference on Sgnals, Systems, and Computers, pp. 155-166 (Pacfc Grove, CA), November 1-4 1998. [] J.-Y. Chen, C.A. Bouman and J. C. Dalton, Actve Browsng usng Smlarty Pyramds, Proc. Of IS&T / SPIE Conference on Storage and Retreval for Image and Vdeo Database VII, vol. 3656, pp. 144-154, (San Jose, Calforna), January 1999. [3] T. Kohonen, Self-Organzaton and Assocatve Memory, Sprnger Seres n Informaton Scence, vol. 8, 1984. [4] T. Kohonen, Self-Organzng Maps, Sprnger Seres n Informaton Scence, vol. 30, 1995. [5] H. Zhang and D. Zhong, A Scheme for Vsual Feature based Image Indexng, n Proc. Of IS&T / SPIE Conference on Storage and Retreval for Image and Vdeo Database III, vol. 40, pp. 36-46, (San Jose, CA), February 9-10 1995. [6] K. A. Han, J. C. Lee and C. J. Hwang, Image Clusterng usng Self-organzng feature map wth Refnement, Proceedngs IEEE Internatonal Conference on Neural Networks, Vol. 1, pp. 465-469, 1995. [7] M. Swan and D. Ballard, Color Indexng, Internatonal Journal of Computer Vson, 7:1, pp 11-3, 1991. [8] B. S. Manunath and W. Y. Ma, Texture Features for Browsng and Retreval of Image Data, IEEE Transactons on Pattern Analyss and Machne Intellgence, Vol. 8, No. 18, pp. 837-84, August 1996. [9] C. C. Chen and C. C. Chen, Flterng Methods for Texture Dscraton, Pattern Recognton Letters, vol. 0, pp. 793-790, 1999. [10] H. Chen, C. Schuffels and R. Orwg, Internet Categorzaton and Search: A Self-Organzng Approach, Journal of Vsual Communcaton and Image Representaton, Vol. 7, No.1, pp. 88-10, March 1996.