Background Removal n Image ndexng and Retreval Y Lu and Hong Guo Department of Electrcal and Computer Engneerng The Unversty of Mchgan-Dearborn Dearborn Mchgan 4818-1491, U.S.A. Voce: 313-593-508, Fax: 313-593-9967 ylu@umch.edu Abstract Ths paper presents our research n mage content based ndexng and retreval, a key technology n dgtal mage lbrares. In most of the exstng mage content-based technques, mage features used for ndexng and retreval are global, features are computed over the entre mage. The major problem wth the global mage feature based retreval methods s that background features can be easly mstakenly taken as object features. When a user attempts to retreve mages usng color features, he/she usually means the color feature of an object or objects of nterests contaned n the mage. The approach we descrbe n ths paper utlzes color clusters for mage background analyss. Once the background regons are dentfed, they are removed from the mage ndexng procedure; therefore, no longer nterferng wth the meanngful mage content durng the retreval process. The algorthm conssts of three major steps of computaton, fuzzy clusterng, color mage segmentaton, and background analyss. 1. Introducton Dgtal magery s a convenent medum for descrbng and storng spatal, temporal, spectral and physcal components of nformaton contaned n a varety of domans. Large databases contan thousands of dgtal mages that can occupy ggabytes of space, and hence t s dffcult for a user to fnd useful nformaton usng smple browsng technques. Therefore, effcent and automatc algorthms are requred for ndexng and retrevng mages from databases. Conventonally mages are ndexed usng text nformaton, such as keywords, date, artst, etc. [Gro88, ag85, FJL97]. However, the nadequacy of textual descrptons s obvous because the complexty of the nformaton mbedded n mages (number and types of objects, ther attrbutes and spatal relatonshps) can not be syntheszed n a few key words. Untl recently wth the fast growng technology n computer vson, mage content based ndexng and retreval are the new areas of research and development [Ja93, PPS94, FS95, Gup6, FDW96]. There s a growng demand for systems usng pctoral nformaton for both commercal and cultural applcatons. Research n content-based mage retreval can be characterzed n the followng categores: query-by-vsual sample, pctoral queres, and/or lngustc queres. In a query by vsual sample, a user must submt a sample mage to the system, and then the system searches ts database to fnd the mages most smlar to the sample mage. The smlarty measure s usually taken from mage features such as color, shape and texture. The IBM QBIC system [FS95] and the Vrage mage search engne both use ths type of query[gup96]. Pctoral queres are sketch-based queres, and these systems usually provde sketch tools to allow the user to draw mage features, such as shape, sze, color, and texture that occur n the mages to be retreved. The system uses the sketch as a query and search for mages that contan the sketch. VsualSEEK, a content-based mage/vdeo retreval system at Columba Unversty [SmC96], uses pctoral queres extensvely. Belonge et al[bcg98] used blobworld to represent the composton of mage content. The blobworld descrptors uses -D ellpses, or blobs, each of whch possesses a number of attrbutes. Lngustc queres that drectly refer to mage contents are popular among many users. These queres clearly dentfy salent vsual propertes contaned n the desred mages usng the languages are understandable by most 1
populaton. These mage features can be low level, such as red, or hgher level such as horzon or horse. The Dgtal Lbrary Intatve at the Unversty of Calforna at Berkeley has made a sgnfcant effort n ths drecton[wl96]. In most of the exstng mage content-based technques, mage features used for ndexng and retreval are global features, features are computed over the entre mage. For example, retrevng smlar mages n color, the most of the exstng technques use a color hstogram generated from the entre mage[hse95, JAV96]. The major problem wth the global color hstogram based retreval methods s that they do not possess spatal relatonshp between object regons and t does not dfferentate the color between the background and foreground. When a user attempts to retreve mages usng color features, he/she usually means the color feature of an object or objects of nterests contaned n the mage. For example, f a desgn engneer wants to retreve the mages of automobles usng color feature, he would lkely submt queres such as red cars or blue seat covers, etc. Therefore n many applcatons, t makes more sense to ndex and retreve mages based on the color nformaton of objects. If a retreval system only uses color as the feature for mage retreval, a user may get many red cars wth black background for a gven query black car. The dfferentaton of background from foreground s an ll-posed problem n mage analyss. For moton pctures, the background features n the consecutve mages do not change much, therefore we can use the dfference mage of two consecutve mages n the tme doman to dfferentate background from foreground. In stll mages, mage background s not well defned. Generally speakng, the background of an mage ncludes all the features n the mage that do not belong to the objects of nterests. In many mages, background can be easly recognzed by a human beng but dffcult for computer, because the human bengs have excellent object recognton and analyss abltes. In many mages mage background features are even ambguous to human observers. Our approach to ths problem s to elmnate regons that are certan to be backgrounds. For those mages that are ambguous to human percepton, we wll not nterfere. We do not ntend to use object recognton n mage retreval, whch s a very complcated classc computer vson problem. Instead, we are nvestgatng a computatonally effcency and yet effectve background removal algorthm that can facltate mage retreval. The approach we descrbe n ths paper utlzes color clusters for mage background analyss. Once the background regons are dentfed, they are removed from the mage ndexng procedure; therefore, they are no longer nterferng wth the meanngful mage content durng the retreval process. We have mplemented the algorthm at our web ste, and tested the system on mages from three dfferent collectons of mages, car mages, Earth and space mages, and crashed car mages. We have compared the algorthm wth a global color hstogram based approach and the results show that the algorthm wth background removal s very effectve n retrevng mages contanng a few major objects. Due to the lack of color prntng n ths publcaton, we have reduced number of mage examples and converted all color mages to gray scales. A color verson of the paper can be found at webste:[guo: put the address here].. Background regon analyss Fgure 1 gves an over vew of our system that performs background regon analyss. The system has three major computatonal components: fuzzy clusterng algorthm, color mage segmentaton, and background regon analyss. The fuzzy clusterng algorthm attempts to fnd smlar colors n the Luv space usng fuzzy logc. The algorthm consders a cluster of smlar colors as a fuzzy set, and represents the lkelness of a color pxel belongng to a fuzzy set by a fuzzy membershp a color mage Fuzzy clusterng n color hstogram doman CL 1 Mappng clusters CL to mage doman Color mage segmentaton CL 3 : a set of color regons Background regon analyss [BR 1,, BR k ] Fgure 1. Background feature of a color mage analyss
functon. The fuzzy clusterng algorthm yelds a decomposton of the 3D-color hstogram; namely a number of non-overlappng heaps n the 3D-color hstogram, CL 1. The labelng of the clusters n CL 1 results n a multthresholded mage. The detals of ths algorthm can be found n [ChL98]. In ths paper, we wll only dscuss the color mage segmentaton and the background analyss algorthm, and how the system can be used n mage content based ndexng and retreval..1 Color regon segmentaton The fuzzy clusterng algorthm mentoned above generates a set of clusters, CL 1, each of the clusters n CL 1 represents a sub-spectrum of colors n the Luv space. The mappng of clusters n CL 1 to the mage doman takes two steps. Frst we compute the central color of each cluster n CL 1 usng the formula below. Assume M clusters n CL 1 and each mage pxel belongs to one of the clusters. C( p )K denotes belongs to the Kth cluster, and ts Luv values are [l, u, v ]. The central color of cluster K s represented by ( K ) ( K ) ( K ) [ l, u, v ], whch s calculated as follows: l l ( K ) u v ( K ) ( K ) C( p ) K # of pxels n cluster K C( p ) K #of pxels n cluster K v #of pxels n cluster K u C( p ) K Each pxel n the mage doman s assgned the representatve color value of ts cluster. We obtan the second set of clusters, CL, by computng the connected pxels of the same cluster color. In general, one cluster n CL 1 may generate more than one clusters n CL, therefore, CL s usually much larger than CL 1. For example, n one of our experments, the algorthm generated, from a CL 1 wth CL1 64, the CL that contans more than fve hundred clusters. The clusters contaned n CL have the followng propertes: Pxels n the same cluster are spatally connected and are assgned of the same color: the representatve color of the cluster. Therefore, we sometmes refer the clusters n CL as color regons. The regons n CL are usually small and p p a meanngful mage regon contans several clusters n CL. The next step n the mage segmentaton s an agglomeratve process. It attempts to merge clusters n CL based on the color dstances among neghborng clusters n the spatal doman, cluster szes and the maxmum number of clusters n CL 3. Durng the agglomeratve process, the dstance between two clusters s defned as the color dstance between the representatve colors of the two clusters, and the color dstance functons we use are defned n the Luv space. For two dark colors,.e. the colors whose L, u and v components all have very small values, then color dstance s dependent on the L component only,.e., dst([ L1, u1, v1 ],[ L, u, v]) L1 L. For other colors, we use the Eucldean dstance. It s mportant to use a dfferent dstance functon for dark colors. As we found out emprcally that human percepton does not match the Eucldean dstance to dfferentate dark colors. For example n Fgure, regon A and regon B have very close colors accordng to human percepton and, therefore, they should be regarded as belongng to the same regon. However, we found that the Eucldean dstance between the two regons s large. For example, one typcal Luv value of a pxel n regon A s (39.45,11.96,-7.3), and one typcal Luv value of a pxel n regon B s (30.87,7., -4.78), and hhe Eucldean dstance of the two color pxels s dst 8.54, whch s qute large. However, the dark color dstance value s dst 8.58, whch s more consstent wth human percepton. If we used the Eucldean dstance functon n the mage segmentaton, the background regon n ths mage s very lkely to be dvded nto several smaller regons. Ths example suggests that for dark colors, the saturaton components are nsgnfcant. B A 3
Fgure. Illustraton of dark color dstance functon. Regon A and B look smlar from human percepton. Whle ther Luv dstance s large. We have explored the followng four dfferent methods n mergng clusters. In the frst three methods, a common parameter, max_cls s used to control the maxmum number of clusters n CL3. Method 1 consders the clusters that are adjacent and have smlar colors as the frst canddates for mergng. It uses a control parameter, cl_dff_th to denote color dfference threshold. It frst attempts to merge the neghborng clusters whose color dstances are below cl_dff_th. The order of mergng s rrelevant. If the number of clusters at the end of the mergng process s greater than max_cls, then the algorthm selects the smallest cluster and merges the cluster wth one of ts neghbors to whch t has the smallest color dstance. Ths mergng process repeats untl the number of clusters n CL3 s no more than max_cls. The second method consders the sze of clusters as the only mergng crteron. It selects the smallest cluster and merges the cluster wth one of ts neghbors to whch t has the smallest color dstance. The process s repeated untl the number of clusters n CL3 s no more than max_cls. The thrd method consders the color dstance as the most mportant crteron n spatal clusterng. However, the computaton n fndng the mnmum color dstance between two adjacent clusters s qute tme consumng f a large number of clusters exst. To allevate the computatonal burden, we frst repeatedly merges the smallest clusters wth ther neghbors n the closest color dstance untl the total number of clusters s reasonable, e.g. 100. In the second pass, we select the two adjacent clusters whose color dstance s the smallest wthn the entre mage and merges the two clusters. Ths process repeats untl the top max_cls clusters n sze contan a large percentage of the mage pxels (e.g. 99%). Snce the largest max_cl clusters already cover the major percent of the entre mage pxels, the small clusters below the top max_cl do not affect much the fnal segmentaton result. Therefore at the thrd pass, the algorthm repeatedly merges the smallest cluster wth ts closest neghbor n color dstance untl the total number of clusters n CL 3 s no more than max_cls. The last method we explored used two dfferent thresholds for color dstance functons. For clusters that have smaller sze, they are merged f ther color dstance s less than the larger threshold, and for larger clusters, they are merged f ther color dstance s less than the smaller threshold. The threshold for small clusters s determned by the mage sze. In all these methods, the center of the new regon s calculated after every merge, the sze of the regon has changed, and so do the neghbors. The representatve color of a new regon merged from C 1 and C s calculated usng the followng formula: L1 1 + L L + 1 u1 1 + u u + 1 v1 1 + v v + 1 where ( L 1, u1, v1 ) and ( L, u, v ) are the representatve colors of C 1 and C respectvely. ote the order of mergng may affect the clusterng result n the mage doman. The results from the spatal clusterng s a set of color regons, CL 3, n whch all the pxels n the same regon n CL 3 are spatally connected and have the same representatve color. From computatonal pont of vew, Method 1 and are more effcent than Method 3. Method 4 s the more effcent n terms of computaton than method 3 and more effectve than Method 1 and. However, n many cases, Method 3 generates better results than the others do. In the trade off of computatonal tme and segmentaton results, we chose to use method 4 n our background analyss algorthm. Fgure 4 shows an example of mage segmentaton results generated by Method 4. Fgure 4 (a) shows the orgnal mage, (b) shows the mage of clusters generated by mappng the clusters generated by the fuzzy clusterng algorthm drectly to the mage doman, and (c) shows the output from the segmentaton algorthm. For the purpose of demonstraton, the mage shown n (b) s drawn n pseudo colors. The colors shown n (c), the resultng mage, are true representatve colors of color regons. It can be seen, the sgnfcant color features are well captured after color mage segmentaton.. Image background analyss The background analyss algorthm s developed based on the followng hypotheses: 4
(a) (b) (c) Fgure 3. Example 1 of mage segmentaton. (a)the orgnal mage. (b) The mage after the color clusterng. There are 1 color clusters and 598 spatal clusters. (c) The segmentaton result. A background regon must be moderate n sze and must be connected to the mage borders. The man purpose of ths algorthm s to dentfy bg color regons that are not belongng to objects of nterests, snce only large background regons can hamper the retrevng results. Regons occupy mage corners are more lkely to be background. For a gven mage collecton, we may have specfed background colors. However, snce a background color can also appear n objects of nterests n the mages of the same collecton, and not all the mages have the same background colors, the pror knowledge of background colors should be used cautously. Assume there are M color regons n the mage, denoted by R 1, R, R M. Every regon s descrbed by { C, x 1, y 1, x,y }, where C s the representatve color of the th regon and (x 1, y 1 ), (x,y ) are the coordnates of the top-left and bottom-rght corners of the boundng box of the regon. We frst dvde the mage plane nto nne blocks shown n Fgure 4. 0 1 3 6 7 8 4 Fgure 4. An mage plane s dvded nto nne blocks for background analyss. The szes of the nne blocks are determned by the mage sze. In general we want the center block to be large enough to contan the mage contents. Based on our experence, the followng szes of the nne blocks are feasble. Let the wdth and heght of an mage be W and H respectvely. The center block 4 s 0.7W0.7H, the four corner blocks, 0,, 6 and 8, have sze of 5 0.15H0.15W, the four lateral blocks, 1 and 7, have sze of 0.7W0.15H, 3 and 5 of 0.15Hx0.7W. For each regon R j, we calculate ts percentage r j over each mage block, (0, 1,.8). From the percentage, we can obtan the number of corners and lateral regons one cluster occupes. Let c and l be the number of corners and the number of lateral regons a cluster occupes respectvely. We set these two varables as follows: c the number of corner regons whose occupancy rato s greater than a gven threshold θ. l the number of lateral regons whose occupancy rato s greater than a gven threshold. The algorthm can be summarzed nto the followng rules. Rule1. For a regon R j, f ts representatve color C j belongs to the specfed background color and the regon occupes more than two corners,.e. c >, then R j s marked as a background regon. Rule. For a regon R j, f ts representatve color C j belongs to the specfed background color and the regon occupes more than one corners and one lateral blocks,.e. ( c >0 and l >0), then R j s marked as a background regon. These two rules apply to an mage collecton wth specfed background colors. In certan applcatons ths specfcaton s possble. For examples, mages taken ndoors often have specfc background colors; n our Earth and Space Scence mage collecton, we found that black and whte often appear as background colors. For those mage collectons have no specfed background colors, we use the followng rules: Rule3. For an mage regon R j, f t occupes more than two corner blocks,.e. c >,, and ts percentage over the center block s small, then set R j as a background regon. Rule4. For an mage regon R j, f t occupes one or more corner and lateral blocks, and ts percentage over the center block s small,.e. r j4 < θ b, where θ b s a threshold, then set R j as a background regon. a 5
Rule5. For an mage regon R j, f t occupes at least one corner mage s a color photo of the Pacfc Ocean wth blue or one lateral regons,.e., c >0, or l >0, and most of R j s pxels are n the lateral and corner blocks, then set R j as a background regon. In many mages, background regons occupy the corner and lateral blocks, and therefore, the corner and lateral regons are emphaszed n the background regon analyss. Addtonally only large clusters can be recognzed as background, whch s controlled by θ n the computaton of c and l. For example, f we set θ to 0.85, then a mage regon s consdered as occupyng a corner only f one of r j0, r j, r j6, r j8 s above 85%. In Rule 3 and 4, n addton to the occupancy measure of corners and laterals, we also check occupaton percentage of the mage regon nsde the central block. If ths percentage s small, t mples that the center block contans some other larger regons and the current cluster may be regarded as background. Wthout ths addtonal condton, these rules can be detrmental to the mages that contan meanngful regons that occupy the central block as well as the corners and/or lateral blocks. For example, Fgure 5 shows an example. The orgnal a a color. The large regon connectng the corners and a good porton of the central block s the Pacfc Ocean, whch s the mportant content to the mage. As the result of our background analyss algorthm, ths mage s consdered to have no background regons. Fgure 6 shows an example of the background analyss algorthm. Fgure 6(b) shows the mage of clusters generated by the fuzzy clusterng algorthms n the color space. Fgure 6(c) shows the color regons after the segmentaton algorthm. For the purpose of llustraton we used pseudo colors to dsplay dfferent regons. The whte grds n (b) and (c) dvde the mage nto nne blocks. The labels n (c) ndcate the background regons found by the background analyss algorthm. Fgure 6 (d) shows the representatve colors of regons generated by the segmentaton algorthm. After the background analyss, we dentfed n ths mage three background regons, whch are labeled as 1, and 3 n (c). In ths example, although cluster 1, and 3 are not connected wth each other n the spatal doman, they are all consdered as background because of the same representatve background color, black. 6
(a) orgnal mage (b) Resultng mage from the Clusterng algorthm Fgure 5. An mage has no background regons. 1 (a) (b) (c) (d) 3 Fgure 6. Example of background analyss. (a) The orgnal mage. (b) The mage after of clusterng n pseudo colors. (c) The color regons after segmentaton n pseudo colors and 3 dentfed background regons. (d) The segmentaton result llustrated n the representatve colors. 1 (a) (b) (c) (d) Fgure 6. Example of background analyss. (a) Orgnal mage. (b) The mage of color clusters n pseudo colors. (c) The color regons after segmentaton n pseudo colors and 1 dentfed background regon. (d) Segmentaton results shown n representatve colors. [Guo: choose one of the two fgures based on the qualty of the converted gray scale mages.] 7
Fgure 6 shows another example of mage segmentaton. In ths example, the algorthm found one bg background regon(see (c)) whch has representatve color whte (see (d)). 3. Image ndexng and retreval usng background analyss We have mplemented the background analyss algorthm n IMAGE-SEEKER, a content-based mage retreval system runnng on the web ste http://frey.umd.umch.edu:8080/html/. IMAGE- SEEKER supports ffteen dfferent colors that are commonly used n mage queres. To mplement these queres, we generate the color hstogram usng ffteen domnant colors for each mage and ndex the mage wth the color hstogram. Durng the mage retreval stage, the system takes a color query and looks for mages whose domnant colors match the color query. The background analyss can be used before the color hstogram generaton. If the algorthm dentfed any background regons, we wll remove these regons from the mage, and then compute the color hstogram from the remanng regons n the mage. Currently, our mage lbrary conssts of fve dfferent mage collectons: mages of cars, nteror and exteror mages of a Honda vehcle model, crashed cars, face mages, and mages of Earth and Space. We have tested the background analyss algorthm on all these collectons usng dfferent color queres. We found the algorthm s partcular effectve on mages of cars and crashed cars, and mages of Earth and Space Scence. Due to the lmt of space and the absence of color prntng, we are not able to show the mage retreval results n the paper. Readers may want to referred to the extended verson of ths paper that contans a number of examples mage retreval after background removal. 4. Concluson In ths paper, we presented an algorthm for background regon analyss n color magery and the experment results on the algorthm mplemented n a content-based mage retreval lbrary at a Web ste. We showed that the algorthm was effectvely used n content-based mage retreval on a number of dfferent mage collectons. The background removal algorthm can also be used n combnaton wth mage features other than colors. For examples, we can utlze the dentfed background regons to ndex mage background, dfferentate n-door from out-door mages, acheve more accurate retreval usng shape or texture after removng the dentfed background regons. 5. References [BCG98] Serge Belonge, Chad Carson, Hayt Greenspan, and Jtendra Malk, Color- and Texture-Based Image Segmentaton Usng EM and Its Applcaton to Content-Based Image Retreval, ICCV, 1998. [ChL98] Te Q Chen and Y Lu, Color Image Segmentaton n Color and Spatal Domans submtted to IEEE Trans. PAMI, 1998 [FDW96] Usama M. Fayyad, S. G. Djorgovsk, and cholas Wer, From Dgtzed Images to Onlne Catalogs, AI magazne, pp. 51-66, 1996. [FJL97] C. O. Frost, R. Johnson, S. Lee, and Y. Lu, Research on Human-centered Desgn and Evaluaton of Multmode Image Retreval, Fourth DELOS Workshop San Mnato, August 8-30 1997 [FS95] M. Flckner, H. Sawhney, W. black and etc, Query by Image and Vdeo Content, IEEE Computer, pp. - 3, September 1995. [Gro88]W. I. Grosky, An archtecture for ntegratng mages and text n a database envronment, n: Database machnes, A. K. Sood and A. H. Quresh, (edt.), ATO ASI Seres, vol. F4, 1988 [Gup96] Amarnath Gupta, The Vrage Image Search Engne: An open framework for mage management, SPIE conference on "Storage and Retreval for Stll Image and Vdeo Databases IV," Feb. 1, 1996. [HSE95] J. Hafner, H. S. Sawhney, W. Equtz, et al., Effcent color hstogram ndexng for quadratc form dstance functons, IEEE trans. on PAMI, Vol. 17, o. 7, pp. 79-736, July 1995 [Ja93] R. Jan, SF Workshop on Vsual Informaton Management Systems, SIGmod Record, vol., o. 3, Dec. 1993, pp. 57-75 [JAV96] A. K. Jan and Adtya Valaya, Image retreval usng color and shape, Pattern Recognton, vol. 9, no. 8, pp. 133-144, 1996 [ag85] G. agy, Image database, Image and Vson Computng, vol. 3, no. 3, pp. 111-117, 1985 [PPS94] A. Pentland, R. W. Pcard, and S. Sclaroff, Photobook; Tools for Content-Based Manpulaton of Image Databases, Proc. Storage and Retreval for Image and Vdeo Databases II, Vol., 185, SPIE 1994, pp. 34-47. [SmC94] J. R. Smth and Shh-Fu Chang, Quad-tree segmentaton for texture-based mage query, Proceedngs of nd annual ACM multmeda conference, San Francsco, Ca, Oct. 1994 [Wl96] Robert Wlensky, Toward Work-Centered Dgtal Informaton Servces, IEEE Computer, pp. 37-45, May 1996 8