Searching Large Image Databases using Color Information

Transcription

1 Searchng Large Image Databases usng Color Informaton Ioan Racu CMSC350: Artfcal Intellgence Wnter Quarter 2004 Department of Computer Scence Unversty of Chcago ABSTRACT The goal of ths project s to mplement an exploraton system for large mage databases n order to help the user fnd smlar mages to a query mage n a very effcent method. The methods proposed n ths system should be scalable to the largest databases currently n use; for example, has an mage search database va keywords search of about 1,000,000,000 mages. Whle the proposed system works wth any type of low-level feature representaton of mages, we mplement our system usng color nformaton. The system s bult n three stages: 1) the feature extracton stage n whch mages are represented n a way that allows effcent storage and retreval results closer to the human percepton; 2) the second stage conssts of clusterng the mage database va k-means clusterng n whch the clustrod would allow quck human comprehenson of the type of mages wthn the correspondng cluster; 3) the thrd stage s the vsualzaton stage n whch the results are dsplayed to the user. We evaluated the performance of our system based on the retreval accuracy and on the perceptual smlarty order among retreved mages. Experments usng a general purpose database of 2100 mage were used to show the practcalty of the proposed system n both accuracy and tme complexty. Two features could be added to the current system as future work: 1) buld a herarchy of clusters whch would allow even faster retreval results; 2) mplement mult-dmensonal scalng (MDS) technque to provde a tool for the vsualzaton of the database at dfferent levels of detals. Keywords: mage database, mage retreval, k-means clusterng, color nformaton, HSV 1.0 INTRODUCTION The deas of ths project, buldng an mage retreval system, was nspred from the fact that very few, f any commercal systems exst that allow users to query an mage database va mages rather than keywords. One commercal mage retreval system mplemented by Google.com s shown n Fgure 1. Google.com crawls the Internet and collects mages whch t then stores n a database. For each pcture collected, the crawler also collect textual nformaton whch surrounds the pcture and whch mght even be encoded n the pcture tself (.e. the fle name, drectory name, Page 1 of 24

2 etc ), and then creates a dctonary that assocates keywords wth mages. From ths pont, once the mages have a set of keywords that represent t, the problem gets reduced to a text retreval system whch Google.com has already mplemented and was successful n dong so. Fgure 1: Exstng mage retreval system va keyword queres mplemented by Google.com; left fgure: sunset Note n Fgure 1 the query of the keyword sunset resulted n many results from whch most of them really look lke sunsets; t s nterestng to see that the word sunset appears n each and everyone of the pctures names, so unless a pcture was mstakenly labeled by the owner of the pcture, the retreval of the system based on such keywords would yeld very good results! The mage retreval system s broken up nto two separate phases. The entre frst phase s depcted n Fgure 2. The frst phase can be consdered the tranng part of the system, n whch the raw mage database s massaged nto an mage feature database, whch s later used as the nput to k-means clusterng n order to buld a 1 level herarchy (expandable to more levels for larger databases to mprove performance) of mages organzed by clusters. The leaf cluster contans a group of smlar mages (based on the dstance metrc used, n our case color nformaton), that wll be used n the Page 2 of 24

3 retreval phase as the set of mages most smlar to the query mage. The fnal part of the tranng phase s vsualzng the mage database by dsplayng the clustrods of the database. Stage 1 Stage 2 Stage 3 Clustrod 1: Level 1 Image Database: Image Database: Raw Images Image Features Clusterng Leaf: Images Level 2 Clustrod : Level 1 Vsualzaton Clustrod sqrt(n): Level 1 Fgure 2: Image retreval system overvew: tranng phase The second phase can be depcted n Fgure 3. It conssts of an nterface from whch a user supples a query mage, after whch the mage gets decomposed nto ts representatve mage features. Based on these mage features, the query mage s compared aganst the clustrods, and once the most smlar cluster s found, the query mage s compared to each of the mages n the leaf cluster, and a dstance vector s calculated from the query mage to the rest of the mages n the leaf cluster. The mages n the leaf cluster are ranked accordng to ther dstance from the query mage, and dsplayed n rank order. Page 3 of 24

4 Stage 1 Stage 2 Stage 3 Query Image: Raw Image Query Image: Image Features Clustrod 1: Level 1 Clustrod : Level Clustrod sqrt(n): Level 1 Leaf: Images Level 2 Vsualzaton Fgure 3: Image retreval system overvew: retreval phase 2.0 RELATED WORK Ths secton wll cover related work regardng mage retreval systems, other mplementatons, and the state of the art of today s mage retreval systems. The book Algorthms for Clusterng Data [3] s an excellent source for clusterng technques. There are a few earler research efforts on mage databases exploraton. The system descrbed n Photobook [4] descrbed a content based manpulaton approach for mage databases. The QBIC [5] system was developed at IBM and allows queres by mage and vdeo content. The system n MARS [6] addresses supportng smlarty queres. The system descrbed n [7] s based on a new data structure bult on the mult-lnearzaton of mage attrbutes for effcent organzaton and fast vsual browsng of the mages. The systems proposed n [8]-[9] are based on Herarchcal Self-Organzng Map and are used to organze a complete mage database nto a 2-D grd. In [10]-[11] Mult-Dmensonal Scalng s used to organze mages returned n response to a query and for drect organzaton of a complete database. Actve browsng s proposed n [12] by ntegratng relevance feedback nto the browsng envronment and thus, users can modfy the database organzaton to sut a desred task. Page 4 of 24

5 3.0 IMPLEMENTATION I mplemented the mage retreval system n C++ under Mcrosoft Wndows XP n Vsual Studo.NET. Due to poor support for mage lbrares n C++, I wll be usng a thrd party API called Magck++ [1]. My mplementaton had the followng major components, all mplemented by me, unless otherwse stated: Feature extracton o o o o Readng of varous fle formats from dsk (Magck++) Image segmentaton RGB to HSV converson HSV hstograms Smlarty Metrc K-means clusterng Vsualzaton: buldng the results n rank order o o Wrtng results nto mages on dsk (Magck++) Dsplayng the results to the user n a graphcal nterface (Magck++) In total, I ended up wth about 2500 lnes of source code, and a very effcent and automated CBIR system that could search for mages n relatvely large mage databases. A screen shot of the CBIR system, whch outlnes all the above components can be seen n the fgure below. Fgure 4: Screen shot of the mplemented CBIR system Page 5 of 24

6 3.1 Feature Extracton We used color as the low-level feature representaton n the mages of the database. Because the HSV color model s closely related to the human vsual percepton, we decded to convert all mages from the RGB to the HSV color space. We encode the spatal dstrbuton of features n mages va a fxed parttonng scheme; each mage dvded nto M N overlappng blocks, for whch 3 separate local hstograms (H, S, and V) are calculated for every block. Ths provdes a compact and effcent storage and retreval soluton to an otherwse very hard problem. The locaton of area-peak for every local hstogram determnes the value of the correspondng hstogram. Fgure 5 depcts the process of feature extracton for 1 partcular block; the pxel values (hue, saturaton, or value) are taken and a hstogram s computed, from whch the maxmum s taken; ths maxmum becomes the representatve value for that block for the correspondng component Block Hstogram [ K K ] e11e e e K m1 m2 e mn Occurence Pxel value Fgure 5: Depcts the process of feature extracton for 1 block Table 1 depcts 2 examples of mages decomposed nto ther feature vector and what they look lke when converted back nto ther RGB format. Page 6 of 24

7 Table 1: Examples of mages decomposed nto ther feature vector and what they look lke when converted back nto ther RGB format HSV Color Space It has been verfed expermentally that color s perceved through three ndependent color receptors whch have peak responses at approxmately, red, green and blue wavelengths and thus, t can be represented by a lnear combnaton of the three prmary colors (R, G, B) (Fgure 6). Fgure 6: Mxture of lght (Addtve prmares) Page 7 of 24

8 The characterstcs used to dstngush one color from another are: Hue s an attrbute assocate wth the domnant wavelength n a mxture of lght waves. It represents the domnant color as perceved by observer (for example, orange, red, pnk, etc) Saturaton refers to the relatve purty or the amount of whte lght mxed wth a hue. Pure colors are fully saturated. Colors such as pnk (red and whte) are less saturated wth the saturaton beng nversely proportoned to the amount of whte lght added. Brghtness (or value) embodes the chromatc noton of ntensty. Hue and saturaton together are called chromatcty. A color can be descrbed n terms of ts chromatcty and brghtness. The formulae from Fgure 7 represent the transformaton from RGB to HSV space. Snce the RGB space s not perceptually unform (the proxmty of colors n ths space does not ndcate the color smlarty), ths space s used as a startng pont for generatng other color spaces havng the propertes above. Snce RGB s a complete space, the generated color spaces are all complete even f the transformaton from one space to another s not necessarly lnear (such as n the case of the HSV space). Of the all spaces, only the HSV quantzed color space satsfes the propertes of unformty, compactness, completeness, and naturalness that s needed for workng wth mages. Fgure 7: The relatonshps between prmary colors and hue, saturaton, and brghtness components Page 8 of 24

9 3.2 Smlarty metrc Clusterng methods requre that an ndex of proxmty or assocatons be establshed between pars of patterns; a proxmty ndex s ether a smlarty or dssmlarty. Snce our retreval system s desgned to retreve the most smlar mages wth a query mage, the proxmty ndex wll be defned wth respect to smlarty. Dfferent smlarty measures have been suggested n the lterature to compare mages [14, 13]. Let q and t represent the block number n two mages Q and T, respectvely. Let ( h,s, v ) and ( h,s, v ) represent the domnant hue-saturaton par of the selected block n the two mages Q and T. The block smlarty s defned by the followng relatonshp [15]: q q q t t t S ( q,t ) = 1+ a Dh ( h,h ) + b* D ( s,s ) + c D ( v,v ) q t 1 s q t v q t Here D h, Ds and Dv represent the functons that measure smlarty n hue, saturaton and value. The constants a, b and c defne the relatve mportance of hue, saturaton and value n smlarty components. Snce human percepton s more senstve to hue, a hgher value s assgned to a than to b. The followng functon [15] was used to calculate D h : The functon D h k 1 π ( h,h ) = 1 cos h h q t 256 q Dh explctly takes nto account the fact that hue s measured as an angle. Through emprcal evaluatons, t 2 when k=2 provdes a good non-lnearty n the smlarty measure to approxmate the subjectve judgment of the hue smlarty. The saturaton smlarty [15] s calculated by: D s ( s,s ) q t = s q s 256 t Page 9 of 24

10 The value smlarty s calculated by usng the same formula as for saturaton smlarty. The range for D s and D v s [ 0..1] snce the saturaton and value smlarty measures are normalzed by dvdng wth ther maxmum values; snce the cos functon belongs to [ 1..1], D h belongs to [ 0..1]. Takng nto account the ranges for D h, Ds and D v, the range for the functon S that measures the smlarty between two mage blocks s a + b + c. In other words, a smlarty value of 1 means that two mages are dentcal, values close to 1 ndcate smlar mages, whle values close to 0 ndcate dssmlar mages. Note that the smlarty metrc can never reach 0 because the lower bound s 1 over a fnte number; however, ths does not pose any lmtatons on our approach snce we are nterested n measurng the smlarty between mages and therefore we are lookng for smlarty values close to 1. Usng the smlartes between the correspondng blocks from mage Q and mage T, the smlarty between two mages s calculated by the followng expresson [15]: S ( Q,T ) = M N m S = 1 M N m = 1 ( q,t ) The quantty blocks. m n the above expresson represents the maskng bt for block and M N stands for the number of Page 10 of 24

11 4.0 EMPIRICAL PERFORMANCE RESULTS I used an mage database of 2100 mages to test my proof of concept, but a larger database would prove the scalablty of the system emprcally. A real nce database that I found s the Corel Stock Photos [2]; t conssts of over 39,000 general purpose mages; the bggest dsadvantage to ths mage database s that t s prohbtvely expensve, and hence I cannot use t. 4.1 Performance Results From the performance results gven n the next few tables and fgures, t can be seen that clusterng s a very good optmzaton from the performance pont of vew, and that the mplemented system s very scalable. It should be noted that the entre search tme n the entre database s about 3 mllseconds wth the 1 level of clusterng, whle t s about 80 mllseconds wth no clusterng. The reason t can search so fast through the entre database even wthout any clusterng s because of the effcent representaton of the mages by the feature vectors wth only 192 components for each mage! The pre-processng stage takes a lttle over 1 second per mage, but when searchng through the database, there s normally only 1 mage that s the query, and thus ths does not hnder the scalablty of the system. Clusters 1 37 Query Space Feature Space Tme to Cluster (seconds) Tme to search (seconds) Tme to extract features n 1 mage (seconds) Compare mages per second Table 2: System tmng data comparng the clusterng approach wth no clusterng at all The next table shows the amount of off-lne tme the system needs to prep tself before queres can be made to the system. To compute the feature vectors and computng the dstance matrx from scratch, t would take about 30 mnutes for about 1400 mages, but snce ths s an off-lne process, ths s OK. Once these are computed, the data can be loaded n memory n less than 10 seconds for the entre mage database. Tme Readng Features Vectors (seconds) Computng Features Vectors (seconds) Readng Dstance Matrx (seconds) Computng Dstance Matrx (seconds) Page 11 of 24

12 Table 3: System Performance tmng data for loadng/computng the off-lne nformaton The next two fgures show how scalable our system s n comparson to a system that would search through the entre database. For example, has about 1,000,000,000 mages n ther database. Even wth ths large of a database, the search (query) space would be less than 100,000 mages whch can be done on the order of a few seconds. If we mprove the system by havng the herarchy of clusters, we can reduce the search space to only a few hundred mages, and hence retreve results n a matter of mllseconds! Query Space Query Space (Number of Images) E+07 1E+08 1E+09 Database Sze (Number of Images) QS - NO C QS - 1 C QS - H sq C QS - H log C Fgure 8: Query Space In Fgure 8 and Fgure 9, QS/QT - NO C represents no clusterng, QS/QT 1 C represents 1 level of clusterng, QS/QT H sq C represents herarchcal clusterng wth square root of n clusters, where n s the number of mages n the level above, and QS/QT H log C represents herarchcal clusterng wth logarthmc number of clusters n relaton to the level above. The notable feature s that even wth very large databases (hundreds of mllons of mages), the CBIR system could stll retreve the most relevant mages n less than a second, whch s actually qute amazng when thnkng of the sze of the databases we are tryng to search. Page 12 of 24

13 Query Tme Query Tme (Seconds) E+08 Database Sze (Number of Images) QT - NO C QT - 1 C QT - H sq C QT - H log C Fgure 9: Query Tme Database Sze Mem Req (MB) QS - NO C QS - 1 C QS - H sq C QS - H log C Table 4: Memory requrements (n megabytes) for the query space In the above table, we see that all three clusterng mplementatons would have a very small memory requrement, needng only on the order of dozens of MB whch can easly be done even on commodty workstatons. Lastly, we calculated the performance of the system by measurng some statstcs regardng the accuracy of the results. Table 5 depcts these fndngs. We used 1383 mages for the mage database, and we used the remanng 718 mages to test the system wth. These 718 mage were chosen randomly from the entre 2100 mages whch makes up the database. For each query, we retreved the 4 most smlar mages, and rank them by smlarty. The frst result ndcates Page 13 of 24

14 that all retreved mages were color smlar to the query mage. The second result shows that the semantc meanng mpled by the query mage was preserved across 78% of the retreved mages. The thrd results shows that the most smlar mage to the query mage retreved was 91% of the tmes semantcally smlar to the query mage. The last result shows that n all query tests done, at least 1 of the 4 mages retreved was semantcally smlar to the query mage. Descrpton Percentage % all retreved color smlar 100% % all retreved semantc smlar 78% % hghest rank semantc smlar 91% % at least 1 semantc smlar 100% 4.2 Sample Query Results Table 5: Query retreval performance results Table 6 through Table 10 show some sample queres performed on the mage database. All the rest of the queres (almost 700 more queres) can be found onlne at my web ste at The clustrods of the database can be depcted n Fgure 10. Note the varyng colors of the clustrods, rangng from black to varous shades of blue, green, orange, red, brown, yellow, and whte; t vrtually covers the entre color space. Ths s the result of a very good smlarty metrc and the success of the clusterng technque used! The retreval accuracy s very good, gven that the amount of nformaton present n the feature vectors s very small n comparson to the orgnal mages themselves. Page 14 of 24

15 Fgure 10: Clustrods of the mage database Page 15 of 24

16 Table 6: 5 sample queres; the query mage s the leftmost mage, and the value depcted above the mage name denotes the smlarty between the partcular mage and the query mage (1 most smlar and 0 least smlar) Page 16 of 24

17 Table 7: 5 sample queres; the query mage s the leftmost mage, and the value depcted above the mage name denotes the smlarty between the partcular mage and the query mage (1 most smlar and 0 least smlar) Page 17 of 24

18 Table 8: 5 sample queres; the query mage s the leftmost mage, and the value depcted above the mage name denotes the smlarty between the partcular mage and the query mage (1 most smlar and 0 least smlar) Page 18 of 24

19 Table 9: 5 sample queres; the query mage s the leftmost mage, and the value depcted above the mage name denotes the smlarty between the partcular mage and the query mage (1 most smlar and 0 least smlar) Page 19 of 24

20 Table 10: 5 sample queres; the query mage s the leftmost mage, and the value depcted above the mage name denotes the smlarty between the partcular mage and the query mage (1 most smlar and 0 least smlar) Page 20 of 24

21 5.0 CONCLUSION AND FUTURE WORK In concluson, we demonstrated that a CBIR system can be bult effcently that works on real world general mages very well. As the number of dgtal mages ncreases n our daly lves, the need for some systems wll grow and eventually, these systems wll become as common as text search engnes have become n the past 10 years! Good CBIR systems can eventually be used n many domans that currently rely on hand annotatng and hand analyzng large number of mages. As future work, the current system could be extended relatvely smple wthout any new sgnfcant theoretcal work to support a search tme complexty that s logarthmc n relaton to the sze of the database usng herarchcal clusterng. In order to support very large databases (on the order of hundreds of mllons of mages), better vsualzaton tools would be needed as well even to nterpret the large amounts of results. One effcent method that s sutable n ths context s multdmentonal scalng (MDS) whch can be mplemented n the vsualzaton stage of our system to vsualze the results n a 2-dmentonal space as the mages relate to each other va ther smlarty metrc. Stage 1 Stage 2 Stage 3 Image Database: Image Database: Raw Images Image Features Clusterng Clustrod: Level 1 Clustrod: Level 1 Clustrod: Level Clustrod: Level 2... Leaf: Images Level n MDS Fgure 11: Image retreval system overvew: tranng phase The new enhanced mage retreval system s depcted n Fgure 2. The frst phase s dentcal to our system; we use the mage feature vectors as the nput to k-means clusterng n order to buld a herarchy of mages organzed by clusters. The leaf cluster contans a group of smlar mages (based on the dstance metrc used, n our case color nformaton), that wll be used n the retreval phase as the set of mages most smlar to the query mage. The fnal part of the Page 21 of 24

22 tranng phase s the mult-dmensonal scalng (MDS) technque that wll be used to vsualze the mage database at varyng levels of detal. The second phase can be depcted n Fgure 3. It conssts of an nterface from whch a user supples a query mage, after whch the mage gets decomposed nto ts representatve mage features. Based on these mage features, the query mage s compared aganst the clustrods at the frst level of the herarchy, and the search contnues deeper nto the herarchy only along the best path, based on the dstance metrc used. The query mage s compared at each level wth the clustrods, and eventually, the query mage wll arrve at a leaf cluster. The query mage s compared to each of the mages n the leaf cluster, and a dstance vector s calculated from the query mage to the rest of the mages n the leaf cluster. The dstance vector s then the nput to the MDS module, whch graphcally wll dsplay the results n a 2- dmentonal space wth the most smlar mages close to the query mage and the most deferent mages far away from the query mage Fgure 12: Image retreval system overvew: retreval phase From a performance pont of vew (gven that I can get a larger mage database), t would be nterestng to mplement the herarchcal clusterng, whch should be relatvely smple to modfy the current code to support the mult level herarchy rather than the sngle level I currently used. From a utlty pont of vew, varous smlarty metrcs should be expermented wth for the color feature vectors the system extracts. Also, modfyng the feature vectors to contan other nformaton such as texture nformaton, shape nformaton, and annotated text would help n augmentng the results n Page 22 of 24

23 the system and create a complete end-to-end soluton for a CBIR system that would work excellent and effcent at the same tme! More query results, electronc verson of ths document, the presentaton on ths project, and source code can be found at my web ste at For more nformaton, comments, or suggestons, I can be reached at racu@cs.uchcago.edu. 6.0 REFERENCES [1] [2] [3] A. K. Jan and R. C. Dubes, Algorthms for Clusterng Data, Prentce Hall Advanced Reference Seres [4] A. Pentland, R. W. Pcard, S. Sclaroff, Photobook: Content Based Manpulaton of mage databases, n Multmeda Tools and Applcatons, edtor Burko Furht, Kluwer Academc Publsher, Boston, pp , [5] Flckner, M., and et al., Query by Image and Vdeo Content: The QBIC System, IEEE Computer, [6] M. Ortega, Y. Ru, K. Chakrabart, S. Mehrotra, and T. S. Huang, Supportng smlarty queres n MARS," ACM Conf. on Multmeda, [7] S. Craver, B.-L. Yeo, and M.M. Yeung. Image browsng usng data structure based on multple space-fllng curves. Proceedngs of the Thrty-sx Aslomar Conference on Sgnals, Systems, and Computers, Nov. 1-4, 1998 [8] I.K. Seth and I. Coman. Image retreval usng herarchcal self-organzng feature maps. Pattern Recognton Letters 20, pp , 1999 [9] H. Zhang and D. Zhong. A scheme for vsual feature based mage ndexng. Proceedngs of SPIE/IS&T Conf. on Storage and Retreval for Image and Vdeo Databases III, vol. 2420, pp , 1995 [10] J. MacCush, A. McPherson, J. Barros, and P. Kelly. Interactve layout mechansms for mage database retreval. Proceedngs of SPIE/IS&T Conf. on Vsual Data Exploraton and Analyss, III, vol.2656, pp , 1996 [11] Y. Rubner, L. Gubas, and C. Tomas. The earth s mover dstance, mult-dmensonal scalng, and color-based mage retreval. Proceedngs of the ARPA Image Understandng Workshop, May 1997 Page 23 of 24

24 [12] J.-Y. Chen, C.A. Bouman, and J.C. Dalton. Smlarty pyramds for browsng and organzaton of large mage databases. Human Vson and Electronc Imagng III, pp , 1998 [13] Faloutsos C. et al. (1993). Effcent and effectve queryng by mage content, Techncal Report, IBM. [14] Swan M.J. and Ballard D.H. (1991). Color Indexng, Internatonal Journal of Computer Vson, 7(1), pp [15] Seth I. K., Coman I., Day B. et al. (1998). Color-WISE: A system for mage smlarty retreval usng color, Proceedngs SPIE: Storage and Retreval for Image and Vdeo Databases, 3132, pp Page 24 of 24