Object-driven content-based image retrieval

Transcription

1 th Int. Worshop on Systems Sgnals & Image Processng -4 September 005 Chalda Greece 89 Obect-drven content-based mage retreval Ioanns Pratas* Baslos Gatos and Stavros Perantons Computatonal Intellgence Laboratory Insttute of Informatcs and Telecommuncatons Natonal Center for Scentfc Research "Demortos" 53 0 Athens Greece E-mal: {prata bgat sper}@t.demortos.gr *Correspondng author Irs Vanhamel and Hchem Sahl Electroncs & Informatcs Department Vre Unverstet Brussel 050 BrusselsBelgum E-mal: {uvanham hsahl}@etro.vub.ac.be Abstract: Ths paper presents a novel unsupervsed strategy for content-based mage retreval. It s based on a meanngful segmentaton procedure that can provde proper dstrbutons for matchng va the Earth mover's dstance as a smlarty metrc. The segmentaton procedure s based on a herarchcal watershed-drven algorthm that extracts meanngful regons automatcally. In ths framewor the proposed robust feature extracton and the many-to-many regon matchng along wth the novel regon weghtng for enhancng feature dscrmnaton play a maor role. Expermental results demonstrate the performance of the proposed strategy. Keywords: mage segmentaton content-based mage retreval Reference to ths paper should be made as follows: Pratas I. Vanhamel I. Sahl H. B. Gatos and Sahl H. (005 Obect-drven content-based mage retreval Int. J. of Sgnal and Imagng Systems Engneerng Vol. x No. x pp.xx xx. Bographcal notes: Ioanns Pratas receved the Dploma degree n Electrcal Engneerng from the Demortus Unversty of Thrace Xanth Greece n 99 and the Ph.D. degree n Appled Scences from Vre Unverstet Brussel Brussels Belgum n 998. From March 999 to March 000 he was at IRISA/VSTA group Rennes France as an INRIA postdoctoral fellow. He s currently worng as a Research Scentst at the Insttute of Informatcs and Telecommuncatons of the NCSR Demortos. Hs research nterests nclude D and 3D mage analyss mage and volume sequence analyss as well as content-based mage / 3D models search and retreval. Baslos G. Gatos receved hs Electrcal Engneerng Dploma n 99 and hs Ph.D. degree n 998 both from the Electrcal and Computer Engneerng Department of Democrtus Unversty of Thrace Xanth Greece. He s currently worng as a Researcher at the Insttute of Informatcs and Telecommuncatons of the Natonal Center for Scentfc Research "Demortos" Athens Greece. Hs man research nterests are n Image Processng and Document Image Analyss OCR and Pattern Recognton. Stavros J. Perantons s the holder of a BS degree n Physcs from the Department of Physcs Unversty of Athens an M.Sc. degree n Computer Scence from the Department of Computer Scence Unversty of Lverpool and a D. Phl. Degree n Computatonal Physcs from the Department of Physcs Unversty of Oxford. Snce 99 he has been wth the Insttute of Informatcs and Telecommuncatons NCSRR Demortos where he currently holds the poston of Senor Researcher and Head of the Computatonal Intellgence Laboratory. Hs man research nterests are n Image Processng and Document Image Analyss OCR and Pattern Recognton. Copyrght 005 Inderscence Enterprses Ltd.

2 90 I. Pratas B. Gatos S. Perantons I. Vanhamel H. Sahl Irs Vanhamel receved the MSc degree n electrotechncal engneerng and nformaton processng at the Vre Unverstet Brussel (VUB n 998. She s currently pursung the PhD degree at the Electroncs and Informatcs department at VUB. Her research nterests nclude mage segmentaton mathematcal morphology scale-space theory and multspectral mage processng. Hchem Sahl s currently Professor of mage analyss and computer vson wth the Department of Electroncs and Informatcs at Vre Unverstet Brussel (VUB Brussels Belgum. He coordnates the research team n computer vson. Hs research nterests nclude mage analyss and nterpretaton computer vson mathematcal morphology scale-space theory mage regstraton mage sequence analyss multspectral mage processng. INTRODUCTION Increasng amounts of magery due to advances n computer technologes and the advent of World Wde Web (WWW have made apparent the need for effectve and effcent magery ndexng and retreval based not only on the metadata assocated wth t (e.g. captons and annotatons but also drectly on the vsual content. Durng the evoluton perod of Content-Based Image Retreval (CBIR research the maor bottlenec has been the gap between low level features and hgh level semantc concepts. Therefore the obvous effort toward mprovng a CBIR system s to focus on methodologes that wll enable a reducton or even n the best case brdgng of the aforementoned gap. Image segmentaton plays a ey role toward the semantc descrpton of an mage snce t provdes the delneaton of the obects that are present n an mage. Although contemporary algorthms can not provde a perfect segmentaton some can produce a rch set of meanngful regons upon whch robust dscrmnant regonal features can be computed. Ths paper presents a strategy for content-based mage retreval. It s based on a meanngful segmentaton procedure that can provde proper dstrbutons for matchng va the Earth mover s dstance as a smlarty metrc. The segmentaton procedure reles on a herarchcal watersheddrven algorthm that extracts meanngful regons automatcally. In ths framewor the proposed robust feature extracton along wth a novel regon weghtng that enhances feature dscrmnaton play a maor role. The complete process for queryng and retreval does not requre any supervson by the user. The only user s nteracton s the selecton of an example mage as query. Expermental results demonstrate the performance of the proposed strategy. Ths paper s organzed as follows: Secton refers to the mage representaton along wth the proposed feature set whch s extracted out of each regon. Secton 3 s dedcated to the descrpton of the selected smlarty metrc and a novel regon weghtng factor whle n Secton 4 expermental results demonstrate the performance of the proposed CBIR strategy. IMAGE REPRESENTATION. Automatc Multscale Watershed Segmentaton The proposed watershed-drven herarchcal segmentaton scheme s based on a modfed verson of an mage segmentaton approach for vector-valued mages presented prevously n Vanhamel et al. (003. It conssts of three basc modules. The frst module (Salent Measure Module s dedcated to a scale-space analyss based on multscale watershed segmentaton and nonlnear dffuson flterng. Ths module creates a weghted regon adacency graph (RAG where the weghts ncorporate the noton of scale. Usng the obtaned multscale RAG the second module (Herarchcal Level Selecton Module extracts a set of parttonng that have dfferent levels of abstracton denoted as herarchcal levels. The last module (Segmentaton Evaluaton Module dentfes the most sutable herarchcal level for further processng whch n ths wor corresponds to the level contanng all sgnfcant mage features.. Regon features Havng obtaned a parttonng of the mage n sgnfcant regons a set of feature based manly on color texture and spatal characterstcs wll be estmated for each regon. We dd not use geometrc propertes snce mage segmentaton does not always provde a sngle regon for each obect n the mage and therefore t s meanngless to compute representatve shape features from such regons. The color space that we use s the RGB color space. Although t does not provde the color compacton of YCrCb and YIQ color space nether the perceptual sgnfcance of Lab and YUV our expermental results showed very good performance for retreval. Let R be a regon n the segmented set { R} wth a set of adacent regons{ N( R }. In our feature set we do not only characterze each sngle regon R but we also characterze ts neghborhood by computng relatonal features. More specfcally the features we compute are descrbed n the followng : mean Color component AR ( C ( x y = µ C( R = ( AR (

3 Obect-Drven Content-Based Image Retreval 9 mean Texture component µ T ( R = W dxdy ( varance Texture component T( R = ( W µ T( R dxdy (3 Area-weghted adacent regon contrast µ Con( R Card ( N ( R = = A( R ( µ C ( R µ C ( R Card ( N ( R = Regon geometrc centrod AR ( (4 AR ( AR ( x y = = (5 GR ( ; xy = ( AR ( AR ( where { R G B} C denotes the th color component value wth T denotes the th texture component value wth [..4] W denotes the magntude of the transform coeffcents of the th texture component as t s gven n Equaton (0 A( R denotes the area of Regon R Card( N( R denotes cardnalty of regon s R neghborhood and ( x y denotes the coordnates of a pxel that belongs to regon. For the texture component we use the log-gabor flters snce natural textures often exhbt a lnearly decreasng log power spectrum. In the frequency doman the log-gabor flter ban (Bgün and Buf 994 s defned as: G ( ( G o o = (6 r r r where ( r are polar coordnates o s the logarthm of r the center frequency at scale MG o s the th orentaton ( NG and G r r G exp exp r = r where and r s defned as: (7 are the parameters of the Gaussan. The N orentatons are taen equdstant Equaton (8 and G the scales are obtaned by dvdng the frequency range nto MG octaves n Equaton (9. max mn = π N G = ( 0 r = 0 r mn ( 3( = + + max mn where = whch yelds M ( G M octaves 4... G. Note that the maxmum MG (8 (9 frequency cannot be larger than the Nyqust frequency and the DC-component of the mage s removed before flterng. We apply the log-gabor flter on the lumnance component of the color mage to extract the raw texture features. W = g L (0 where g s the G counterpart for the spatal doman L s the lumnance component for whch the DC component s removed and denotes the convoluton. 3 IMAGE RETRIEVAL 3. Image smlarty measure The Earth Mover s Dstance (EMD (Rubner and Tomas 003 s orgnally ntroduced as a flexble smlarty measure between multdmensonal dstrbutons. Formally let Q = {( q wq ( q w q ( q } m wq m be the query mage wth m regons and T = {( t w ( t w ( t w } be another mage of the t t n t n database wth n regons where q t denote the regon feature set and w w denote the correspondng weght of q t the regon. Also let d ( q t be the ground dstance between q and t. The EMD between Q and T s then: EMD( Q T m n fd q t = = m n f = = = ( ( where f s the optmal admssble flow from q to t that mnmzes the numerator of Equaton ( subect to the followng constrants: n q t = = m f w f w ( m n m n f = mn( w w (3 q t = = = = In the proposed approach we defne the ground dstance as follows: 3 = µ + β µ + = 4 4 ( µ T + ( T + = = d( q t ( ( C ( Con β( Gx ( ; + β( Gy ( ; (4 where β s a weghtng parameter that enhances the mportance of the correspondng features. 3. Regon weghtng An addtonal goal durng the mage retreval process s to dentfy and consequently to attrbute an mportance n the regons produced by the segmentaton process. Formally we have to valuate the weghtng factors w and q wt n

4 9 I. Pratas B. Gatos S. Perantons I. Vanhamel H. Sahl Equaton (3. Most regon-based approaches (Greenspan et al. 004; Wang et al. 00 relate mportance wth the area sze of a regon. The larger the area s the more mportant the regon becomes. In our approach we defne an enhanced weghtng factor whch combnes area wth scale and global contrast whch can all be expressed by the valuaton of dynamcs of contours n scale-space (Pratas et al. ( t0 ( ( t 999. Let La ( { t a = a a a } be the lnage lst for the contour a where t o s the localzaton scale and the scale t a s the annhlaton scale.e. the last scale n whch the contour was detected (annhlaton scale. The dynamcs of contours n scale space (DCS are defned as: DCS( a = DC( b (5 b L( a More precsely the weghtng factor s computed as follows: w A( R DCS wq = (6 Card ( R w A( R w DCS = DCS Card ( N ( R = (max DCS( αc = (7 Card( N( R For each produced regon we compute the feature set that s descrbed n Secton.. We would le to note that for "EMD JSEG" we compute regon weghts by tang nto account the area of the regon only. In the produced P/R curves we can observe that both "EMD JSEG" and "EMD hwsh" outperform the "EMD RGB". The "EMD JSEG" and "EMD hwsh" methods have a very good absolute performance after a severe testng of usng 0 dfferent queres for each category. Ths can be attrbuted to the proposed strategy that s supported by a meanngful proposed feature set along wth the proposed smlarty metrc that both approaches use. Fnally a comparson between "EMD JSEG" and "EMD hwsh" provdes a better performance for the proposed scheme ("EMD hwsh. Ths can be attrbuted to a better parttonng that can be acheved usng the proposed segmentaton scheme compared to JSEG. Consderng the overall expermental results we strongly beleve that the proposed strategy for unsupervsed mage retreval can gude CBIR applcatons n a robust way not only because t can lead to a relatvely better performance compared to schemes that other segmentaton methods are used but also because our precson / recall curves show that the proposed scheme can acheve an absolute hgh accuracy. where a c denotes the common border of two adacent regons at the localzaton scale A(R denotes the area of regon R and N(R denotes the number of neghbours for regon R. 4 EXPERIMENTAL RESULTS The proposed strategy for content-based mage retreval has been evaluated wth a general-purpose mage database of 600 mages from the Corel photo galleres that contan 6 categores (00 mages per category. The categores are: beaches buses elephants flowers horses and mountans. Evaluaton s performed usng precson versus recall (P/R curves. Precson s the rato of the number of relevant mages to the number of retreved mages. Recall s the rato of the number of relevant mages to the total number of relevant mages that exst n the database. To be obectve we have used 0 dfferent queres for each category and we have averaged the precson/recall values for each answer set. Furthermore we have used a varety of answer sets that range from 0 to 90 mages usng a step of 0. For comparson we have tested our approach denoted as "EMD hwsh" wth two other regon-based mage retreval approaches. All three approaches use as smlarty metrc the Earth Mover s Dstance (EMD whch s adapted to the underlyng feature set of each method. The frst approach s based on a -means clusterng (Kanungo et al. 00 n the RGB color space whch feeds the EMD wth the produced dstrbutons. In the presented (P/R curves (Fgure ths approach s denoted as "EMD RGB". The second approach for comparson that s denoted as "EMD JSEG" uses the state-of-the-art JSEG algorthm (Deng and BManunath 00 for mage segmentaton.

5 Obect-Drven Content-Based Image Retreval 93 REFERENCES Bgün J. and du Buf J.M. (994 N-folded symmetres by complex moments n Gabor space and ther applcaton to unsupervsed texture segmentaton IEEE Transactons on Pattern Analyss and Machne Intellgence Vol. 6 No. pp Deng Y. and Manunath B.S. (00 Unsupervsed segmentaton of color-texture regons n mages and vdeo. IEEE Transactons on Pattern Analyss and Machne Intellgence Vol. 3 No. 8 pp Greenspan H. Dvr G. and Rubner Y. (004 Context-dependent segmentaton and matchng n mage databases Computer Vson and Image Understandng Vol. 93 pp Kanungo T. Mount D. Pato C.D. Netanyahu N.S. Slverman R. and Wu. A.Y. (00 An effcent -means clusterng algorthm: Analyss and mplementaton IEEE Transactons on Pattern Analyss and Machne Intellgence Vol. 4 No. 7 pp Pratas I. Sahl H. and Cornels J. (999 Herarchcal segmentaton usng dynamcs of multscale gradent watersheds. In th Scandnavan Conference on Image Analyss (SCIA 99 pages Rubner Y. and Tomas C. (000 Perceptual metrcs for mage database navgaton Kluwer Academc Publshers Boston. Vanhamel I. Pratas I. and Sahl H. (003 Multscale gradent watersheds of color mages IEEE Transactons on Image Processng Vol. No. 6 pp Wang J.Z. L J. and Wederhold G. (00 SIMPLIcty: Semantcs-Senstve ntegrated Matchng for pcture lbrares IEEE Transactons on Pattern Analyss and Machne Intellgence Vol. 3 No. 9 pp Fgure : Precson / recall curves