Image Interpretation Based On Similarity Measures of Visual Content Descriptors An Insight Mungamuru Nirmala

Transcription

1 Internatonal Journal of Computer Scence & Emergng Technologes (E-ISS: ) 242 Image Interpretaton Based On Smlarty Measures of Vsual Content Descrptors An Insght Mungamuru rmala Lecturer, Department of Computer Scence, Ertrea Insttute of Technology, Asmara, State of Ertrea. Kalyaperumal Karthkeyan, Lecturer, Department of Computer Scence, Ertrea Insttute of Technology, Asmara, State of Ertrea. Sreedhar Appalabatla Faulty, Department of Informaton and Communcaton Technology Zoba Maekel, Mnstry of Educaton, Asmara, Ertrea, orth East Afrca Emal: Raja Adeel Ahmed Lecturer, Department of Computer Scence, Ertrea Insttute of Technology, Asmara, State of Ertrea. Emal: Abstract: Effcent and effectve retreval technques of mages are desred because of the explosve growth of dgtal mages. Content based mage retreval s a promsng approach because of ts automatc ndexng and retreval based on ther semantc features and vsual appearance. Interest n the potental of dgtal mages has ncreased enormously over the last few years. Content-Based Image Retreval (CBIR) s desrable because most webs based mage search engnes rely purely on the meta-data whch produces a lot of garbage n the result. In the content-based mage retreval, the search s based on the smlarty of the content of the mages such as color, texture and shape. However, a pcture s worth a thousand words. Image contents are much more versatle compared wth text, and the amount of vsual data s already enormous and stll expandng very rapdly. Most content based mage retreval systems focus on overall and qualtatve smlarty of scenes. Conventonal nformaton retreval s based solely on text, and these approaches to textual nformaton retreval have been transplanted nto mage retreval n a varety of ways, ncludng the representaton of an mage as a vector of feature values. By measurng the smlarty between mage n the database and the query mage usng a smlarty measure, one can retreve the mage. Ths paper descrbes Content-Based Image Retreval methods usng vsual content descrptors. Keywords: CBIR (Content Based Image Retreval), vsual content descrptors, smlarty measure, Dgtal mage, (CBVIR) Content-Based Vsual Informaton Retreval, TBIR (Text Based Image Retreval). Introducton Content-Based Vsual Informaton Retreval (CBVIR) s an applcaton of the computer vson to the problem of dgtal pctures retreval n a large data base. Informaton contaned n an mage can be vsual nformaton or semantc nformaton. The vsual nformaton can be stated n general contexts n form of colors, textures, shapes, spatal relatons, or n other specfed forms whch vald n the doman of certan problems. CBIR s a retreval technque whch uses the vsual nformaton by retrevng collectons of dgtal mages. The vsual nformaton are then extracted and stated as a feature vector whch n the sequel then forms a feature database. The retreval of data stored n the form of text or documents s carred out by gvng the keywords n the search engne. The keywords are compared wth the text of the documents n the database. Based on the degree of comparson, the most pertnent documents are retreved. Snce 990s the content-based mage retreval system has been a fast advancng research area and remarkable progress has been acheved n theoretcal research and n developng the retreval system. The deal CBIR system from a user perspectve would nvolve what s referred to as semantc retreval. An advance n CBIR was marked by commercal development of mage retreval system for government organzaton, prvate nsttutons and hosptals. 2. Content Based Image Retreval CBIR has been an nterestng topc of research that attracts many researchers snce the early of 90 s. In the last decade, a lot of progress attaned n theoretcal research CBIR or n the development of the CBIR system. But, up to now there are stll many

2 Internatonal Journal of Computer Scence & Emergng Technologes (E-ISS: ) 243 challengng problems n the feld of CBIR whch attracts attenton of many scentsts from varous dscplnes. Late of 970 s s the begnnng era of research n CBIR. On 979 there was conference on applcaton of database technque n mage, held n Florence. Snce then, the applcaton of mage database management technque became an area of research that attracted many scentsts. The technque used n the begnnng of CBIR dd not use the vsual content yet, but reled on the textual nformaton of each mage -Text Based Image Retreval (TBIR). On the other hand, addtonal textual nformaton s needed of every mage before retreval. Then the mages can be retreved by usng the textual approach DBMS. Text based mage retreval uses the tradtonal database technque to manage the mage database. Through textual descrpton, mage can be organzed based on topc or level of herarches to make the navgaton process and browsng easer. Generally n an mage retreval system, the vsual content of mages s stored n a mult dmenson feature vector. To retreve an mage, users enter nputs of the form mage query or sketch. Then the CBIR system computes the feature vector of the query mages or sketch. Smlarty between the feature vectors of query mage/sketch s obtaned based on a measure of dstance or ndex scheme. Ths means that a good CBIR retreval system must be supported by an accurate smlarty measure. The Robust Dstance for Smlarty Measure of Content Based Image Retreval, Dyah E. Herwndat and San M Isa [3] Fg 2: Typcal flow of CBIR Shape matchng s an mportant ngredent n shape retreval, recognton and classfcaton, algnment and regstraton, and approxmaton and smplfcaton. Varous aspects that are needed to solve shape matchng problems: choosng the precse problem, selectng the propertes of the smlarty measure that are needed for the problem, choosng the specfc smlarty measure, and constructng the algorthm to compute the smlarty. Remco C. Veltkamp, "Shape Matchng: Smlarty Measures and Algorthms," sm, pp.088, Internatonal Conference on Shape Modelng & Applcatons, 200 [7] 4. Vsual Contents Fg : Content based mage retreval System overvew Some of the commercal CBIR systems are IBM s QBIC, Vrage s VIR Image Engne, and Excalbur s Image Retreval Ware. The current CBIR system s lmted to effectvely operate at the prmtve feature level. They are not effcent to search for some semantc queres. For example, say, a photo of a cat. But some semantc queres can be handled by specfyng them n terms of prmtves. 3. Lterature Revew An mage s called the same wth an mage n the database f the value of smlarty measure s small. Vsual contents are dvded as prmtve features and semantc features. Prmtve features are the low-level vsual features such as color, texture, shape and spatal relatonshps - drectly related to perceptual aspects of mage content. It s usually easy to extract and represent these features and farly convenent to desgn smlarty measures. System of mage retreval generally stores vsual contents of an mage n a mult dmenson feature vector. It s a very challengng task to extract and manage meanngful semantcs and to make use of them to acheve more ntellgent and userfrendly retreval. In an mage retreval process, users enter nputs of the form query mage. Dependng upon the mage a sutable content can be chosen and a descrptor s defned for retreval. 4. Content Descrptor A descrptor s the collecton of features or attrbutes of an object. A good vsual content descrptor must be nvarant to the varance caused by the process of mage formaton or local. The global descrptor makes

3 Internatonal Journal of Computer Scence & Emergng Technologes (E-ISS: ) 244 use vsual nformaton from the whole mages, meanwhle the local descrptor makes use vsual nformaton of mage regon to descrbe the vsual content of mages. Any object or mage should satsfy the crtera of both local and global features. To obtan the local descrptors, the mage s parttoned nto parts of equal sze and shape. A better method s to use some crtera to dvde the mage nto homogeneous regons or a regon segmentaton algorthm. A complex method of dvdng an mage s to consder a complete object segmentaton to obtan semantcally meanngful objects lke dog, ball, donkey etc. Dependng upon the mage, the descrpton of the global features and the local features wll dffer wth respect to the vsual content under consderaton. 4.2 Feature Vectors CBR system computes the feature vector of the query mage. Smlarty between the feature vectors of the query mages s obtaned based on the measurement of dstance of ndex scheme. The features of the vsual contents are extracted from the mage. The collecton of features of the contents s known as a feature vector. Therefore, they are mult-dmensonal vectors. Feature vectors descrbe partcular characterstcs of an mage based on the nature of the extracton method. For feature extracton, a number of extracton algorthms have been proposed. Vsual feature extracton s the bass of any content-based mage retreval technque. For example, propertes of a boundng box, a curvature, sphercal functons etc. The feature vectors are used for ndexng and mage retreval usng a smlarty measure. The collecton of feature vectors s termed as feature database of the mages n the database. Feature vector and the dfferent approaches on feature-based smlarty search technques are dscussed n [4] and [5]. 5. Color Color provdes a sgnfcant porton of vsual nformaton to human bengs and enhances ther abltes of object detecton. Each pxel n an mage can be represented as a pont n a 3D color space. The common color spaces used for mage retreval are RGB, CIE L*a*b*, CIE L*u*v*, HSV (or HSL, HSB), and opponent color space. The perceptual attrbutes of color are brghtness, hue and saturaton. Brghtness s the lumnance perceved from the color. Hue refers to ts redness, greenness etc. Unformty of a color s the desrable characterstcs of an approprate color space for mage retreval. If two color pars are equal n smlarty dstance n a color space then they are perceved as equal by the vewer. 5. Color Descrptor Color s wdely used for content-based mage and vdeo retreval n multmeda databases. The retreval method based on the content, color, uses the smlarty measure of features derved from color. For example, the proporton of the colors n the mages s computed for the sample mage. Then the mages n the database havng more or less the same proporton of the colors can be retreved usng the smlarty measures. The descrptors are defned based on the attrbutes of color. The color descrptors are Color Hstogram, Color Moments, Color Coherence Vector, and Color Correlogram. The descrptors color hstogram and color moments do not nclude spatal color dstrbuton. 5.2 Color Hstogram Color hstogram s the most commonly used descrptor n mage retreval. Images added to the collecton are analyzed to compute the color hstogram, whch shows the proporton of pxels of each color wthn the mage. The color hstogram s stored n the database. The user has to specfy the proporton of each color to search for a desred mage. The user can also nput an mage from whch a color hstogram s calculated. The matchng process retreves those mages whose color hstograms match wth those of the query mages very closely. Further the retreval system ncludes: () the defnton of adequate color space wth respect to a specfc applcaton () an approprate extracton algorthms () evaluaton of smlarty measures. The color hstogram extracton algorthm can be dvded nto three steps: () Partton of the color space nto cells () Assocate each cell to a hstogram bn () Count the number of mage pxels of each cell and store ths count n the correspondng hstogram bn. The descrptor s nvarant to translaton and rotaton. 5.3 Color Moments Color moments have been used n many retreval systems, when the mage contans just the object. The frst three moments have been proved to be effcent and effectve n representng color dstrbutons of mages. They are defned as: ( s ( j x j ( x j j ( x j j ) ) 3 ) ) 2 / 2 /3 Where x j s the value of the th color component of the mage pxel j and s the number of the pxels n the mage. 5.4 Color Coherence Vector (CCV)

4 Internatonal Journal of Computer Scence & Emergng Technologes (E-ISS: ) 245 Ths descrptor ncorporates spatal nformaton nto the color hstogram. Each pxel of the mage s classfed as ether coherent or ncoherent, dependng upon whether or not t s a part of a large smlarlycolored regon. A regon s assumed to be large f ts sze exceeds a fxed user-set value. By countng coherence and ncoherent pxels separately, the method offers a fner dstncton between mages than color hstograms. For each color c the number of coherent pxels, α c, and the number of non-coherent pxels, β c, are computed; each component n the CCV s a par (α c, β c ), called a coherence par. The coherence vector s gven by V c, ),(, ),...,(, ) ( c c c2 c2 c c The sum α c + β c s the number of pxels of color c present n the mage; the set of sums for =,2,, represents the color hstogram. 5.5 Color Correlogram The color correlogram encodes the spatal correlaton of colors. The frst and second dmenson of the threedmensonal hstogram s the colors of any pxel par and the thrd dmenson s ther spatal dstance. The color correlogram s a table ndexed by color par, where the k-th entry for <, j> specfes the probablty of fndng a pxel of color j at a dstance k from a pxel of color, n the mage. Let I represent the entre set of pxels of the mage and I c(0 represent the set of pxels whose colors are c(). The color correlogram s defned as: r ( k), j Pr p I, p I c( ) 2 [ p2 Ic( j) / p p2 k] Where, j {, 2,, }, k{,2,,d} and p -p 2 s the dstance between pxels p and p 2. If all possble combnatons of color pars are consdered the sze of the color correlogram wll be very large (O ( 2, d)). But nstead, color auto-correlogram captures the spatal correlaton between dentcal colors, hence the sze s reduced to O(d). Even though the autocorrelogram s havng hgh dmensonalty when compared to color hstogram and CCV, ts computatonal cost s very hgh. 6. Texture Texture s a low-level descrptor for mage search and retreval applcatons. There are three texture descrptors consdered n MPEG-7. It descrbes spatal relatonshps among grey-levels n an mage. Texture s observed n the structural patterns of surfaces of objects such as wood, gran, sand, grass and cloth. The ablty to match on texture smlarty can be useful n dstngushng between areas of mages wth smlar color such as sea and sky, grass and leaves. A varety of technques have been used for measurng texture smlarty. A technque named as modelng-fromrealty [5] has been proposed for creatng geometrc models of vrtual objects, and s used for texture mappng of color mages. The color and texture descrptors [6] of MPEG-7 standard are descrbed and the effectveness of these descrptors n smlarty retreval s also evaluated. 6. Texture Descrptors A basc texture element s known as texels. A Texel contans several pxels. The placement of the texel could be perodc or random. atural textures are random and artfcal textures are perodc or determnstc. Texture can be smooth, regular, rregular, lnear, rppled etc. Texture s broadly classfed nto two man categores, namely Statstcal and Structural. Textures that are random n nature are sutable for statstcal categorzaton. Structural textures are determnstc texels, whch repeat accordng to some placement rules, determnstc or random. A Texel s solated by dentfyng a group of pxels havng certan nvarant propertes, whch repeat n the gven mage. The pxel can be defned by ts gray level, shape, or homogenety of some local property lke, sze, orentaton or concurrence matrx. The placement rules are the spatal relatonshp of the pxels. In determnstc rule, the spatal relatonshps may be expressed n terms of adjacency, closest dstance, etc and the texture s sad to be strong. For randomly placed texels, the assocated texture s sad to be weak and the placement rules may be expressed n terms of edge densty, run length of maxmally connected texels, relatve extreme densty. Apart from these two classfcatons there s another model called Mosac model, whch s a combnaton of both statstcal and structural approaches. The texture s descrbed based on ts features such as, coarseness, contrast, drectonalty, lkelness, regularty and roughness. These features are desgned based on the psychologcal studes on the human percepton of texture. 6.2 Coarseness Coarseness s a measure of the granularty of the texture. Coarseness s calculated usng movng averages. Let M k (x, y) denote the movng average at each pxel (x, y) n the wndow of sze 2 k x 2 k, where k = 0,,, 5) M ( x, y) k x2 k - x2 y2 - k - k j y2 k - g(, j)/ 2 Where g(, j) s the pxel ntensty at (, j). The dfferences between the pars of non-overlappng movng averages n the horzontal and vertcal drectons for each pxel are computed. The value of k, whch maxmzes, the dfference calculated n ether drecton s used to set the best sze (say) S for each pxel. S best (x, y) = 2 k 2k

5 Internatonal Journal of Computer Scence & Emergng Technologes (E-ISS: ) 246 The coarseness s computed by averagng S best over the entre mage. 6.3 Contrast coarseness mxn m n j S best (, j) Contrast may be defned as the dfference n perceved brghtness. Detecton of lght spots depends on the brghtness, sze of the space and duraton as well as the contrast between the spot and the background. contrast / 4 4 Where the kurtoss α 4 = μ 4 / 4, μ 4 s the fourth central moment, and 2 s the varance. The contrast can be computed usng ths formula for both the entre mage and a regon of the mage. 7. Shape Retreval The shape of an object refers to ts profle and physcal structure. These characterstcs can be represented by the boundary, regon, moment, and structural representatons. Shapes are dvded nto two man categores namely, statc shapes and dynamc shapes. Statc shapes are rgd shapes. They do not change due to deformaton or artculaton. For example the shape of a rgd object lke a water jug s a statc shape. The object lke human face s a dynamc shape as the shape of the human face changes wth the change n expressons and actons of the human beng. A number of features of the object shape can be computed for each stored mage. The same set of features s computed for the query mage. The mages that are havng close smlarty wth the query mage are retreved from the database. The varous aspects that are needed to solve shape matchng problems lke choosng precse problem, selecton propertes of smlarty measure are stated n [7]. 7. Shape Descrptors Shape descrptors are classfed nto boundary-based and regon-based methods. Ths classfcaton takes nto account whether shape features are extracted from the contour or from the whole shape regon. That can be further dvded nto structural (local) and global descrptors. If the shape s represented by segments or sectons, t s structural and f t s from the whole shape regon, t s global. Another classfcaton categorzes the shape descrpton nto spatal and transform doman technques, dependng on whether drect measurements of the shape are used or a transformaton s appled. The lterature on contentbased retreval methods are evaluated [8] wth respect to several requrements of the retreval system. 7.2 Shape Matchng The retreval method based on shape can be dvded nto three broad categores: () feature based methods, (2) graph based method and (3) other methods. Feature Based Methods: The shape feature can be classfed nto regeneratve features and measurement features. Boundares, regons, moments, structural and syntactc features are dentcal to the regeneratve features. Geometry and Moments are pared wth measurement features. In 3D shapes, features denote geometrc and topologcal propertes of 3D shapes. Based on the type of shape feature used, the feature based method can be dvded nto: () global features, (2) spatal maps, (3) global feature dstrbutons and (4) local features. The frst three represent features of a shape usng a sngle descrptor. The descrptor s a vector of n-dmenson, and n s fxed for all shapes. The global features are used to characterze the overall shape of the objects. These methods use the global features lke that of area; volume, statstcal moments, and Fourer transform coeffcents. These methods do not dscrmnate above the object detals. These methods support the user feed back. The global feature dstrbuton method s a refnement of the global feature method. Spatal maps are representatons that use the spatal locaton of an object. The entres n the map are locatons of the object and are arranged n the relatve postons of the features n an object. Local features are derved from the segment or part of the mage. At the outset, the mage s parttoned usng a sutable crteron nto equal szes or meanngful objects. Then based on the features, smlarty s measured. Graph Based Methods: In graph based method, the geometrc meanng of a shape s extracted and a graph s constructed. The graph represents the shape components and the lnks. The graph based method ncludes () model graph, (2) Reeb graph, and (3) Skeletons. Model based graph are useful to 3D sold models created usng CAD systems. Ths method s dffcult to fnd the smlarty for models of natural shapes lke humans and anmals. Because of the fact that the shape should be sold, t s dffcult to represent the natural shapes as smlar to a sphere, a cylnder, or a place. The skeletal ponts are connected n an undrected acyclc shape graph, usng Mnmum Spannng Tree algorthm. The shape nformaton of the 3D objects [9] are used to form a skeletal graph and a graph matchng technque s used for retreval Moreover, as an extenson of the skeleton method of shape comparson the objects may be segmented as semantcally meanngful parts (shapes), the skeleton of the parts can be used for smlarty measure. The skeleton of the parts can be converted nto graph, and

6 Internatonal Journal of Computer Scence & Emergng Technologes (E-ISS: ) 247 graph matchng technques can be used for comparson. 8. Smlarty Measure The measure of smlarty between two objects s obtaned, based on the dstance between pars of descrptors usng a dssmlarty measure. If the dstance s small, then t means small dssmlarty and large smlarty. The dssmlarty measure can be defned as a non-negatve valued functon. Let d be the dssmlarty measure on a set S. d: S x S R+ U {0}. The followng propertes are defned on d. () Identty: For all x Є S, d(x, x) = 0 () Postvty: For all x, y Є S, d(x, y)>0, where x y () Symmetry: : For all x, y Є S, d(x, y) = d(y, x) (v) Trangle Inequalty: For all x, y, z Є S, d(x, z) = d(x, y) + d(y, z) (v) Transformaton Invarance: For any chosen transformaton group T, for all x, y Є S, t Є T, d(t(x),t(y)) = d(x, y). The dentty property states that the descrptor s completely smlar to tself. The postve property mples that dfferent descrptors are never completely smlar. Ths s a very strong property for a hgh-level descrptor and t s rarely contented. Ths wll not affect the result much f the dssmlarty s on the neglgble part of the mage. Accordng to human percepton, a vsual content of the mage, say, shape s not always smlar. Human percepton does not fnd a shape x smlar to y, as y s smlar to x. If partal matchng of objects s used, the Trangle Inequalty s not satsfed. Snce the part of the object s matched the dstance between the object would be very small. Transformaton Invarance should be satsfed n all types of descrptors, as the comparson and the extracton process of the descrptors are ndependent of the place, orentaton and scale of the object n the Cartesan coordnate system. If a dssmlarty measure s affected by any transformaton, an alternatve formulaton may be used. For example, (v) can be defned as (v) Transformaton Invarance: For any chosen transformaton group T, for all x,y Є S, t Є T, d(t(x),y) = d(x,y). If all the propertes ()-(v) hold, then dssmlarty s called a metrc. It s called pseudo-metrc f (), () and (v) hold and sem-metrc f only (), () and (). Future Work: The future study nvolves n dervng a retreval method for natural objects n mages. It ncludes comparson of dfferent smlarty search methods and the feature extracton methods for the shape of the objects n the mage. Some of the Challenges are: Semantc gap o The semantc gap s the lack of concdence between the nformaton that one can extract from the vsual data and the nterpretaton that the same data have for a user n a gven stuaton. o User seeks semantc smlarty, but the database can only provde smlarty by data processng. Huge amount of objects to search among. Incomplete query specfcaton. Incomplete mage descrpton. 9. Concluson In ths paper, the basc concept of Content-Based Image Retreval methods usng vsual content s descrbed. The descrptors should be transformaton nvarance to measure the smlarty between any two descrptors. Instead of comparng the query mage as a whole wth the mages n the database, the descrptors are compared for retreval. Wth an extenson of ths comparson the feedback of the retreved mages can be used to further refne the retreval process. Based on the feedback the descrptor can be redefned and an teratve smlarty checkng can be mplemented to mprove the retreval of proper mages wth reference to the query mage. Another approach n CBIR system s that a pror feature extracton s defned. The features are selected from the predefned set of features. Ths method uses a set of prmtve features and logcal features. Based on both prmtve and logcal features, the query s processed. Ths may result n the reducton of cost n feature extracton. References. Eakns, John & Margaret Graham: Content-Based Image Retreval. JISC Technology Applcatons. Report 39: -65 (999). 2. Benjamn, Bustos, Kem Danel, Saupe Detmar & Schreck Tobas: Content-based 3D Object Retreval. (2007). 3. Isa Dyah E. Herwndat and San M: The Robust Dstance for Smlarty Measure of Content Based Image Retreval. Proceedngs of the World Congress on Engneerng Vol II WCE 2009, July - 3, 2009, London, U.K.Benjamn, Bustos, Kem Danel, Saupe Detmar, Schreck Tobas & Dejan V. Vranc: Feature-Based Smlarty Search n 3D Object Databases. ACM Computng Surveys (CSUR), 37: (2005).

7 Internatonal Journal of Computer Scence & Emergng Technologes (E-ISS: ) Benjamn, Bustos, Kem Danel, Saupe Detmar & Dejan V. Vranc: An Expermental Effectveness Comparson of Methods for 3D Smlarty Search. Internatonal Journal on Dgtal Lbrares, Specal Issue on Multmeda Contents and Management, 6(): (2006). 6. Kurazume Ryo, Ko shno, Zhengyou Zhang & Katsush Ikeuch: Smultaneous 2D mages and 3D geometrc model regstraton for texture mappng utlzng reflectance attrbute. Paper presented n the 5th Asan Conference on Computer Vson, Melbourne, Australa (2002). 7. Manjunath, B. S., Jens-Raner Ohm, Vnod V. Vasudevan, Ako Yamada: Color and Texture Descrptors. IEEE Transactons on Crcuts and Systems for Vdeo Technology /6: (200). 8. Remco C. Veltkamp, "Shape Matchng: Smlarty Measures and Algorthms," sm, pp.088, Internatonal Conference on Shape Modelng & Applcatons, 200 Veltkamp. C. Remco (200) Shape Matchng: Smlarty Measures and Algorthms SMI (200). 9. Tangelder, Johan W.H. & Remco C. Veltkamp: A Survey of Content Based 3D Shape Retreval Methods. (2005).