A Fuzzy Image Matching Algorithm with Linguistic Spatial Queries

Transcription

1 Fuzzy Matchng lgorthm wth Lngustc Spatal Queres TZUNG-PEI HONG, SZU-PO WNG, TIEN-HIN WNG, EEN-HIN HIEN epartment of Electrcal Engneerng, Natonal Unversty of Kaohsung Insttute of Informaton Management, I-Shou Unversty epartment of Informaton Engneerng, I-Shou Unversty Kaohsung, Tawan, R.O.. bstract: - The functon for spatal queres has been provded n some mage retreval systems for fndng mages satsfyng gven spatal relatons. However, mages themselves may contan ambguty or the mage processng technologes adopted may extract objects not accurately enough. Processng mage queres n a flexble way s thus desred. In ths paper, we thus propose a fuzzy mage matchng algorthm to calculate the fuzzy match degrees of mages wth lngustc spatal queres. It conssts of two man stages. In the frst stage, the objects n a lngustc spatal query are frst used to flter the mages n the mage database, thus avodng some unnecessary checng n the second stage. The fuzzy match degrees between the converted spatal queres and the promsng mages are then calculated n the second stage by usng the fuzzy operatons and the membershp functons. The proposed algorthm s thus sutable for mages wth uncertan objects and satsfes users lngustc queres n a flexble and effcent way. Key-Words: - fuzzy set, fuzzy match, mage database, lngustc query, spatal relaton, membershp functons Introducton Multmeda applcatons have grown very rapdly n recent years due to the dramatc ncrease n computng power and the concomtant decrease n computng cost. Examples nclude educatonal servces, move ndustry, travel ndustry, home shoppng, medcal care, and others []. mong the varous types of multmeda data, mage data are qute commonly seen n real applcatons snce they provde a trade-off between vsblty and data sze. Spatal queres are provded n some mage retrevng systems for fndng mages satsfyng spatal relatons gven. In the past, spatal queres were usually processed n a crsp way. That s, each mage was judged to ether match or non-match a gven query. s themselves may, however, contan ambguty. For example, the boundary of a cloud n an mage s usually not precse. The mage processng technologes adopted may also extract objects not accurately enough, such as f only rectangles are used to represent objects, then an object wth an rregular shape may have a certan degree of error. Processng mage queres n a more flexble way s thus desred. Recently, the fuzzy set theory has been more and more frequently used n real applcatons because of ts smplcty and smlarty to human reasonng. The theory has been appled to many felds such as manufacturng, engneerng, dagnoss, economcs, and others [9][][]. In ths paper, we thus adopt the fuzzy concepts to ncrease the matchng flexblty of mage data. fuzzy mage matchng algorthm s desgned to calculate the fuzzy match degrees wth lngustc spatal queres. It can evaluate the smlarty degrees between queres and mages n a fuzzy way and output the promsng ones. It conssts of two man stages n calculatng the matchng degrees. In the frst stage, t collects the objects n a lngustc spatal query and uses them to flter the mages n the mage database. In the second stage, t then calculates the fuzzy matchng degrees of the spatal relatons between the lngustc query and the promsng mages by the fuzzy operatons. The mages satsfyng user-defned crtera are then output n a descendng matchng order. Revew of Related pproaches great number of technologes based on mage processng and nformaton extracton have been mplemented and appled [5]. ndexng technques have been largely used to support pctoral nformaton retreval from an mage database [4][6][]. n mage can usually be assocated wth two nds of descrptors: nformaton about ts content and nformaton about the spatal relatons of ts pctoral elements [][][7][][4][5]. pproprate data structures must be adopted to store the related nformaton and mae the mage query feasble. In the past, several famous approaches for effectvely retrevng spatal mages were proposed.

2 hang et al. proposed the -Strng approach to represent the spatal relatons of objects n an mage [][]. Lee and Hsu then generalzed -Strng and proposed -Strng to represent the spatal relatons of objects [][4][5]. Some other varants based on -Strng were also proposed n the lterature []. Petragla et al. then proposed the concept of vrtual mages based on -Strng for spatal mage retreval [8]. Snce mages themselves may contan ambguty or the mage processng technologes adopted may extract objects not accurately enough, applyng fuzzy concepts to ncrease the retreval flexblty of mage data s thus a good attempt. heng et al. [7][8] proposed an algorthm for users to retreve smlar mages by fuzzy match. The smlarty values of query mages wth stored mages n an mage database are calculated from the relatve dstances of objects n these mages. In ths paper, we process the mage query n a lngustc way. The proposed approach can process users lngustc spatal queres and fnd a set of promsng mages as output. Revew of Related Fuzzy oncepts Fuzzy set theory was frst proposed by Zadeh and Goguen n 965 []. Fuzzy set theory s prmarly concerned wth quantfyng and reasonng usng natural language n whch words can have ambguous meanngs. Ths can be thought of as an extenson of tradtonal crsp sets, n whch each element must ether be n or not n a set. fuzzy set can be defned by a membershp functon as: µ : X [, ] where [, ] denotes the nterval of real numbers from to, nclusve. The functon can also be generalzed to any real nterval nstead of [,]. There are a varety of fuzzy set operatons. mong them, three basc and commonly used operatons are complement, unon and ntersecton. These operatons wll be used later for fuzzy match n mage databases. of the top edge and the bottom edge of the object. lngustc spatal relaton between two objects n an mage conssts of one or more constrants. Each constrant s a logcal expresson evaluated by the proposed fuzzy matchng algorthm. For example, the spatal relaton Left of (, ) has only the followng one constrant: XU ( ) XL( ) < {[ XU( ) XL( )] + [ XU( ) XL( )]}/ However, the spatal relaton Identcal (, ) have the followng four constrants: XL ( ) XL( ) = {[ XU( ) XL( )]/ + [ XU( ) XL( )]/}/ XU ( ) XU( ) {[ XU( ) XL( )]/ + [ XU( ) XL( )]/ YL ( ) YL ( ) {[ YU ( ) YL ( )] / + [ YU ( ) YL ( )] / YU ( ) YU ( ) {[ YU ( ) YL ( )] / + [ YU ( ) YL ( )] }/ } / / } / = = = The denomnators n the constrants are used to normalze the fuzzy evaluaton values when the constrants are combned. lngustc spatal query may conssts of one or more spatal relatons connected by the and or or logcal connectors. Membershp functons are predefned and used by the fuzzy matchng algorthm to evaluate the fuzzy matchng degrees of mages. Fve membershp functons are defned, respectvely for the comparson operators <<, <, =, > and >>, as shown n Fgure. Y < = > << >> 4 Related efntons n mage table descrbng the nformaton of objects n an mage conssts of seven felds: _Name, Obj_I, Obj_Name, XL, XU, YL and YU. The feld _Name descrbes the name of the mage n whch the object wth Obj_I and Obj_Name appears. fferent objects must have dfferent Obj_Is, but may have the same Obj_Names. XL and XU respectvely represent the X-axs values of the left edge and the rght edge of the object. Smlarly, YL and YU represent the Y-axs values Fgure. The membershp functons for the comparson operators. Fuzzy evaluaton s useful for performng lngustc spatal queres for ambguous and not-accurately processed mages. For example, X

3 Fgure shows three mages wth two objects and. If we mae a query of s left of, then I should best match the query among the three mages n Fgure. I matches better than snce s left of n I, but meets n. In I and, s not certanly left of f and have some ambguty or mprecson n the boundares. It s thus reasonable that these three mages have dfferent fuzzy match values wth the gven query. I I Fgure : The three mages wth dfferent fuzzy match values for the query s left of. The proposed fuzzy mage matchng algorthm thus calculates the fuzzy match degrees of lngustc spatal queres wth stored mages and outputs the ones wth hgh matchng values. The proposed algorthm s descrbed as follows. The fuzzy mage matchng algorthm: Input: lngustc spatal query, an mage database, an mage table descrbng the nformaton of objects n the mages. Output: set of promsng mages matchng the lngustc spatal query. STEP : ollect the object names ncluded n the query. enote them as O,O,,O r. STEP : Fnd the defnton of each spatal relaton R between any two objects O and O j n the query. STEP : Fnd the mages that nclude all the objects n the query from the mage table. STEP 4: o STEPs 5 to 9 for each mage I found n STEP. m STEP 5: alculate the left-hand-sde value v s of m the m-th constrant n, m= to, for each possble combnaton s of O and O j n I, where s the number of constrants for the spatal relaton defnton. m STEP 6: Transform v s nto a fuzzy value m f s usng the gven membershp functons accordng to the comparson m operator n the constrant STEP 7: Evaluate the fuzzy value f s of s satsfyng the spatal relaton defnton. If two m constrants m and are connected by an and logcal operaton, set the resultng membershp value as: m m mn ( f, f ) ; s s If they are connected by an or logcal operator, set the resultng membershp functon as: m m max( f, f ). STEP 8: Set the fuzzy value f of mage I satsfyng the spatal relaton defnton as: f = s l Max s = where l s the number of possble combnatons of O and O j n mage I. STEP 9: Evaluate the fuzzy value f I of mage I n matchng the gven query. If two spatal relatons and exst n the s f lngustc query and are connected by an and logcal operaton, set the resultng membershp value as: mn ( f, f ) ; s If they are connected by an or logcal operator, set the resultng membershp functon as: max( f, f ). j STEP : Output the mages satsfyng user-defned crtera n a descendant order of matchng degrees. Note that several nds of user-defned crtera can be adopted n STEP. For example, users may requre that only the best several matched mages are output or that the mages wth ther matchng degrees larger than a threshold α are output. 5 Two Illustratng Examples In ths secton, two examples are gven to llustrate the proposed fuzzy mage matchng algorthm. Example : ssume an mage database ncludes sx mages I to I 6, whch are shown n Fgure. I I I 4 I 5 I 6 Fgure : n mage database used n Example. j

4 ll the mages have two objects and wth dfferent spatal relatons. ssume each object n an mage has been segmented out as a rectangle. n mage table descrbng the nformaton of objects s shown n Table for effectve mage retreval. Table : The mage table for the gven sx mages. _Name ObjI Obj_Name I O.7.4. I O I O I O.9.6. O O I 4 O I 4 O I 5 O I 5 O I 6 O I ssume now there s a query of fndng the mages where s to the left of. The algorthm proceeds as follows. Step : The objects ncluded n the query are collected. In ths example, the query objects are and. Step : The defnton for the spatal relaton to the left of s: XU ( ) XL( ) <. {[ XU( ) XL( )] + [ XU( ) XL( )]}/ Only one constrant exsts n the defnton. Step : The mages wth both objects and are extracted. In ths example, all the sx mages n Fgure are extracted. Step 4: Steps 5 to 9 are done for mages I to I 6. Step 5: Snce only one constrant exsts for the only one basc spatal relaton n the query, m s thus. The left-sde value of the constrant for each possble combnaton of and n each mage s then calculated. In ths example, only one combnaton of objects and exst for each mage. Tae mage I as an example. The value n I s.-.5/{[.-.7]+[.-.5]}/, whch s The left-sde values of the constrant for all the sx mages are lsted n the second column of Table. Table : The left-sde values and the fuzzy values of the constrant for the matched sx mages. O 6 XL XU( ) XL( ) {[ XU( ) XL( )] + [ XU ( ) XL( )]} / XU YL Fuzzy value I -.77 I I 4..9 I I 6 YU Step 6: The left-sde values (XU()-XL()) of the matched mages n Table are then transformed nto fuzzy values by usng the membershp functon of >. The fuzzy values for mages I to I 6 are lsted n the thrd column of Table. Steps 7 to 9: These steps are omtted n ths example snce only one constrant must be checed and only one combnaton of objects and exsts n each mage. The matched degrees of mages I to I 6 are thus the same as those shown n the thrd column of Table. Step : ssume the user-defned crteron s above a matchng threshold α=.. The mages I, I,, and I 4 are thus output snce ther matchng values are larger than or equal to.. Example above shows a smplest fuzzy query n whch only one constrant exsts. elow, we gve another example to show the processng of a query wth more than one combnaton of matched objects. Example : ssume an mage database ncludng sx mages I to I 6 s shown n Fgure 4. I I I 4 I 5 Fgure 4: n mage database used n Example. The mage table descrbng the nformaton of objects n Fgure 4 s shown n Table. Table : The mage table for the gven sx mages n Fgure 4. _Name I O.5.6. I O I O.7..4 I O I O.8.. I O I O I O O.8. O ObjI O Obj_Name XL.4 O O I 4 O I 4 O I 4 O I 4 O I5 O I 5 O I 5 O I5 O I 6 O I 6 O 6.6. I6 O I 6 O XU I 6 YL.6 YU.5 4

5 ssume now there s a query of fndng the mages where s to the rght of and top-meets. The proposed algorthm proceeds as follows. Step : The objects ncluded n the query of fndng the mages where s to the rght of and top-meets are collected. In ths example, the query objects are,, and. Step : The defnton for the frst spatal relaton s to the rght of ncludes the followng one constrant: XU ( ) XL( ) <. {[ XU( ) XL( )] + [ XU( ) XL( )]} / The defnton for the second spatal relaton top-meets ncludes the followng three constrants: YU ( ) YL( ) =, {[ YU( ) YL( )] + [ YU( ) YL( )]}/ XU ( ) XL( ) >, {[ XU ( ) XL( )] + [ XU ( ) XL( )]/ } XU ( ) XL( ) >. {[ XU( ) XL( )] + [ XU( ) XL( )]} / The two spatal relatons are then ndvdually evaluated. Step : The mages wth the four objects,, and are extracted. In ths example, all the sx mages n Fgure 4 are extracted. Step 4: Steps 5 to 9 are done for each mage. Step 5: The left-sde values of each constrant for each possble combnaton n each mage s then calculated. Tae mage I as an example. ll the left-sde values of the three constrants n the spatal relaton top-meets for all mages are shown n Table 4. Note that there are two possble combnatons of objects and for mage. Table 4: The left-sde values and the fuzzy values of the constrants wth top-meets for the matched sx mages. ombnaton of objects I I.6.4 I I 5 I 6 top meets top meets top meets top meets onstrant f onstrant f onstrant f f Smlarly, the left-sde value n the spatal relaton s to the rght of for each mage s shown n Table 5. Step 6: The left-sde values of the matched mages are then transformed nto fuzzy values by usng the membershp functons of = and > n Fgure Table 5: The left-sde value and the fuzzy value of the constrant wth s to the rght of for the matched sx mages. ombnaton of objects rght of onstrant f rght of f I -.8 I -.8 I 4 I 5 I 6. The results are also lsted n Tables 4 and 5. Step 7: The fuzzy value of each combnaton satsfyng the second spatal relaton top-meets of top meets s calculated as Mn f. Results are top meets = s f s column of Table 4. The shown n the fuzzy value of each combnaton satsfyng the frst spatal relaton s to the rght of s the same as that calculated n Step 6 snce only one constrant exsts for ths spatal relaton. Step 8: There are two combnatons of and n mage. The match degree of wth the spatal relaton top-meets s thus the maxmum of the fuzzy values of the two combnatons. The results are lsted n the last column of Table 4. Step 9: Snce the two spatal relatons s to the rght of and top-meets n the query are connected by an and logcal operaton, the mnmum operaton s adopted to fnd the fnal match degrees. The resultng match degrees are shown n Table 6. Table 6: The resultng match degrees of the sx mages. rght of f top meets f Match degree I I I 4 I 5 I Step : ssume the user-defned crteron s a matchng threshold α=.. The mages I, I, I 4, and I 6 are thus sorted and output snce ther matchng values are larger than or equal to.. 6 onclusons In ths paper, we have proposed a fuzzy mage 5

6 matchng algorthm to calculate the fuzzy match degrees of mages wth lngustc spatal queres. It conssts of two man stages. In the frst stage, the objects n a lngustc spatal query are frst used to flter the mages n the mage database, thus avodng some unnecessary checng n the second stage. The fuzzy match degrees between the converted spatal queres and the promsng mages are then calculated n the second stage by usng the fuzzy operatons and the membershp functons. The proposed algorthm s thus sutable for mages wth uncertan objects and satsfes users lngustc queres n a flexble and effcent way. References []. G.. uchanan and E. H. Shortlffe, Rule-ased Expert System: The MYIN Experments of the Standford Heurstc Programmng Projects, ddson- Wesley, M., 984. [] S. K. hang, Q. Y. Sh, and. W. Yan, Iconc ndexng by -strng, IEEE Transactons on Pattern nalyss and Machne Intellgence, Vol. 9, No., pp. 4-48, 987. [] S. K. hang, E. Jungert and Y. L, Representaton and retreval of symbolc pctures usng generalzed -Strng, SPIE Proc. Vsual ommuncatons and Processng, pp. 6-7, 989. [4] S. K. hang and. Hsu, nformaton systems: Where do we go from here? IEEE Transactons on Knowledge and ata Engneerng, Vol. 4, No. 5, pp. 4-44, 99. [5].. hang and. F. Lee, Relatve coordnates orented symbolc strng for spatal relatonshp retreval, Pattern Recognton, Vol. 8, No. 4, pp 56-57, 995. [6] S. K. hang and E. Jungert, Symbolc Projecton for Informaton Retreval and Spatal Reasonng, cademc Press, 996. [7].. hen and P.. heng, Spatal reasonng and retreval of pctures usng relatve dstances, The 997 Internatonal Symposum on Multmeda Informaton Processng, pp. -5, 997. [8].. hen and P.. heng, new representaton and smlarty retreval algorthm for spatal databases, The Internatonal onference on Systems, Man and ybernetcs, 997 [9] I. Graham and P. L. Jones, Expert Systems - Knowledge, Uncertanty and ecson, hapman and omputng, oston, pp. 7-58, 988. [] P. W. Huang and Y. R. Jean, Usng +-strngs as spatal nowledge representaton for mage database systems, Pattern Recognton, Vol. 7, No. 9, pp.49-57, 994. []. Kandel, Fuzzy Expert Systems, R Press, oca Raton, pp. 8 9, 99. [] W.. Lam and P. H. han, retreval based on fuzzy strng matchng, IEEE Internatonal onference on Systems, Man and ybernetcs, pp , 996. [] S. Y. Lee and F. J. Hsu, -strng: a new spatal nowledge representaton for mage database systems, Pattern Recognton, Vol., pp , 99. [4] S. Y. Lee and F. J. Hsu, Pcture algebra for spatal reasonng of conc mages represented n -Strng, Pattern Recognton Letters, Vol., pp , 99. [5] S. Y. Lee and F. J. Hsu, Spatal reasonng and smlarty retreval of mages usng -Strng nowledge representaton, Pattern Recognton, Vol., pp , 988. [6] G. Petragla, M. Sebollo, M. Tucc and G. Tortora, Towards normalzed conc ndexng, The 99 IEEE Internatonal Symposum on Vsual Language, pp. 9-94, 99. [7] G. Perragla, M. Sebollo, M. Tucc and G. Tortora, Rotaton nvarant conc ndexng for mage database retreval, Progress n nalyss and Processng, Vol., S. Impedovo, ed., World Scentfc, pp. 7-78, 994. [8] G.Petragla, M. Sebllo, M. Tucc and G. Tortora, Vrtual mage for smlarty retreval n mage database, IEEE Transactons on Knowledge and ata Engneerng, Vol., No. 6, pp ,. [9] G. Shafer and R. Logan, Implementng empster's rule for herarchcal evdence, rtfcal Intellgence, Vol., No., pp. 7-98, 987. [] V. S. Subrahmanan, Multmeda atabase Systems, Morgan Kaufmann Publshers, 998. [] S. L. Tanmoto, n conc symbolc data structurng scheme, Pattern Recognton and rtfcal Intellgence, New Yor: cademc Press, pp , 976. [] L.. Zadeh, Fuzzy logc, IEEE omputer, pp. 8-9, 988. [] H. J. Zmmermann, Fuzzy Sets, ecson Mang, and Expert Systems, Kluwer cademc Publshers, oston, 987. [4] H. J. Zmmermann, Fuzzy Set Theory and Its pplcatons, Kluwer cademc Publsher, oston, 99. 6